From X-Pack to OpenSearch: Part 2 - Users

OpenSearch

2022-08-31

From X-Pack to OpenSearch: Part 2 - Users

We explain how to migrate users from Elasticsearch X-Pack Security to OpenSearch security

X-Pack Native Realm

The native realm is the internal user database in X-Pack. In OpenSearch Security it is called “Internal User Database”.
Native realm users are stored in a hidden index named .security-7 and they can be added, updated and removed via the user management APIs.
The native realm can have the following settings which we need also to migrate:
    cachte.ttl
    cache.max_users
    cache.hash_algo
    authentication.enabled

Internal User Database

The internal user database in OpenSearch is enabled by default and requires normally no extra configuration.
The configuration resides in config.yml and looks typically like:
config:
  dynamic:
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern
Changes to config.yml must be uploaded into the cluster with securityadmin.sh
In this blogpost we add users through the OpenSearch security API. If you intend to use internal_users.yml to manage the users just copy the migrated users into internal_users.yml and upload via securityadmin.sh
But this is an advanced technique we will not pursue in this article.

Step by Step Migration

We walk through a step by step migration scenario.

Get Users from Native Realm

To migrate the native users configured in X-Pack we need to first retrieve them. There are two options to accomplish that:
    Use the Get User API (needs at least manage_security privileges) to retrieve the user configured in the native realm. This approach has the disadvantage that no password hash is included. If you plan to anyhow assign new passwords to the migrated users in OpenSearch then this not relevant. But if you like to keep the passwords the next option is more suitable.
    Query the .security-7 index to retrieve the users. This time the password hash is included and can be used to migrate the user with the same password he had in X-Pack. That is what we use in this blog post. Your user need the appropriate permissions to query this index like for example the elasticuser has.
To query the index you can use the following curlcommand:
$ curl -Ss -u "elastic:<password>" https://elasticsearch_node:9200/.security-7/_search?pretty&size=10000&q=type:user
This results in something like:
"hits" : [
      {
        "_index" : ".security-7",
        "_type" : "_doc",
        "_id" : "user-johnfoo",
        "_score" : 1.4816045,
        "_source" : {
          "username" : "johnfoo",
          "password" : "$2a$12$QzOf63.lc/QaesXkvmk6DOrmmaL001QqMJ403CoDZPgPjLnag/PQC",
          "roles" : [
            "admin",
            "other_role1"
          ],
          "full_name" : "John Foo",
          "email" : "[email protected]",
          "metadata" : {
            "custom_attribute" : "xyz"
          },
          "enabled" : true,
          "type" : "user"
        }
      }, ...
The fields of interest are:
    _source.username: The username of the user to login with
    _source.password: The hashed password (more on this later)
    _source.roles: The roles the user have
    _source.metadata: Optional metadata associated with the user
To make this less cluttering you can filter the search request with a tool called jq like:
$ curl -Ss -u "elastic:<password>" https://elasticsearch_node:9200/.security-7/_search?pretty&size=10000&q=type:user | jq '.hits.hits[]._source | {(.username): {hash: .password, opendistro_security_roles: .roles, attributes: .metadata}}'
to produce a more clean json output with a few renamings applied (which will help us later):
{
  "johnfoo": {
    "hash": "$2a$12$QzOf63.lc/QaesXkvmk6DOrmmaL001QqMJ403CoDZPgPjLnag/PQC",
    "opendistro_security_roles": [
      "admin",
      "other_role1"
    ],
    "attributes": {
      "custom_attribute" : "xyz"
    }
  }
}
Please note that jq is optional and not strictly required.

Import Users into OpenSeach Internal User Database

To import the users into the internal user database we will use the Create user API:
curl -Ss -X PUT -u "opensearch_admin:<password>" https://opensearch_node:9200/_plugins/_security/api/internalusers/johnfoo -d '
{
  "hash": "$2a$12$QzOf63.lc/QaesXkvmk6DOrmmaL001QqMJ403CoDZPgPjLnag/PQC",
  "opendistro_security_roles": [
  	"admin",
  	"other_role1"
  ],
  "attributes": {
    "custom_attribute" : "xyz"
  }
}'
The values of interest are:
    johnfoo (in the url) mapped from _source.username
    hash mapped from _source.password
    opendistro_security_roles mapped from _source.roles
    attributes mapped from _source.metadata
With combination of bash, jq, curl and a few other unix tools it should be easy to automate user export and import. If you only have a few users you maybe want to do this just manually by copy and paste.
Make sure the roles you refer to in opendistro_security_roles exist (more on that later). If such a roles does not exist the API responds with:
{"status":"NOT_FOUND","message":"Role 'other_role1' is not available for role-mapping."}

Migrate Native Realm Settings

As outlined above we have a few realm settings we need possibly migrate to.
    cachte.ttl: Time to live for cached users. Default in X-Pack is 20 minutes. The corresponding setting in opensearch.yml is
    # default is 60 minutes
    plugins.security.cache.ttl_minutes: 20
    
    cache.max_users: Maximum number of cached users. There is no such configuration option in OpenSearch
    cache.hash_algo: Hashing algorithmus for in-memory cache. There is no such configuration option in OpenSearch
    authentication.enabled: Set to false to disable authentication for native realm. To disable the internal user database you need to adjust the config.yml with securityadmin.sh like explained above.

Migrating Roles and Permissions

When migrating users this normally means also migrating roles and permissions.
In this blog post we map the X-Pack roles directly to OpenSearch roles. As long as you are only using X-Pack built-in roles it is not too hard to map them to the OpenSearch built-in roles.
X-Pack built-in roles OpenSearch built-in role
superuser all_access
kibana_user opensearch_dashboards_user
machine_learning_admin anomaly_full_access
machine_learning_user anomaly_read_access
logstash_admin logstash
reporting_user reports_read_access
snapshot_user manage_snapshots
watcher_admin alerting_full_access
watcher_user alerting_read_access
The next blog post will outline how to create custom roles.

Next Steps

In our next article we will cover the migration of custom roles, role mappings, action groups and permissions.

Where to go Next

Get in touch with us
Send us a message and talk to the team today.
Need help with your setup?
OpenSearch or Elastic Stack – We’ve got you covered.