OpenSearch

2022-08-31

From X-Pack to OpenSearch: Part 2 - Users

We explain how to migrate users from Elasticsearch X-Pack Security to OpenSearch security.

X-Pack Native Realm

The native realm is the internal user database in X-Pack. In OpenSearch Security it is called “Internal User Database”.

Native realm users are stored in a hidden index named .security-7 and they can be added, updated and removed via the user management APIs.

The native realm can have the following settings which we need also to migrate:

cachte.ttl

cache.max_users

cache.hash_algo

authentication.enabled

Internal User Database

The internal user database in OpenSearch is enabled by default and requires normally no extra configuration.

The configuration resides in config.yml and looks typically like:

config:
  dynamic:
    authc:
      basic_internal_auth_domain:
        description: "Authenticate via HTTP Basic against internal users database"
        http_enabled: true
        transport_enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern

Changes to config.yml must be uploaded into the cluster with securityadmin.sh

In this blogpost we add users through the OpenSearch security API. If you intend to use internal_users.yml to manage the users just copy the migrated users into internal_users.yml and upload via securityadmin.sh

But this is an advanced technique we will not pursue in this article.

Step by Step Migration

We walk through a step by step migration scenario.

Get Users from Native Realm

To migrate the native users configured in X-Pack we need to first retrieve them. There are two options to accomplish that:

Use the Get User API (needs at least manage_security privileges) to retrieve the user configured in the native realm. This approach has the disadvantage that no password hash is included. If you plan to anyhow assign new passwords to the migrated users in OpenSearch then this not relevant. But if you like to keep the passwords the next option is more suitable.

Query the .security-7 index to retrieve the users. This time the password hash is included and can be used to migrate the user with the same password he had in X-Pack. That is what we use in this blog post. Your user need the appropriate permissions to query this index like for example the elasticuser has.

To query the index you can use the following curlcommand:

$ curl -Ss -u "elastic:<password>" https://elasticsearch_node:9200/.security-7/_search?pretty&size=10000&q=type:user

This results in something like:

"hits" : [
      {
        "_index" : ".security-7",
        "_type" : "_doc",
        "_id" : "user-johnfoo",
        "_score" : 1.4816045,
        "_source" : {
          "username" : "johnfoo",
          "password" : "$2a$12$QzOf63.lc/QaesXkvmk6DOrmmaL001QqMJ403CoDZPgPjLnag/PQC",
          "roles" : [
            "admin",
            "other_role1"
          ],
          "full_name" : "John Foo",
          "email" : "[email protected]",
          "metadata" : {
            "custom_attribute" : "xyz"
          },
          "enabled" : true,
          "type" : "user"
        }
      }, ...

The fields of interest are:

_source.username: The username of the user to login with

_source.password: The hashed password (more on this later)

_source.roles: The roles the user have

_source.metadata: Optional metadata associated with the user

To make this less cluttering you can filter the search request with a tool called jq like:

$ curl -Ss -u "elastic:<password>" https://elasticsearch_node:9200/.security-7/_search?pretty&size=10000&q=type:user | jq '.hits.hits[]._source | {(.username): {hash: .password, opendistro_security_roles: .roles, attributes: .metadata}}'

to produce a more clean json output with a few renamings applied (which will help us later):

{
  "johnfoo": {
    "hash": "$2a$12$QzOf63.lc/QaesXkvmk6DOrmmaL001QqMJ403CoDZPgPjLnag/PQC",
    "opendistro_security_roles": [
      "admin",
      "other_role1"
    ],
    "attributes": {
      "custom_attribute" : "xyz"
    }
  }
}

Please note that jq is optional and not strictly required.

Import Users into OpenSeach Internal User Database

To import the users into the internal user database we will use the Create user API:

curl -Ss -X PUT -u "opensearch_admin:<password>" https://opensearch_node:9200/_plugins/_security/api/internalusers/johnfoo -d '
{
  "hash": "$2a$12$QzOf63.lc/QaesXkvmk6DOrmmaL001QqMJ403CoDZPgPjLnag/PQC",
  "opendistro_security_roles": [
  	"admin",
  	"other_role1"
  ],
  "attributes": {
    "custom_attribute" : "xyz"
  }
}'

The values of interest are:

johnfoo (in the url) mapped from _source.username

hash mapped from _source.password

opendistro_security_roles mapped from _source.roles

attributes mapped from _source.metadata

With combination of bash, jq, curl and a few other unix tools it should be easy to automate user export and import. If you only have a few users you maybe want to do this just manually by copy and paste.

Make sure the roles you refer to in opendistro_security_roles exist (more on that later). If such a roles does not exist the API responds with:

{"status":"NOT_FOUND","message":"Role 'other_role1' is not available for role-mapping."}

Migrate Native Realm Settings

As outlined above we have a few realm settings we need possibly migrate to.

cachte.ttl: Time to live for cached users. Default in X-Pack is 20 minutes. The corresponding setting in opensearch.yml is

# default is 60 minutes
plugins.security.cache.ttl_minutes: 20

cache.max_users: Maximum number of cached users. There is no such configuration option in OpenSearch

cache.hash_algo: Hashing algorithmus for in-memory cache. There is no such configuration option in OpenSearch

authentication.enabled: Set to false to disable authentication for native realm. To disable the internal user database you need to adjust the config.yml with securityadmin.sh like explained above.

Migrating Roles and Permissions

When migrating users this normally means also migrating roles and permissions.

In this blog post we map the X-Pack roles directly to OpenSearch roles. As long as you are only using X-Pack built-in roles it is not too hard to map them to the OpenSearch built-in roles.

X-Pack built-in roles	OpenSearch built-in role
superuser	all_access
kibana_user	opensearch_dashboards_user
machine_learning_admin	anomaly_full_access
machine_learning_user	anomaly_read_access
logstash_admin	logstash
reporting_user	reports_read_access
snapshot_user	manage_snapshots
watcher_admin	alerting_full_access
watcher_user	alerting_read_access