From X-Pack to OpenSearch: Part 3 – Migrating Custom Roles

This article describes how to migrate custom roles from X-Pack Security to OpenSearch Security.

X-Pack Roles

In X-Pack there are privileges which are a named pre-defined set of actions for operating on indices or the Elasticsearch cluster. For example write is an index level privilege granting write operations on documents stored in indices. The cluster level privilege create_snapshot would allow creating snapshots and to list existing repositories. Privileges are the lowest level of granularity in X-Pack Security - you can not change the pre-defined privileges or operate directly on the action level.

One or more privileges are then combined with a secured resource (typically an index or index pattern) into a permission. For example, we define a permission which grants the read and write privileges to all indices matching the my-index* pattern.

A role, which can be assigned to users via role mapping, is a named set of permissions and look like this:

{
  "cluster":[
    "all"
  ],
  "indices":[
    {
      "names":[
        "index*",
        "some"
      ],
      "privileges":[
        "all"
      ],
      "allow_restricted_indices":false
    },
    {
      "names":[
        "index1*",
        "some1"
      ],
      "privileges":[
        "all"
      ],
      "allow_restricted_indices":false
    },
    {
      "names":[
        "index2*",
        "some1"
      ],
      "privileges":[
        "read"
      ],
      "allow_restricted_indices":false
    }
  ]
}

Roles and role mappings can be retrieved through the X-Pack Security APIs or by querying the .security-7 index described in our last blog post. That is what we do in this blog post.

OpenSearch Security Roles

In OpenSearch Security, roles, mappings, permissions, and actions exist. Contrary to X-Pack Security you can define permissions also on an action level. Pre-defined sets of actions are not called privileges but action groups. This is outlined here:

X-Pack Security	OpenSearch Security	Consists of	Remarks
action^*	action	-	Only in OpenSearch Security
privilege	action group	actions^*
permission	permission	privileges or action groups
role	role	permissions
role mapping	roles mapping	roles and users

*Not directly configurable in X-Pack Security

A role in OpenSearch Security looks like this:

{
  "cluster_permissions":[
    "cluster_all"
  ],
  "index_permissions":[
    {
      "index_patterns":[
        "index*",
        "some"
      ],
      "allowed_actions":[
        "indices_all"
      ]
    },
    {
      "index_patterns":[
        "index1*",
        "some1"
      ],
      "allowed_actions":[
        "indices_all"
      ]
    },
    {
      "index_patterns":[
        "index2*",
        "some1"
      ],
      "allowed_actions":[
        "read"
      ]
    }
  ]
}

Migrate Roles

To migrate our custom roles we need to retrieve the roles from Elasticsearch, change the syntax and structure, map the privilege names to action groups and import the role into OpenSearch. Within the scope of this article, we outline some basic concepts to accomplish this. To deal with JSON output on the command line, we recommend that you install a tool called jq and use the bash shell for the commands stated below.

Let’s issue a curl request to retrieve all our Elasticsearch custom roles:

curl -Ss -u "elastic:<password>" localhost:9200/.security-7/_search?pretty\&size=1000\&q=type:role

Now we leverage jq to make some changes to the roles to make them look more like what we need to import them into OpenSearch:

curl -Ss -u "elastic:<password>" localhost:9200/.security-*/_search?pretty\&size=1000\&q=type:role | jq '.hits.hits[] | { _rolename: ._id | split("-")[1], cluster_permissions: ._source.cluster, index_permissions: ._source.indices } | del(.index_permissions[].allow_restricted_indices)

This yields a result like:

{
  "_rolename": "my_custom_role",
  "cluster_permissions": [
    "all"
  ],
  "index_permissions": [
    {
      "names": [
        "index*",
        "some"
      ],
      "privileges": [
        "all"
      ]
    },
    {
      "names": [
        "index1*",
        "some1"
      ],
      "privileges": [
        "all"
      ]
    },
    {
      "names": [
        "index2*",
        "some1"
      ],
      "privileges": [
        "read"
      ]
    }
  ]
}

Not bad but we also need to rename some JSON fields and map the privilege names to pre-defined default action groups.

Let’s add some bash scripting to tweak the output:

# Save the results from above in a file called roles_mig.tmp
curl -Ss -u "elastic:<password>" localhost:9200/.security-*/_search?pretty\&size=1000\&q=type:role | jq '[.hits.hits[] | { _rolename: ._id | split("-")[1], cluster_permissions: ._source.cluster, index_permissions: ._source.indices } | del(.index_permissions[].allow_restricted_indices) | del(.index_permissions[].allow_restricted_indices)]' > roles_mig_tmp.json

# Loop roles_mig.tmp (every row is a role)
for row in $(cat "roles_mig_tmp.json" | jq -r '.[] | @base64'); do

   # get the count of how many index permissions we have in this role
   index_permissions_count=$(echo ${row} | base64 --decode | jq  '.index_permissions | length')
   let index_permissions_count=index_permissions_count-1

   JQ_NAME_FIX=""
   for i in $(eval echo "{0..$index_permissions_count}")
   do
     # Rename .names to .index_patterns and .privileges to .allowed_actions
     JQ_NAME_FIX=".index_permissions[$i].index_patterns=.index_permissions[$i].names | $JQ_NAME_FIX "
     JQ_NAME_FIX=".index_permissions[$i].allowed_actions=.index_permissions[$i].privileges | $JQ_NAME_FIX "
   done

   # TODO
   # Map privilege names to action groups and/or create custom action groups

	# Extract the role name
   ROLE_NAME=$(echo ${row} | base64 --decode | jq  '._rolename' | tr -d '"')

   # Remove all fields we do not longer need like ._rolename or .privileges
   JSON="$(echo ${row} | base64 --decode | jq  "$JQ_NAME_FIX del(.index_permissions[].names) | del(.index_permissions[].privileges) | del(._rolename)")"

   echo "curl -k -u opensearch_admin:<password> -X PUT https://localhost:9200/_plugins/_security/api/roles/$ROLE_NAME -H 'content-type: application/json' -d '$(echo ${JSON} | jq .)'"
done

The output are almost ready-to-use curl commands to create roles by using the Create role API:

curl -k -u opensearch_admin:<password> -X PUT https://localhost:9200/_plugins/_security/api/roles/my_other_role2 -H 'content-type: application/json' -d '{
  "cluster_permissions": [
    "all"
  ],
  "index_permissions": [
    {
      "allowed_actions": [
        "all"
      ],
      "index_patterns": [
        "index*",
        "some"
      ]
    },
    {
      "allowed_actions": [
        "all"
      ],
      "index_patterns": [
        "index1*",
        "some1"
      ]
    },
    {
      "allowed_actions": [
        "read"
      ],
      "index_patterns": [
        "index2*",
        "some1"
      ]
    }
  ]
}'

Note: The script does NOT replace the privilege names automatically! This is something you need to do manually like described in the next chapter.

Mapping Privileges to Action Groups

In most cases, it should be too hard to map X-Pack Security privileges to OpenSearch Security default action groups. Keep in mind that, even if the name of the privilege and action group sounds similar, the granted actions might not be exactly the same. We recommend reviewing every privilege in the original Elasticsearch role and double-checking that the corresponding action group will at least not permit more permissions than the original privilege.

If there are no default action groups that match your specific needs, you can of course replace a privilege with more than one action group or define a new action group. See also OpenSearch Security Part 1: Concepts.

Cluster Privileges

X-Pack Security privilege	OpenSearch Security default action group
all	cluster_all
create_snapshot	manage_snapshots
manage_index_templates	cluster_manage_index_templates
manage_ingest_pipelines	delete
index	write
manage	manage
manage_pipeline	cluster_manage_pipelines
monitor	cluster_monitor

Indices Privileges

X-Pack Security privilege	OpenSearch Security default action group
all	indices_all
create	index
create_index	create_index
delete	delete
index	write
manage	manage
monitor	indices_monitor
read	read
write	write + delete

Next Steps

In our next article, we will cover the migration of more complex roles containing field- and document-level security as well as user impersonation, tenants and role mappings.