Migrating from Elasticsearch to OpenSearch and Eliatra Suite
Eliatra Suite ships a Compatibility Edition, which makes it possible to migrate from Elasticsearch with Search Guard installed to OpenSearch with Eliatra Suite, without downtime.
Reading time: 5 minutes
In this article, we guide you through all the required steps.
Migrating Kibana to OpenSearch Dashboards is also covered, albeit this migration is impossible without a short downtime. The reason is that you can not run Kibana and Dashboards simultaneously.
Before you Start
You can only migrate from Elasticsearch 7.10.2 with Search Guard FLX installed to OpenSearch 2.5.x and 2.6.x. Ensure you have a backup of your Elasticsearch cluster if something goes wrong. You will run a mixed cluster of Elasticsearch and OpenSearch nodes during the migration, and you should try to limit the time you are running this mixed cluster to a minimum. If possible, stop ingesting data during this time frame.
The Eliatra Suite Compatibility Edition is necessary for this migration. You can download it for free from our Maven repository. Depending on the OpenSearch version you are running, download the
- eliatra-suite-opensearch-plugin-1.0.0-compat-edition-os-2.5.0.zip
- eliatra-suite-opensearch-plugin-1.0.0-compat-edition-os-2.6.0.zip
Prerequisites
Versions
Check that you run an Elasticsearch cluster 7.10.2 OSS with installed Search Guard FLX >= 1.1.1. If you are using Kibana, make sure to use Kibana 7.10.2 OSS with the Search Guard FLX Kibana Plugin >= 1.1.0 installed.
Check FLX Configuration Format
The migration is only possible if your Elasticsearch cluster runs Search Guard FLX with the new configuration format. If in doubt, check the following:
- You are using sg_authc.yml and not the legacy sg_config.yml configuration file format
- If you are using DLS, you need to use the new FLX DLS implementation. Check that a sg_authz_dlsfls.yml config is present and that use_impl: flx is set
TLS Certificates
You need TLS certificates for your new OpenSearch cluster that can connect to the existing Elasticsearch cluster. The certificates should be signed by a common Root CA or Intermediate CA.
Index Settings
Each index must have at least one replica.
Migration Order
Define a migration order of your Elasticsearch nodes. Start with data nodes, then every non-master eligible node and then all master eligible nodes.
Migration Tool
Download the Eliatra Suite Migration Tool that we need during the migration process.
Migrating from Elasticsearch to Open Search Eliatra Suite Compatibility Edition
Since Search Guard and Eliatra Suite use different indices to store sensitive information, you should ensure that all these indices are protected against regular user access. This step is optional but recommended. When omitting this step, your cluster is vulnerable during the migration procedure.
Add the following lines to elasticsearch.yml and perform a rolling cluster restart:
copy
searchguard.admin_only_indices:
- "searchguard"
- ".searchguard"
- ".searchguard_*"
- ".signals_watches*"
- ".signals_accounts"
- ".signals_settings"
- ".eliatra_internal__*"
Next, run the Migration Tool against the running Elasticsearch cluster:
copy
./migrationtool-1.0.0.sh migrate-indices
Afterwards, prevent allocation of shards to all Elasticsearch nodes during the migration:
copy
sgctl.sh rest put _cluster/settings --json '{ "persistent" : { "cluster.routing.allocation.exclude.<ATTRIBUTE>" : "<PATTERN matching all elastic nodes>" }}'
Example:
copy
sgctl.sh rest put _cluster/settings --json '{ "persistent" : { "cluster.routing.allocation.exclude._name_" : "elastic-node*" }}'
Performing the Migration
For every single data node in your Elasticsearch cluster:
- Shutdown the node
- Install OpenSearch with Eliatra Suite Compatibility Edition on this node
- Make sure the data directory points to the old Elasticsearch data directory (or copy the data to a new location)
- Edit opensearch.yml and add/change the TLS settings and any other setting you would like to set
- Do not set cluster.initial_cluster_manager_nodes`
- Include the last elastic node you will migrate to discovery.seed_hosts (alongside the other open search nodes)
- Start OpenSearch and wait until the node joins the cluster
- Wait until local recovery finished
- Wait until there are no unassigned, relocating or initializing shards
- Proceed with the next data node
For every other non-master eligible node:
- Install OpenSearch with Eliatra Suite Compatibility Edition on this node
- Make sure the data directory point to the old elastic data directory (or copy the data to a new location)
- Edit opensearch.yml and add/change the TLS settings and any other setting you would like to set
- Do not set cluster.initial_cluster_manager_nodes`
- Include the last elastic node you will migrate to discovery.seed_hosts (alongside the other open search nodes)
- Start OpenSearch and wait until the node joined the cluster
- Proceed with the next node
For every master eligible node
- If this is the last node, now is the time to shut down all Kibana instances
- Install OpenSearch with Eliatra Suite Compatibility Edition on this node
- Make sure the data directory point to the old elastic data directory (or copy the data to a new location)
- Edit opensearch.yml and add/change the TLS settings and any other setting you would like to set
- Do not set cluster.initial_cluster_manager_nodes`
- Include the last elastic node you will migrate to discovery.seed_hosts (alongside the other open search nodes) unless this is the last node you are migrating
- Start OpenSearch and wait until the node joined the cluster
- Proceed with the next node
Congrats! You should now have an OpenSearch only cluster with all your data and a green status!
Kibana / Dashboards
Make sure you have stopped all Kibana instances in the previous steps. Install OpenSearch Dashboards and Eliatra Suite Dashboards Plugin, configure accordingly and start one instance. After OpenSearch Dashboards has finished the saved objects migration and is running properly, you can start additional instances.
Delete the “cluster.routing.allocation.exclude” cluster settings:
copy
sgctl.sh rest put _cluster/settings --json '{ "persistent" : { "cluster.routing.allocation.exclude.<ATTRIBUTE>" : null }}'
Example:
copy
sgctl.sh rest put _cluster/settings --json '{ "persistent" : { "cluster.routing.allocation.exclude._name_" : null }}'
Cleanup and Licensing
If you are running the Enterprise Edition of Eliatra Suite, upload your Eliatra Suite license to the cluster via spctl, e.g.:
copy
./spctl.sh update-license --license=license_eliatra_suite.txt
Download the Eliatra Suite configuration and check sp_roles_mapping.yml, sp_roles.yml and sp_actiongroups.yml wether they refer to legacy roles or actiongroups starting with the SGS_ prefix. If so, remove the prefix from all references and upload the configuration.
Example:
Download the current configuration
copy
spctl.sh get-config --output=config_os
In sp_roles.yml, replace
copy
EXAMPLE_ROLE:
description: "Sample role definition for demonstration purposes"
cluster_permissions:
- "SGS_CLUSTER_MONITOR"
index_permissions:
- index_patterns:
- "*"
allowed_actions:
- "SGS_INDICES_MONITOR"
...
...
With
copy
EXAMPLE_ROLE:
description: "Sample role definition for demonstration purposes"
cluster_permissions:
- "CLUSTER_MONITOR"
index_permissions:
- index_patterns:
- "*"
allowed_actions:
- "INDICES_MONITOR"
...
...
Then upload the configuration via spctl:
copy
spctl.sh update-config config_os/
Migrating to Open Search Eliatra Suite
Eliatra Suite Compatibility Edition is supposed to run only for a short period of time. You should migrate as soon as possible to the Eliatra Suite Default Edition. Since all migrations are already done, you can now do a regular rolling update and replace the Compatibility Edition with the regular Eliatra Suite Edition.
Ready to get started?!
Let's work together to navigate your OpenSearch journey. Send us a message and talk to the team today!
Get in touch