OpenSearch is a highly scalable open-source search and analytics platform for a broad range of use cases. Data can be ingested from various sources and then easily searched, analysed, and visualised. Main use cases include log analytics, application monitoring, anomaly detection, and website search.
OpenSearch and OpenSearch Dashboards were originally derived from Elasticsearch 7.10.2 and Kibana 7.10.2. AWS
started the project at the beginning of this year. The goal is to make OpenSearch into a genuine community-driven effort with several different maintainers. It consists of two main applications plus a set of plugins that further extend the capabilities of OpenSearch.
Also called the “engine,” OpenSearch is a distributed, highly scalable, and highly available search and analytics engine at its core. You can ingest any data in JSON format and use the advanced search, analytics, and aggregation capabilities to make use of your data.
Your data is stored in a cluster and distributed amongst the participating nodes in this cluster. Nodes can be added and removed dynamically, and OpenSearch automatically takes care of balancing and distributing the data amongst the nodes. An OpenSearch cluster can scale up to petabytes of data while still providing ultra-low latency when searching or aggregating data.
While the most prominent use case is probably logs analytics, the possibilities are limitless. For example:
Storing and analyzing financial transactions and scanning them for anomalies to detect possible fraud.
Monitoring application performance in real-time.
Storing medical information like patient data or health-related data points from trackers, smartwatches, scales, and other devices.
Implementing a highly accurate and super fast application search.
OpenSearch runs on all major platforms, including Docker and Kubernetes.
OpenSearch Dashboard is the data visualisation and management tool for OpenSearch. Use it to create stunning visualisations and dashboards of your data stored in Elasticsearch. Dashboards and visualisations can be shared and also embedded in other websites.
Dashboards also come with a couple of management and administrative features which you can use to monitor and configure OpenSearch.
OpenSearch provides a plugin infrastructure so that anyone can extend the already rich capabilities of OpenSearch with new features. Both the OpenSearch engine and Dashboards support plugins.
AWS has already released a lot of powerful plugins such as:
Security: Adds highly advanced role-based security controls so you can define what data a user is allowed to see. This includes index-, document- and field-level controls and an audit log for tracking and storing activities in your cluster for meeting compliance regulations.
Alerting: Get notified automatically if the data in OpenSearch meets specific criteria. For example, receive an email or slack message if the rate of errors on your application logs increased by 20% in the last 30 minutes.
Machine Learning: Use the machine learning capabilities to automatically detect anomalies in your data or for automatic data classification.
SQL: Instead of using the OpenSearch query DSL, you can also use plain-old SQL to query and aggregate data in OpenSearch. No need to learn a new language.
The OpenSearch plugins provided by AWS are based on the Open Distro for Elasticsearch
set of plugins. As OpenSearch, all plugins are licensed under Apache2.
OpenSearch is Vendor- and Cloud-agnostic
Although AWS initiated OpenSearch, it is by no means an “AWS only” project. OpenSearch is a community endeavor, and other companies have already joined in as well:
“We are not alone in our commitment to OpenSearch. Organizations as diverse as Red Hat, SAP, Capital One, and Logz.io have joined us in support.”
(AWS: Introdcing OpenSearch
“Organizations as diverse as Red Hat, SAP, Capital One, and Logz.io have joined us in support.”
All code is published under the Apache2 license. This true open-source license allows you to download, run, modify, and distribute OpenSearch any way you want without consulting legal first.
OpenSearch is not limited to AWS or AWS cloud services. You can run it on any cloud platform of your choice.
The Future of OpenSearch: Our Thoughts
We at Eliatra are very excited about OpenSearch for various reasons.
First, this is a true community-driven open-source project that will always be licensed under Apache2. Elastic announced a couple of license changes this year
, moving away from open-source software altogether. This caused a lot of confusion amongst users, and especially amongst companies that offered Elasticsearch as a managed service.
Constant changes in the licensing model put your investment at risk.
For some companies, the Elastic license changes broke their business model altogether. There is also uncertainty in the market regarding potential future modifications of the Elastic license model. We believe OpenSearch is a better choice when it comes to safety regarding your investment in big data analytics.
AWS promised that while they will probably remain the steward of the OpenSearch project, it is open for anyone to participate and contribute. With big players like Red Hat, SAP or logz.io already contributing, we think we will see a thriving community and ecosystem in the future. By using OpenSearch, you can avoid the dreaded vendor lock-in. No more surprises when “circumstances have changed
Should You Migrate Now?
At the moment, no. OpenSearch is not production-ready yet, but it’s moving fast. While there is no official release date for GA yet, we expect
it in Q3 or Q4 this year.
Should You Plan to Migrate Now?
Absolutely yes. For all the reasons listed above (and more), you should look into OpenSearch or even start with a PoC right now. If you need support or professional services to migrate from Elasticsearch to OpenSearch, Eliatra has it all: