From Compatible to Production-Ready: How Eliatra Implements OpenTelemetry on OpenSearch

Part 1 of this series made the case that OpenTelemetry and OpenSearch are a natural technical fit. That’s true, and it’s also only half the story. “The pieces are compatible” and “this runs reliably in production” are different claims, and the distance between them is exactly where Eliatra does its work. This article walks through what that implementation work actually involves.

Data Prepper is OpenSearch’s server-side collection component, and it has purpose-built processors for OpenTelemetry traces and metrics — including OTel’s exponential histograms and other metric types. That capability existing is not the same as a pipeline that performs well at real volume. Eliatra’s pipeline design work covers the parts that don’t show up in a quickstart guide: buffering and backpressure so a traffic spike upstream doesn’t take down ingestion, dead-letter handling so malformed spans get captured rather than silently dropped, and pipeline topology that separates trace, metric, and log processing so one signal type’s load doesn’t degrade another’s.

This is also where OTel Collector configuration and Data Prepper configuration have to agree with each other — batch sizes, export intervals, and retry behaviour all need to be tuned as a pair, not independently, or the two ends of the pipeline end up working against each other under load.

Observability data is a specific kind of hard for a search backend to handle well: high cardinality (unique trace IDs, span IDs, and attribute combinations by the millions), high volume, and a strong bias toward recent data for most queries. Eliatra tunes index strategy — rollover policy, shard count and sizing, and index state management — around those characteristics specifically, rather than applying the same defaults used for a general-purpose search index. Getting this wrong doesn’t fail loudly; it shows up months later as slow dashboards and an oversized cluster, which is a more expensive problem to unwind than to avoid.

The Simple Schema for Observability (SS4O) is what makes it possible, in principle, to correlate a trace with the logs and metrics from the same request. In practice, that only works if every ingestion path — traces through Data Prepper, metrics through their own processors, logs from whatever collection agents are in use — is mapped to SS4O consistently. A schema that’s 90% aligned still breaks correlation queries, because the one field that doesn’t match is usually the join key. This consistency work, across every signal type and every source, is a core part of what Eliatra configures rather than something a team layers on after the fact.

The Observability plugin’s Trace Analytics feature builds service maps and enriches spans automatically once trace data is flowing correctly — but “flowing correctly” is doing a lot of work in that sentence. Eliatra’s configuration work here includes setting up service map enrichment so it reflects the organisation’s actual service topology, and — for organisations migrating off an existing Jaeger deployment — configuring OpenSearch to consume Jaeger-formatted trace data alongside OTel data, so the two can run in parallel during a transition instead of forcing a hard cutover.

This is the point that’s easiest to skip and the most consequential to skip. Traces and logs routinely carry sensitive operational detail — internal service names, request paths, sometimes customer data embedded in span attributes — and observability indices are frequently configured with looser access controls than the rest of a cluster simply because they were the last thing added. Eliatra’s grounding is in IT security specifically: the team behind Eliatra’s Encryption at Rest plugin, which protects OpenSearch data at rest across cloud and on-premises deployments. That same rigour is applied to observability indices as a matter of course, not as an add-on requested later.

None of this is a criticism of OpenSearch or OpenTelemetry — the underlying compatibility described in Part 1 is genuine. It’s a description of the distance between a proof of concept and a production system that an on-call engineer trusts at 3 a.m. Eliatra’s custom development and professional services exist specifically to close that distance: correctly configured Data Prepper pipelines, an SS4O-aligned schema that actually enables correlation, and a security configuration that treats observability data with the same care as everything else in the cluster.