OpenTelemetry Explained

Monitoring versus observability

Monitoring checks known conditions: CPU is high, error rate crossed a threshold. You decide the questions in advance. Observability is the ability to ask new questions of a running system using the data it emits, without shipping new code. It needs rich, correlated telemetry rather than a fixed set of dashboards.

OpenTelemetry (OTel) is the open standard for generating that telemetry. It defines an API, SDKs, a wire protocol and a Collector, but deliberately does not provide storage or a UI. You send the data to whichever backend you choose, so switching vendors does not mean re-instrumenting.

The three signals

Signal	What it answers
Traces	What path did one request take across services, and where did time go?
Metrics	How is the system behaving in aggregate over time?
Logs	What exactly happened at a given moment?

Traces and spans

A trace is a tree of spans. Each span records an operation with a name, start and end time, attributes, events, a status and a parent. Its span context carries the trace ID, span ID and flags. Span kinds (server, client, producer, consumer, internal) describe the role of the operation. Span links connect causally related spans that are not parent and child, such as a batch job consuming many messages.

Metrics

Instruments include counters, up-down counters, gauges and histograms. Measurements are aggregated in the SDK before export. Beware of cardinality: each unique attribute combination becomes a time series, so avoid unbounded attribute values.

Logs

Existing logging libraries are bridged into OTel so that log records can carry the current trace and span IDs. That correlation is what lets you jump from an error log to the exact trace.

Resources and semantic conventions

A resource describes the entity producing telemetry (service.name, version, host, pod). Semantic conventions standardise attribute names such as http.request.method, so different services and tools agree on meaning. Always set service.name.

API versus SDK

The API is what instrumentation code calls: create a tracer, start a span, record a measurement. On its own it does nothing (a no-op). That means libraries can instrument themselves against the API without forcing any dependency on a particular implementation.

The SDK is the implementation the application owner wires in. It handles sampling, span processors, metric readers and aggregation, resource detection and exporters. The separation defines roles:

• Library authors depend only on the API.
• Application owners install and configure the SDK.
• Plugin authors write exporters, samplers or propagators against SDK interfaces.

Instrumentation

There are three levels, and they combine well:

1. Zero-code (automatic): an agent or loader attaches at startup and instruments common frameworks and clients, for example through bytecode manipulation on the JVM or monkey-patching in interpreted languages. Fastest way to get coverage.
2. Library instrumentation: packages that wrap a specific HTTP framework, database driver or message client.
3. Manual: you add spans and attributes around your own business logic.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("checkout")

with tracer.start_as_current_span("charge-card") as span:
    span.set_attribute("order.items", len(items))
    result = payment_client.charge(order)
    if not result.ok:
        span.set_status(Status(StatusCode.ERROR))

On the SDK side, a sampler decides which traces to keep (for example a fixed percentage), a span processor hands finished spans to an exporter, and batching processors are preferred in production because they avoid a network call per span.

Context propagation and baggage

A distributed trace only works if every service knows which trace it belongs to. The caller injects context into the outgoing request headers and the callee extracts it. The default format is the W3C Trace Context standard, carried in the traceparent header (version, trace ID, parent span ID, flags) and optionally tracestate. Other formats such as B3 are available through propagators, and a composite propagator can handle several at once.

traceparent: 00-<32-hex-trace-id>-<16-hex-span-id>-01

Baggage is separate key-value data that travels with the context across service boundaries, such as a tenant ID or feature flag. It is not attached to spans automatically; you read it and copy what you need into attributes.

Propagation breaks silently when a proxy strips headers, or when work crosses an async boundary (thread pool, queue) without explicitly carrying the context. Missing parent links in a trace usually point to one of these.

OTLP

The OpenTelemetry Protocol is the native format for all three signals. It is defined with Protocol Buffers and comes in two transports:

• OTLP/gRPC, conventionally on port 4317.
• OTLP/HTTP with protobuf encoding, conventionally on port 4318, using paths such as /v1/traces, /v1/metrics and /v1/logs. A JSON encoding is also defined.

HTTP is friendlier to proxies and load balancers, while gRPC is efficient for streaming. Most SDKs can be pointed at an endpoint with standard environment variables such as OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME and OTEL_TRACES_SAMPLER.

The Collector

The Collector is a standalone service that receives, processes and forwards telemetry. It is optional but valuable: applications send to a nearby Collector and stay simple, while retries, batching, filtering and routing are handled centrally. It is not a storage system or query tool.

A pipeline is made of three kinds of component:

• Receivers ingest data (OTLP, and many other formats such as Prometheus scraping).
• Processors transform it: batch, memory_limiter, attribute edits, filtering, tail sampling.
• Exporters send it onward to backends.

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  batch: {}

exporters:
  otlphttp:
    endpoint: https://telemetry.example.com

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

A component only runs if it is listed under service.pipelines; defining it at the top is not enough. Order of processors matters, with memory_limiter first. Common deployment shapes are an agent per node or sidecar and a central gateway tier; tail-based sampling needs all spans of a trace to land on the same gateway instance.