OBI changelog
Behavior changes in each OBI (OpenTelemetry eBPF Instrumentation) release that may affect existing dashboards, queries, span filters, and log parsers. Review the relevant version before upgrading. For current behavior, see the OBI overview and Exported metrics.
v1.0.0 is OBI's first stable release. It promotes the v0.15.2 instrumentation unchanged (no additional metrics, attributes, or feature flags) and establishes a compatibility contract for the v1 line.
- Stable surfaces. The v1 contract covers the public Go API of
go.opentelemetry.io/obi, Config v2 and its schema, documented CLI flags and environment variables, container and binary behavior, and the telemetry classified as stable (its OTLP and Prometheus forms). Within v1, minor releases add only backward-compatible changes and incompatible changes require a new major version. Telemetry classified as development, alpha, beta, or release candidate stays outside the contract, and network-flow attribute names remain excluded pending semantic-convention alignment. - Configuration v2 is the v1 configuration contract. Configuration v1 remains supported but is frozen and receives no fixes or feature work. Use
obi config validateandobi config migrateto move to v2, and canary before rollout. - The telemetry schema advances to
1.0.0(from0.15.2), keeping the existing transformation history without additional renames. - The Collector receiver stays on its own v0 release line. v1.0.0 tags only the core module; the receiver remains
go.opentelemetry.io/obi/collectoratcollector/v0.15.2. Pair them explicitly:go get go.opentelemetry.io/obi/collector@v0.15.2 go.opentelemetry.io/obi@v1.0.0. See OBI as an OpenTelemetry Collector receiver. - Known limitations. Application metrics can intermittently stay absent after a Pod starts or restarts until OBI is restarted; child spans can occasionally start before their parents; and port-based service discovery can select a helper or port-forwarder instead of the workload, so use executable-path, container, or Kubernetes selectors when several processes share a port.
The Coralogix otel-integration Helm chart ships OBI v1.0.0 starting with chart version 0.0.375.
If you are upgrading from before v0.15.2, review the v0.15.2 changes below first: span names, rpc.method values, and several default metric attributes changed.
v0.15.2 supersedes the unpublished v0.15.0 and v0.15.1 releases and carries their changes. It is dominated by span-name changes that align OBI with the OpenTelemetry semantic conventions, and it is the instrumentation that v1.0.0 promotes to stable.
- Span names across many protocols now follow the OpenTelemetry semantic conventions, which recreates span-metric series. Span names feed the
span_namelabel on span metrics (traces_span_metrics_*and the legacytraces_spanmetrics_*), so these renames recreate the affected series. The ones most likely to affect existing queries: gRPC span names andrpc.methoddrop the leading slash (an unreadable path falls back to the span namegrpcand method_OTHER, with the raw value kept inrpc.method_original); Redis span names no longer include the database namespace; AWS span names useS3.,SQS., andSNS.prefixes; GraphQL server span names drop theGraphQLprefix; and GenAI client span names use the operation and request model, with Anthropic'smessageoperation becomingchat. Database and ONC RPC span names also shift. Update dashboards, span filters, and queries that match old span names orrpc.methodvalues; the gRPC and ONC RPCrpc.methodchanges also recreate therpc.client.durationandrpc.server.durationseries. - Three metric attributes become default, recreating their Prometheus series. Database duration metrics (client and server) now carry
db.collection.name, gRPC client duration carriesserver.port, and failed gRPC server duration carrieserror.type. Each adds a Prometheus label (db_collection_name,server_port,error_type), so the affected series are recreated. For Elasticsearch and OpenSearch,db.collection.namecan raise cardinality, exclude it throughattributes->selectif needed. See Exported metrics. - Configuration v2 protocol metric selection now covers span metrics and service graphs. Setting a protocol's
enabled.metrics: falsesuppresses that protocol's span metrics and service graphs in both the OTLP and Prometheus outputs, not just its application metrics. Theobi config migrateCLI reports any coverage it can't preserve when converting a Configuration v1 file; Configuration v1 behavior is unchanged. - HTTP client
server.addressdrops the Host-header port.server.portandurl.fullkeep it. Kubernetes Service hosts that include a port now resolve to the service and namespace, which can change peer and service-graph names. - Messaging attribute changes for MQTT and AMQP. MQTT SUBSCRIBE spans and process metrics use
messaging.destination.subscription.namefor the topic filter instead of the destination name, and shared subscriptions also emitmessaging.consumer.group.name. AMQP OTLP metrics omit empty destinations and add a defaultmessaging.operation.type. These add or shift messaging metric labels. - GraphQL, MCP, and JSON-RPC server spans drop their HTTP attributes (method, status code, route, URL, body size, user agent, and
obi.http.response.observed), keeping the protocol, address, network, and error attributes. Captured HTTP headers are now limited to plain HTTP and GraphQL spans. AWS SNS spans droprpc.system.nameandrpc.method, keeping theirSNS.span names andcloud.region. - Added metrics and instrumentation. .NET CPU (
dotnet.process.cpu.count,dotnet.process.cpu.time) and last-collection heap size (dotnet.gc.last_collection.heap.size) runtime metrics; expanded .NET and Rails route harvesting; an opt-inurl.templateattribute on HTTP client spans; opt-in AWS Route53 resolution of peer names from hosted-zone records; and internal ring-buffer write counters (obi.bpf.ringbuf.writes,obi.bpf.ringbuf.write.failures) in the opt-in eBPF metrics family. See Exported metrics. - Reliability and security fixes. OBI no longer injects trace context into sockets misclassified as HTTP/2, OTLP traces are encoded as JSON when the protocol is
http/json, distinct metric label tuples no longer collide, duplicatetarget.infocounts are fixed when a process moves to an existing service, and the Collector receiver keeps running after its start context ends. Security updates bumpgo.opentelemetry.io/otel/sdkto v1.45.0,golang.org/x/cryptoto v0.52.0,golang.org/x/netto v0.55.0, andgo-gitto v5.19.2. - Go Trace API auto-instrumentation gains an embedded Auto SDK path. OBI can now activate the Auto SDK embedded in
go.opentelemetry.io/otel/tracewithout the separatego.opentelemetry.io/auto/sdkmodule. See Go Trace API auto-instrumentation. - The Collector receiver ships as a separate Go module. It is now
go.opentelemetry.io/obi/collector, with its own release line, and keeps the existing import path. Collector build manifests must require that module. See OBI as an OpenTelemetry Collector receiver.
- Several span attributes are now opt-in.
service.peer.name,http.request.body.size,http.response.body.size, andobi.http.response.observedare no longer emitted by default. Select the ones you need throughattributes->select->traces->include. Service-graph generation that relies onservice.peer.namemust select it explicitly. See Exported metrics. - HTTP metrics carry
error.typeon failed requests. HTTP duration and body-size metrics now adderror.typeby default when a request fails, so the corresponding Prometheus series gain anerror_typelabel (empty for successful requests). This recreates the existing series, so update dashboards, alerts, and recording rules that aggregate them. - HTTP body-size histograms are now opt-in behind
application_sizes. Theapplicationmetrics feature still includes them, butapplication_reddoes not. Addapplication_sizes(or keepapplication) if you needhttp.*.body.size. host.idleaves the span-metric data points. The value is unchanged on the resource, sotarget_infostill carrieshost_id. Consumers that flatten OTLP span metrics to Prometheus and key on a per-serieshost_idshould read it fromtarget_infoinstead; OBI's own Prometheus exporter is unaffected.- Empty-valued span attributes are omitted instead of emitted as empty strings. Update span filters and queries that match empty-string values. Manual span attributes, resource attributes, and metric labels keep their existing behavior.
- External trace-to-profile correlation is now opt-in. The pinned
traces_ctx_v1map is no longer populated by default. Setebpf.populate_trace_context: true(envOTEL_EBPF_BPF_POPULATE_TRACE_CONTEXT=true) if you correlate traces with external profiles or relied on the Go channel span-link fallback. The log enricher and the Node.js manual-span bridge enable it automatically, so trace-log correlation is unaffected. - Elastic Cloud database metrics split by cluster. Elastic Cloud client spans and database-operation metrics now carry
db.namespacefrom theX-Found-Handling-Clusterheader, so existing series split per cluster. AWS SQS spans no longer carryaws.extended_request_id. - Added runtime and protocol metrics. A full .NET runtime metrics family (requires .NET 8+), a JVM
jvm.gc.durationhistogram, the Node.jsv8js.resource.activegauge, MCP operation and session duration histograms, GPU/CUDA metrics, and anobi.stat.tcp.successful.connectionscounter. See Exported metrics. - Broader instrumentation coverage. Adds AWS SNS instrumentation, HTTP route harvesting for Java (now enabled by default on Linux), Django, FastAPI, Flask, Rails, .NET, Symfony, Laravel, and Slim, generic Python
asyncioserver context propagation, and uprobe/kprobe attachment withoutCAP_SYS_ADMINon kernels between 5.14 and 6.6 (memory-level Go context propagation still requiresCAP_SYS_ADMIN).
service.nameandservice.namespaceare no longer default metric labels. They remain resource attributes, available throughtarget_infoand the OTLP resource, but they no longer decorate metric series by default. Dashboards, queries, and alerts that group or filter metrics byservice.nameorservice.namespacemust now select them explicitly through theattributes->selectconfiguration. See Exported metrics.error.typevalues are unified across spans and metrics. The attribute name is unchanged, but its values are now consistent between spans and metrics. Queries and span filters that match specificerror.typevalues may need updating.- Python, JVM, and Node.js runtime metrics are expanded. With the
application_runtimefeature enabled, Python services export CPython garbage-collection metrics, JVM services add class, thread, and CPU metrics alongside the existing memory metrics, and Node.js services add V8 garbage-collection and heap-space metrics. See Exported metrics. - Database and messaging metrics carry additional attributes. Database duration metrics gain
db.namespace,db.response.status_code, andserver.port; messaging spans gainmessaging.operation.name. - Context-propagation and protocol reliability fixes. A group of memory-safety fixes stops the HTTP header injector from writing a
traceparentinto an unrelated TLS stream (which could reset the connection), bounds message-buffer reads so unrelated kernel memory cannot appear in span payloads, and invalidates stale message buffers. Application-suppliedtraceparentheaders are preserved in Go HTTP/2, TCP client and server roles are classified by the local port (fixing swappednetwork.tcp.handshake.rolevalues for same-namespace clients), established idle connections are no longer counted inobi.stat.tcp.failed.connections, and DNS spans follow the OpenTelemetry conventions with IPv6 reverse-DNS support.
- Log enrichment is limited to standard output and standard error. Earlier versions intercepted every pipe write from an instrumented process, which could blank out shell command substitutions and, in some cases, stall shells. OBI now enriches (and suppresses the original of) only writes to a tracked process's standard output or standard error; any other pipe (a command substitution, application IPC, or a pipe on another file descriptor) passes through untouched. See Trace-log correlation.
application_spanandapplication_span_sizesspan-metric features are deprecated. They keep working and still emit the Grafana-conventiontraces_spanmetrics_*metrics, but OBI logs a deprecation warning at startup. Move toapplication_span_otel, which emits the OpenTelemetry-conventiontraces_span_metrics_*metrics.application_span_sizeshas no direct replacement. The closest metrics arehttp.server.request.body.sizeandhttp.server.response.body.size. Enabling a legacy span-metric feature together withapplication_span_otelis rejected, and the*(all) selector keeps only the OTel version. See Exported metrics.- Reliability and accuracy fixes. HTTP/2 headers are decoded in capture order, preventing HPACK dynamic-table drift from attaching the wrong header values to a span; TLS connections that use in-memory BIOs are correlated again; and Aerospike responses that span multiple TCP reads can be reassembled, if you see missed or truncated Aerospike spans, raise the capture buffer with
ebpf.buffer_sizes.aerospike.
OBI v0.12.0 was never published, v0.12.1 supersedes it and carries its changes.
- Server-side database operations report
db.server.operation.duration. Redis, Memcached, and SQL measured on the server side now recorddb.server.operation.duration(db_server_operation_duration_secondsin Prometheus) instead ofdb.client.operation.duration. Update dashboards and alerts that read server-side database latency from the client metric. See Exported metrics. - Four internal eBPF metrics dropped an incorrect counter suffix. In OTLP,
obi.bpf.map.entries_total,obi.bpf.map.max_entries_total,obi.bpf.network.packets.total, andobi.bpf.network.ignored.packets.totalbecomeobi.bpf.map.entries,obi.bpf.map.max_entries,obi.bpf.network.packets, andobi.bpf.network.ignored.packets. In Prometheus, the two map gauges lose the_totalsuffix (obi_bpf_map_entries,obi_bpf_map_max_entries); the packet counters keep it. These belong to the opt-in eBPF metrics family. See Exported metrics. - Node.js event-loop runtime metrics. With the
application_runtimefeature enabled, Node.js services export event-loop utilization, delay statistics (min, max, mean, standard deviation, and percentiles), and busy/idle time. See Exported metrics. - Node.js manual spans can be captured without an SDK. When a Node.js process creates spans through
@opentelemetry/apibut registers no OpenTelemetry SDK, setnodejs.manual_spans: true(envOTEL_EBPF_NODEJS_MANUAL_SPANS) to export them. It stays inactive when an SDK is already present. - Java service-name discovery follows the OpenTelemetry Java conventions. For a Java process without an explicit service name, OBI derives one from the Spring Boot application name, then the JAR manifest
Implementation-Title, then the JAR file name, before falling back tojava.
- Plain-text log enrichment is enabled by default. Every service selected by the log enricher now receives space-separated
trace_id=… span_id=…fields on non-JSON writes, which previously passed through unchanged. This can break downstream parsers for structured non-JSON log formats. To keep the previous behavior, setebpf.log_enricher.plain_text.enabled: falsebefore upgrading. JSON enrichment is unaffected. In Configuration v2 the same setting isextensions.obi.correlation.log_trace_annotation.plain_text.enabled. See Trace-log correlation. - Direct OTLP consumers receive OpenTelemetry dot-notation metric names.
target_infois nowtarget.info,traces_target_infois nowtraces.target.info,traces_host_infois nowtraces.host.info,traces_span_metrics_calls_totalis nowtraces.span.metrics.calls, andtraces_span_metrics_durationis nowtraces.span.metrics.duration. Consumers that scrape through a collector's Prometheus exporter are unaffected, because it maps the dots back to underscores. - Prometheus metric names are now derived from their OTLP definitions, which corrected several of them.
obi_stat_tcp_retransmitsis nowobi_stat_tcp_retransmits_totalandobi_stat_tcp_failed_connectionsis nowobi_stat_tcp_failed_connections_total; the CUDA GPU metrics lost an incorrect suffix, sogpu_cuda_kernel_grid_size_totalis nowgpu_cuda_kernel_grid_size,gpu_cuda_kernel_block_size_totalis nowgpu_cuda_kernel_block_size, andgpu_cuda_memory_copies_bytes_totalis nowgpu_cuda_memory_copies_bytes. Update Prometheus queries, dashboards, and alerts that key on the old names. See Exported metrics. - JVM runtime metrics moved to the shared
application_runtimefeature. The deprecatedapplication_jvmfeature alias and thejvm_runtime_metrics.enabledflag were both removed. Enable JVM and Go runtime metrics throughapplication_runtimeinstead. The OBI-specificobi.jvm.heap.usedmetric (obi_jvm_heap_used_bytesin Prometheus) was dropped; read heap usage fromjvm.memory.usedfiltered onjvm.memory.type=heap. See Exported metrics. - Configuration v2 is available for standalone OBI and the OBI Collector receiver, and Configuration v1 remains supported.
obi config validateandobi config migratecheck and convert existing configurations, failing closed on settings they can't preserve. Native Configuration v2 authoring uses a default-on capture policy: omittingcapture.policy.default_action, or setting it toinclude, captures every process that no rule excludes. Canary the generated configuration before rolling it out, and see the Config v1 to v2 migration guide. The Helm chart values shown throughout these pages use Configuration v1. - Span names and span counts changed for SQL and Kafka. SQL span names follow the OpenTelemetry semantic conventions more closely, SQL type is detected on Go SQL spans, and a multi-topic Kafka
ProduceorFetchrequest now emits one span per topic instead of one span per request. Aggregations that count Kafka spans or group on SQL span names may shift. - Go Trace API auto-instrumentation is activated. Go applications that use the global
otel.TracerAPI without registering an SDK are instrumented automatically. Go runtime telemetry also expands with CPU, memory, scheduler, GC, and histogram metrics. See Go Trace API auto-instrumentation. - gRPC and HTTP/2 context propagation covers connections that were already open when OBI started. Huffman-encoded
traceparentvalues are decoded on ingress (which needs Linux kernel 5.17+), HTTP server context is preserved through response bodies, and Gonet/httpHTTP/1 clients no longer inject a duplicatetraceparent. See Distributed tracing. - Protocol, runtime, and AI/LLM coverage expanded. Aerospike client calls are instrumented; Deno processes are identified as a JavaScript runtime and emit metrics and spans (trace-context propagation for Deno isn't available yet); and Ollama's native
/api/chatand/api/generateendpoints and OpenAI-compatible gateways (LiteLLM, vLLM, LocalAI, OpenRouter, Ollama/v1/) are instrumented, with Gemini SSE streaming responses captured.
- gRPC/HTTP2 network-level context propagation is now supported. OBI propagates the W3C
traceparentheader on gRPC and HTTP/2 requests via HPACK header injection, without requiring memory-level writes. See Distributed tracing. - GenAI attributes aligned with OpenTelemetry semantic conventions. Message shapes for OpenAI, Anthropic, Google Gemini, Qwen, and AWS Bedrock have been normalized. Dashboards or queries keyed on prior attribute names may need updating.
url.queryis now emitted by default on HTTP spans when present, and clienturl.fullincludes the query string. Sensitive query parameters are automatically redacted. The built-in redaction list covers AWS SigV4 (X-Amz-Signature,X-Amz-Credential,X-Amz-Security-Token), legacy AWS signed URLs (AWSAccessKeyId,Signature,SecurityToken), Google (X-Goog-Signature,sig), and common token names (token,access_token,refresh_token,id_token,jwt). Extend the list withOTEL_EBPF_SENSITIVE_QUERY_PARAMS_ADD, or trim it withOTEL_EBPF_SENSITIVE_QUERY_PARAMS_REMOVE.- GraphQL documents are no longer exported by default. Enable them explicitly with
payload_extraction.http.graphql.enabled: true(env:OTEL_EBPF_HTTP_GRAPHQL_ENABLED). - Java route harvesting is disabled by default while JVM-blocking edge cases are investigated. The
http.routeattribute on Java spans falls back to the raw URL path until harvesting is re-enabled upstream. ipcontext-propagation value is deprecated; the undocumentedhttpalias has been removed. Valid values forOTEL_EBPF_BPF_CONTEXT_PROPAGATIONaredisabled,headers,tcp, andall. If you setip, OBI warns and does not enable propagation. Useheadersinstead.- AI/LLM coverage expanded. Tool-call extraction is added for OpenAI, Anthropic, Gemini, and Qwen; MCP tool-call arguments and results are extracted with independent attribute-selection controls; SSE streaming is captured for OpenAI-compatible, Anthropic, and Qwen responses; and vector-database retrieval spans are emitted for Pinecone, Qdrant, Milvus, Zilliz, Chroma, and Weaviate.
- New protocol and platform support. SunRPC/ONC RPC detection is added; automatic OpenShift cluster-name detection is added; JDK 26 and 27 are supported.
- Database error response reporting is now opt-in. OBI no longer captures database error response bodies by default. Enable it explicitly if you previously relied on those fields appearing on SQL spans.
- Generic Go TLS support. TLS visibility extends beyond OpenSSL to custom Go protocols. See Distributed tracing.
- SQL spans now use Server kind. Database spans for SQL instrumentations (MySQL, PostgreSQL, MariaDB) are emitted as
SERVERspans instead ofCLIENT. Update any trace or span filters that key on span kind for SQL operations. - DNS excluded from span and service-graph metrics by default. DNS calls no longer appear in span metrics or in the service graph. If you relied on DNS visibility there, expect those entries to disappear after upgrading.
- Ordinal port guessing disabled by default for network metrics. OBI no longer infers a service identity from ordinal source/destination ports on network flows. Existing queries built on guessed ports may return different results.
- HTTP header enrichment available as an opt-in feature. You can now capture selected HTTP request and response headers as span attributes. Configure under
ebpf.payload_extraction.http.enrichment(env:OTEL_EBPF_HTTP_ENRICHMENT_ENABLED), disabled by default.