Open in ChatGPTOpen in Claude

Recommended configurations

Use this guide to configure Span Metrics for production environments, optimize performance, and integrate features such as sampling, latency buckets, setting metric expiration, Serverless environment, and more.

Generated metrics

When Span Metrics is enabled, the OpenTelemetry Collector generates the following metrics from your spans:

Metric name (OTLP)	Prometheus exported name	Metric type	Auto-generated labels (dimensions)
requests	calls_total	Counter	service.name, span.name, span.kind, status.code
duration	duration_ms_bucket, duration_ms_sum, duration_ms_count	Histogram	service.name, span.name, span.kind, status.code, le (for bucket)
errors	errors_total	Counter	service.name, span.name, span.kind, status.code

Identify Span Metrics among your metrics

Span Metrics use reserved metric names, so you can tell them apart from your application and infrastructure metrics by name and by their span-metric labels (service_name, span_kind, status_code):

• Full Span Metrics: calls_total, errors_total, duration_ms_bucket, duration_ms_sum, duration_ms_count.
• Compact Span Metrics: compact_calls_total and compact_duration_ms_* (Services catalog); db_compact_calls_total and db_compact_duration_ms_* (Databases catalog).

For example, request throughput for a service, taken from Span Metrics, is:

sum(rate(calls_total{service_name="checkout"}[5m]))

Recommended dimensions

The Coralogix values.yaml adds these dimensions to enable APM features and richer analysis:

extraDimensions:
  - name: http.method
  - name: cgx.transaction
  - name: cgx.transaction.root
  - name: status_code
  - name: db.namespace
  - name: db.operation.name
  - name: db.collection.name
  - name: db.system

Enabling the Span Metrics preset also turns on:

• Error tracking (errorTracking): powers the Errors tab.
• Service versioning (serviceVersion): powers grouping service metrics by version. Make sure the service.version attribute is added to your service spans.
• Database metrics (dbMetrics): generates RED metrics for database spans, which is what makes your databases appear as entities in the catalog.

Dimensions in the catalog

The Dimensions you can filter and group by in the Services and Databases catalog are the labels on your span metrics, not raw span attributes. The Traces view, by contrast, shows every span and resource attribute. A span attribute becomes a selectable Dimension only once it is configured as a span-metric dimension and is present on the metric the catalog reads.

Two consequences follow:

• An attribute that is on your spans but not configured as a span-metric dimension does not appear in Dimensions, even though it is still visible on individual traces.
• By default, the catalog reads the Compact Span Metrics, which carry only a small, fixed label set. An attribute outside that set - for example the service.instance.id resource attribute - is not available as a Dimension.

Add a custom dimension

To make an extra span attribute available as a Dimension, add it to the spanMetrics preset:

presets:
  spanMetrics:
    enabled: true
    extraDimensions:
      - name: <span.attribute.name>

If you run your own collector, add it to the spanmetrics connector's dimensions list instead.

Note

extraDimensions promotes span attributes. A resource attribute such as service.instance.id is not promoted this way, and the compact metric the catalog reads keeps only its fixed label set, so adding a resource attribute to extraDimensions does not surface it. To use a resource attribute as a Dimension, first copy it onto the span as a span attribute with a collector transform, then add that span attribute to extraDimensions.

Use span sampling

Span Metrics performs best with minimal span sampling (head or tail) allowing to have enough trace examples to power trace-based features in the UI.

Why sampling matters

While Span Metrics does not require trace sampling to be configured in any specific way, we recommend using a minimal sampling rate, either head or tail sampling, to ensure that enough trace examples remain available for UI features that rely on actual traces. These include service maps, drilldowns, Coralogix traces explorer, Spans Highlights, and more. Span Metrics will still provide complete RED metrics even with sampling, but maintaining a small set of representative traces significantly improves the troubleshooting experience.

By default, head sampling is enabled in our latest Helm chart. This setting can be changed either directly in the values.yaml or from the UI.

To learn more about head and tail sampling, refer to the following documents:

• Head sampling guide
• Tail sampling guide

Manage cardinality

Use the following methods to reduce or control cardinality:

• Normalize span names and URLs.
• Sanitize database statements.
• Remove high-cardinality resource attributes.
• Set aggregationCardinalityLimit and metricsExpiration to protect the pipeline.

For more details, see Span Metrics Cardinality Limiting. To track per-service ingestion and spot services approaching their limit before they trip it, use Service Governance in the Metrics Usage Analyzer.

Manage histogram metric buckets

Histogram buckets define how latency distributions are computed for percentiles, SLOs, and Apdex scoring. Span Metrics does not include default buckets - you must configure them manually.

Configure collector buckets

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [100us, 1ms, 2ms, 4ms, 6ms, 10ms, 100ms, 250ms]

Select thresholds that reflect the latency behavior of your workloads.

Apdex settings

For Apdex scoring, your bucket list must include:

• the Apdex threshold (T)
• the tolerated threshold (4T)

In the example above, the Apdex threshold can be set to 1ms because both T (1ms) and 4T (4ms) are specified.

If either bucket is missing, the Apdex score widget displays an Apdex cannot be calculated message instead of a score. See When Apdex cannot be calculated for the full message and recovery steps.

Modifying buckets used by old SLO or Apdex

If you modify bucket configuration for metrics already used in SLOs or Apdex scoring:

• The active calculation stops.
• The UI shows a bucket mismatch error.
• You must update the SLO/Apdex definition to match the new buckets.
• The calculation restarts as a new SLO.

Selecting buckets

Best practices:

• Include both T and 4T.
• Analyze your latency distribution (minimum, median, maximum).
• Add intermediate buckets for meaningful granularity.
• Exclude buckets for rare anomalies unless they occur frequently.
• Keep bucket lists consistent across all collectors and clusters.

Configure different buckets per application (Kubernetes only)

Use the spanMetricsMulti preset to define bucket sets per service.

presets:
  spanMetricsMulti:
    enabled: false
    defaultHistogramBuckets: [1ms, 4ms, 10ms, 20ms, 50ms, 100ms, 200ms, 500ms, 1s, 2s, 5s]
    configs:
      - selector: route() where attributes["service.name"] == "one"
        histogramBuckets: [1s, 2s]
      - selector: route() where attributes["service.name"] == "two"
        histogramBuckets: [5s, 10s]

Each selector requires an OTTL statement. Find out more here.

Enable spanMetricsMulti in Helm

Use this mode if you want to define metrics for each service. It is recommended to have a broad bucket definition that covers the system. spanMetricsMulti allows for more detailed per-service metrics.

spanMetrics:
  enabled: false

spanMetricsMulti:
  enabled: true

Note

If you use multiple collectors, ensure that the bucket configuration is consistent across all of them.

Set metric expiration

metricsExpiration removes inactive metric series after a defined window to control memory consumption.

metricsExpiration: 5m

Retention behavior

• Metric series are stored only in-memory.
• All series reset on Collector or pod restart.
• If a service stops sending spans for the duration of the expiration period, its series are removed automatically.

Use resource key attributes

Use resource_metrics_key_attributes to group metric series at the resource level.

This stabilizes metric grouping across services, environments, clusters, and tenants.

resource_metrics_key_attributes:
  - service.name
  - service.namespace

Use exemplars

Exemplars link metrics to the traces that generated them, enabling seamless navigation from metric panels to the corresponding span data. This improves issue investigation and helps correlate latency spikes or error rates with specific requests.

Exemplars are enabled by default in most configurations, but you can disable them if your environment if needed:

opentelemetry-agent:
  extraConfig:
    connectors:
      spanmetrics:
        exemplars:
          enabled: true

Configure serverless environments

The Services catalog functions seamlessly with both Span Metrics and Events2Metric, as long as all instructions in the documentation are followed correctly.

To display services based on AWS Lambda in the Services catalog, your organization must send spans or Span Metrics. With Events2Metric (E2M), this is done automatically. For Span Metrics, ensure that Lambda-generated spans are routed through the collector.

Note

This ensures that Lambda-based services, along with their metrics, are displayed in the Services catalog. Serverless catalog is supported only using Events2Metrics.

Next steps

Explore the Span Metrics Cardinality Limiting guide to learn about the three-layered approach to manage and mitigate cardinality issues.