--- title: Cardinality limiting description: Span Metrics convert spans into metrics (for example, requests, errors, and duration). When spans or metric labels include many unique values, you might see a high number of time series. This can degrade performance, increase cost, and break dashboards. url: https://coralogix.com/docs/user-guides/apm-v2/getting-started/span-metrics-cardinality-limits/ product: apm feature: OTel Span Metrics feature_id: apm-otel-collector-spanmetrics feature_identifier: apm/apm-otel-collector-spanmetrics last_updated: '2026-11-12' --- # Span Metrics cardinality limiting Span Metrics convert spans into metrics (for example, requests, errors, and duration). When spans or metric labels include many unique values, you might see a high number of time series. This can degrade performance, increase cost, and break dashboards. This guide introduces the three-layered approach to manage and mitigate cardinality issues: * [Layer 1: Correct instrumentation (prevent cardinality at source)](https://coralogix.com/docs/user-guides/apm-v2/getting-started/span-metrics-cardinality-limits.md#layer-1-correct-instrumentation-prevent-cardinality-at-source) * [Layer 2: Sanitization processors (normalize dynamic data before metrics)](https://coralogix.com/docs/user-guides/apm-v2/getting-started/span-metrics-cardinality-limits.md#layer-2-sanitization-processors-normalize-dynamic-data-before-metrics) * [Layer 3: System-level cardinality limits (protect your metrics pipeline)](https://coralogix.com/docs/user-guides/apm-v2/getting-started/span-metrics-cardinality-limits.md#layer-3-system-level-cardinality-limits-protect-your-metrics-pipeline) ## What is high cardinality? High cardinality happens when a metric has hundreds of thousands of unique label combinations for a single service or operation. Common examples include: * User IDs in span names * Email addresses or IP addresses in attributes * Full URLs with query parameters * Raw SQL statements with literal values When used as metric labels, each unique value may create a separate time series, which leads to: * Explosive time-series growth * Missing or inconsistent metrics * Slow dashboards or stale visualizations * Broken [Fair usage limits](https://coralogix.com/docs/user-guides/account-management/fair-usage/overview.md) Coralogix recommends the following layered approach to manage and mitigate cardinality issues. ## Layer 1: Correct instrumentation (prevent cardinality at source) Prevent high-cardinality data by avoiding dynamic values in span names or labels: * Use generic identifiers instead of specific IDs (for example, `/user/{id}` instead of `/user/12345`). * Avoid labels such as `session.id`, `tenant.id`, `email`, or IP addresses that vary per request. * Use semantic attributes like `http.route`, `db.operation`, `service.version`. ## Layer 2: Sanitization processors (normalize dynamic data before metrics) Sanitization normalizes dynamic or noisy values before Span Metrics generate metric time series. **Why sanitization matters** * Prevents runaway cardinality. * Keeps routes, DB queries, and operations readable. * Ensures clean aggregation for service and database metrics. * Recommended in Span Metrics pipelines. ### Detecting dynamic span names Dynamic span names (for example, names that include IDs, GUIDs, or raw query strings) are a common cause of high cardinality. Use the following method to identify services that generate dynamic span names and to inspect the patterns involved. 1. Find services with many unique span names by running the following query in **Explore spans** to identify services that produce a large number of distinct operation names: ``` source spans | groupby $l.serviceName agg distinct_count($l.operationName) | orderby _distinct_count0 desc ``` This helps you locate services that are likely creating dynamic span names. **Example output:** [![Example](/docs/assets/images/span-metrics-1-b574363fad6cbdc76e374ca8e5a143e0.webp)](https://coralogix.com/docs/assets/images/span-metrics-1-b574363fad6cbdc76e374ca8e5a143e0.webp) 2. Discover dynamic span names within a specific service by running the following: ``` source spans | filter $l.serviceName == `prod` | countby $l.serviceName, $l.operationName | filter _count < 2 ``` Look for non-templatized span names generated by frameworks (for example, GraphQL or auto-generated HTTP routes) or span names that embed IDs, UUIDs, or other request-specific values. **Example output:** [![Example](/docs/assets/images/span-metrics-2-4f6f833199d096d9f01191f3da0ceab9.webp)](https://coralogix.com/docs/assets/images/span-metrics-2-4f6f833199d096d9f01191f3da0ceab9.webp) ### Next step: normalize detected dynamic span names After identifying dynamic span name patterns, proceed to the next section to apply normalization using: * [`spanNameReplacePattern`](https://github.com/coralogix/telemetry-shippers/tree/master/otel-integration/k8s-helm#about-span-metrics) * [`transform` processor patterns](https://github.com/coralogix/opentelemetry-helm-charts/blob/81e7c069480171ca0c5c98a3053dca09eb14ed4d/charts/opentelemetry-collector/examples/daemonset-presets/rendered/configmap-agent.yaml#L175) This ensures dynamic values are replaced with stable templates (for example, `/users/{id}`), significantly reducing cardinality. ### Span name and URL sanitization Span name sanitization is recommended for HTTP client and server spans. For span names, we use a Markov-chain classifier trained on URL path n-grams to detect UUIDs, hashes, and other high-cardinality tokens. Only HTTP client and server spans are affected. Any “gibberish” path segments are replaced with `*`, while the rest of the route remains human-readable. Sanitizers are enabled by default with Span Metrics. You can customize or disable them using the `spanMetricsSanitization` preset. Before: ``` GET /api/v1/users/9f8a7c12-3d4b-4f51-9e2b/profile ``` After: ``` GET /api/v1/users/*/profile ``` Span Metrics sanitization wires the redaction processor into the traces pipeline to scrub high-cardinality span names/HTTP URLs and sanitize database statements whenever Span Metrics presets are enabled. * `enabled` accepts `auto`, `true`, or `false`. The default `auto` turns sanitization on automatically whenever `presets.spanMetrics.enabled` or `presets.spanMetricsMulti.enabled` is true. * Set `enabled: false` to opt out even when Span Metrics are enabled, or `enabled: true` to force sanitization on explicitly. * `sanitize_url` (default `true`) controls whether URL-like span names and URL attributes are normalized (for example, `GET /api/users/42/profile` → `GET /api/users/?/profile`). * `sanitizeDatabases` is an allowlist of database backends whose statements should be scrubbed (valid: sql, redis, memcached, mongo, opensearch, es). All supported backends are sanitized by default. ``` spanMetricsSanitization: enabled: auto # auto: enabled when spanMetrics or spanMetricsMulti presets are enabled sanitize_url: true sanitizeDatabases: - sql - redis - memcached - mongo - opensearch - es ``` ### Database statement sanitization Arguments and literals are stripped; statement shapes remain. Database statements are sanitized using the Datadog obfuscator. SQL, Redis/Valkey, Memcached, MongoDB, OpenSearch, and Elasticsearch payloads have literals and arguments removed, but their overall command structure is preserved. Sanitizers are enabled by default with Span Metrics. You can customize or disable them using the `spanMetricsSanitization` preset. Before: ``` SELECT id, email FROM users WHERE email = 'jan@corp.com' AND id = 9121; ``` After: ``` SELECT id, email FROM users WHERE email = ? AND id = ?; ``` Sanitization runs in the traces pipeline, so it also changes the span names, URL attributes, and database statements on the spans the collector sends to Coralogix, along with the metrics derived from them. ### Keep full database statements on spans To send unmasked database statements to Coralogix, remove the database types you want to keep from `sanitizeDatabases`, or set it to an empty list to turn off database sanitization entirely. URL sanitization stays on. ``` opentelemetry-agent: presets: spanMetricsSanitization: sanitizeDatabases: [] ``` Database span metrics use `db.namespace`, `db.operation.name`, `db.collection.name`, and `db.system` as dimensions, so turning off statement sanitization doesn't add metric cardinality. For other causes of masked or shortened statements, see [Troubleshooting](https://coralogix.com/docs/user-guides/apm/features/database-monitoring.md#troubleshooting) in Databases. Unmasked statements include literal values With database sanitization off, every literal in a query, including personal data passed as a query parameter, is stored on the span in Coralogix. ### Add a URL dimension with a regex filter Span Metrics adds a span attribute to its metrics only when the attribute is listed as a dimension. Adding `url.full` directly creates a time series for every distinct URL, including every query string and resource ID. To keep the URL as a dimension without that growth, copy it into a separate attribute, normalize the copy with regular expressions, and add the copy as the dimension. If your instrumentation sets `http.route`, consider using it instead. It already holds the route template, such as `/users/{id}`. **Kubernetes integration (Helm)** In the Coralogix Helm chart, use two settings of the `spanMetrics` preset: * `transformStatements`: [OTTL](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md) statements that run on each span before Span Metrics reads it. * `extraDimensions`: the span attributes added as metric dimensions. Setting this list replaces the chart's default list, so repeat the defaults and append your attribute. The following example copies `url.full` into `url.full.normalized`, strips the query string, and replaces UUIDs and numeric path segments with `{id}`: ``` opentelemetry-agent: presets: spanMetrics: transformStatements: - set(span.attributes["url.full.normalized"], span.attributes["url.full"]) where span.attributes["url.full"] != nil - replace_pattern(span.attributes["url.full.normalized"], "\\?.*", "") where span.attributes["url.full.normalized"] != nil - replace_pattern(span.attributes["url.full.normalized"], "/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}", "/{id}") where span.attributes["url.full.normalized"] != nil - replace_pattern(span.attributes["url.full.normalized"], "/\\d+", "/{id}") where span.attributes["url.full.normalized"] != nil extraDimensions: # Chart defaults - name: http.method - name: cgx.transaction - name: cgx.transaction.root - name: status_code - name: db.namespace - name: db.operation.name - name: db.collection.name - name: db.system - name: graphql.operation.type # Your URL dimension - name: url.full.normalized ``` With this configuration, `https://shop.example.com/api/orders/1842?ref=email` becomes `https://shop.example.com/api/orders/{id}`. **OpenTelemetry Collector (non-Kubernetes)** Add a `transform` processor to the traces pipeline and list the attribute under `dimensions` in the `spanmetrics` connector: ``` processors: transform/url_dimension: error_mode: ignore trace_statements: - context: span statements: - set(span.attributes["url.full.normalized"], span.attributes["url.full"]) where span.attributes["url.full"] != nil - replace_pattern(span.attributes["url.full.normalized"], "\\?.*", "") where span.attributes["url.full.normalized"] != nil - replace_pattern(span.attributes["url.full.normalized"], "/\\d+", "/{id}") where span.attributes["url.full.normalized"] != nil connectors: spanmetrics: dimensions: - name: url.full.normalized service: pipelines: traces: processors: [transform/url_dimension, batch] exporters: [coralogix, spanmetrics] ``` Before you roll this out: * The normalized attribute is set in the traces pipeline, so it also appears on the spans sent to Coralogix. * Regular expressions catch only the patterns you write. Check the result with the [dynamic span name queries](#detecting-dynamic-span-names), grouping by `url.full.normalized`, and keep the [cardinality limit](#layer-3-system-level-cardinality-limits-protect-your-metrics-pipeline) in place as a safety net. * Coralogix shortens metric label values longer than 512 bytes and appends a checksum, so URLs that differ only after the first 512 bytes still produce distinct series. See [Ingestion & query limits](https://coralogix.com/docs/user-guides/account-management/fair-usage/limits.md). ## Layer 3: System-level cardinality limits (protect your metrics pipeline) Span metrics might generate extremely high cardinality. To mitigate this, Coralogix adopts a mechanism similar to the [OpenTelemetry Metrics SDK cardinality limits](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md). This feature introduces an automatic, configurable cardinality control mechanism within the `spanmetrics` pipeline of the OpenTelemetry Collector. Coralogix detects when services exceed configured cardinality limits and exposes this information so users can identify dropped series and take corrective action early. Detection is currently available at the backend level, with frontend UI visibility planned for future releases. To investigate and act on per-service usage and limit state, use [Service Governance](https://coralogix.com/docs/user-guides/account-management/payment-and-billing/metrics-usage-analyzer/apm.md) in the Metrics Usage Analyzer. ### How the cardinality limit works * A threshold (for example, 100,000) is applied per service per metric. For example, `calls_total{service="order-service"}` will have a 100,000 series cap. * Once the limit is reached, new unique label combinations are not created. * Instead, data is aggregated into a fallback time series marked with: ``` otel_metric_overflow="true" ``` **Example:** Cardinality limit of 3, with 5 time series sent (each with 50 spans): ``` calls_total{span_name="uuid1"} calls_total{span_name="uuid2"} calls_total{span_name="uuid3"} calls_total{otel_metric_overflow="true"} ``` * The first 3 series are preserved. * The remaining 100 spans are collapsed into a single time series tagged with `otel_metric_overflow="true"`. ### Cardinality limit settings To set the cardinality limit with `aggregation_cardinality_limit`, ensure you are using [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CHANGELOG.md#-enhancements--5) version 0.130.0 or later. ### Use aggregation temporality `aggregation_temporality` controls how metric values accumulate. * **Cumulative**: Values increase continuously until the collector restarts; recommended for long-running services. * **Delta**: Values represent changes since the last export; recommended for serverless or short-lived workloads. **Kubernetes integration (Helm)** With the [Coralogix Kubernetes Complete Observability integration](https://coralogix.com/docs/opentelemetry/kubernetes-observability/kubernetes-complete-observability-basic-configuration.md), the cardinality limit is automatically enabled and set to 100,000 by default, starting from Helm chart version v.0.0.203 and later. No additional configuration is required. * To disable the cardinality limit by overriding the default value, add an `aggregationCardinalityLimit` field under the SpanMetrics connector, and set to it to `0`. * To edit the cardinality limit, set the `aggregationCardinalityLimit` field to the desired value, as follows: ``` spanMetrics: enabled: true collectionInterval: "{{.Values.global.collectionInterval}}" metricsExpiration: 5m histogramBuckets: [1ms, 4ms, 10ms, 20ms, 50ms, 100ms, 200ms, 500ms, 1s, 2s, 5s] aggregationCardinalityLimit: 100000 ``` **OpenTelemetry Collector (non-Kubernetes)** Add the `aggregationCardinalityLimit` settings as part of the OTel collector under the `spanmetrics` connector, and set the limit you want. * To disable the cardinality limit, either set the `aggregationCardinalityLimit` field to `0` or remove it entirely. * If you are using the `dbMetric` connector, ensure that the `aggregationCardinalityLimit` field is specified under this connector as well. ``` connectors: spanmetrics: namespace: "" histogram: explicit: buckets: [100us, 1ms, 2ms, 4ms, 6ms, 10ms, 100ms, 250ms] aggregationCardinalityLimit: 100000 ``` ### Retention behavior Tracked time series are stored in-memory only and are cleared when the OpenTelemetry Collector or sending pod restarts - no persistent state is maintained. * If a service stops sending data for 5 minutes, its cache is reset automatically. * If the service is redeployed without stopping data flow, the cache persists; to reset it, either restart the collector or allow the service to idle for 5 minutes. ### Alerting on cardinality overflow Set up [alerts](https://coralogix.com/docs/user-guides/alerting/introduction-to-alerts.md) based on the presence of the label `otel_metric_overflow="true"`. This allows early detection of cardinality issues - as soon as overflow begins, even if only a single value is dropped. Recommended PromQL expression: ``` sum by (service_name) ( duration_ms_bucket{otel_metric_overflow="true"} ) > 0 ``` ## Validation checklist * Check that dimension values and span names are normalized (for example, `/user/{id}` instead of `/user/1234`). * Ensure URL parameters are masked when needed. * Verify database statements have been sanitized. * Confirm histogram buckets and `le` values are consistent across environments. * Use the overflow query to detect dropped series: ``` sum by (service_name) ( traces_spanmetrics_calls_total{otel_metric_overflow="true"} ) ``` ## Best practices summary | Layer | Focus | Techniques | | ----- | -------------------------------------------- | ------------------------------------------------------------------------------------------- | | 1 | Prevent cardinality at source | Use generic names, avoid dynamic labels | | 2 | Normalize remaining dynamic values | Sanitize HTTP spans, rewrite DB statements; Use replace patterns or transform processors | | 3 | Protect the system from leftover cardinality | Add Coralogix cardinality limit setup; Set `aggregationCardinalityLimit`, alert on overflow | ## Next steps Reduce metric volume further by enabling [Compact Span Metric](https://coralogix.com/docs/user-guides/apm-v2/getting-started/using-compact-span-metric.md), which drops dimensions you don't need in production.