--- title: Monitor NVIDIA GPUs with OpenTelemetry description: 'Collect NVIDIA GPU telemetry with Coralogix OpenTelemetry: DCGM Exporter hardware metrics, OBI CUDA workload metrics, or both.' url: https://coralogix.com/docs/opentelemetry/integrations/nvidia-gpu-monitoring/ product: infra-monitoring feature: Infra Monitoring Overview feature_id: infra-monitoring-general feature_identifier: infra-monitoring/infra-monitoring-general last_updated: '2026-10-21' --- # Monitor NVIDIA GPUs with OpenTelemetry Coralogix OpenTelemetry offers two complementary ways to monitor NVIDIA GPUs: * **Hardware telemetry with [NVIDIA DCGM Exporter](https://docs.nvidia.com/datacenter/dcgm/latest/installation/install-dcgm-exporter.html)**: GPU utilization, framebuffer memory, temperature, power, clocks, and hardware errors. The Coralogix OpenTelemetry Collector scrapes the exporter over HTTP. * **CUDA workload telemetry with [OBI](https://coralogix.com/docs/opentelemetry/instrumentation-options/ebpf-auto-instrumentation/overview.md)**: CUDA runtime activity such as kernel launches, device memory allocations, and host/device transfers, attributed to the processes using the GPU. The OBI eBPF agent captures it with no code changes. The two collect different data under different metric names and can run together. Use the comparison below to choose one or both. ## Choose a collection method | | DCGM Exporter | OBI | | ---------------------- | ----------------------------------------------------------------------- | --------------------------------------------------------------------------- | | **What it measures** | GPU hardware health and saturation | CUDA API activity from your applications | | **Example metrics** | Utilization, framebuffer memory, temperature, power, clocks, XID errors | Kernel launches, device memory allocations, memory copies, synchronizations | | **Perspective** | Per GPU device | Per process, pod, and service | | **How it's collected** | Collector scrapes DCGM Exporter over HTTP | OBI eBPF agent instruments the CUDA runtime; no scrape target | | **Requires** | DCGM Exporter deployed (for example, via GPU Operator) | OBI eBPF agent; NVIDIA toolkit present on the node | | **Metric namespace** | `DCGM_FI_*` | `gpu.cuda.*` | Use DCGM Exporter to answer *"is the GPU healthy and how hard is it working?"* and OBI to answer *"what is my application asking the GPU to do, and which workload is responsible?"* Because the two use different metric names, enabling both does not create duplicates or conflicts. ## Collect hardware metrics with DCGM Exporter Use this path to collect NVIDIA GPU hardware metrics exposed by [NVIDIA DCGM Exporter](https://docs.nvidia.com/datacenter/dcgm/latest/installation/install-dcgm-exporter.html). The Coralogix OpenTelemetry Collector scrapes the exporter over HTTP and sends the metrics through the existing metrics pipeline. This path covers hardware telemetry, including GPU utilization, framebuffer memory, temperature, power, clocks, and hardware errors. ### Prerequisites * An NVIDIA GPU with a supported NVIDIA driver. * NVIDIA DCGM Exporter running on the GPU host or Kubernetes cluster. The [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html) can deploy it. * The [Coralogix OpenTelemetry Integration Helm chart](https://coralogix.com/docs/opentelemetry/kubernetes-observability/kubernetes-complete-observability-basic-configuration.md) installed in the cluster. * Network access from the OpenTelemetry Collector to DCGM Exporter's `/metrics` endpoint, normally port `9400`. The Collector only scrapes HTTP metrics. Do not mount GPU devices or grant NVIDIA capabilities to the Collector. Schedule DCGM Exporter only on GPU nodes. Configure the node selector, affinity, and tolerations in the NVIDIA DCGM Exporter chart values; do not add them to the Coralogix OpenTelemetry Integration values. #### AWS EKS For EKS, use a supported GPU node type and an EKS-optimized NVIDIA accelerated AMI. When the AMI already provides the NVIDIA driver and container toolkit, configure GPU Operator not to install either component again. See [EKS-optimized accelerated AMIs](https://docs.aws.amazon.com/eks/latest/userguide/ml-eks-optimized-ami.html). Warning Choose one GPU metric collector. Do not enable this scrape together with AWS CloudWatch Container Insights GPU monitoring, which deploys and manages DCGM collection itself. Running both creates duplicate metrics and ambiguous troubleshooting. ### Verify DCGM Exporter First verify the exporter before changing the Collector configuration. Replace the namespace and Service name with values from your installation. ``` kubectl -n port-forward service/ 9400:9400 curl -s http://localhost:9400/metrics | grep '^DCGM_FI_' | head ``` The command must return one or more `DCGM_FI_*` metrics. If it does not, fix the GPU driver or DCGM Exporter before configuring the Collector. ### Configure Target Allocator discovery Use the integration's Target Allocator with a Prometheus Operator `ServiceMonitor`. Target Allocator requires the Prometheus Operator `ServiceMonitor` CRD. Enable it in the values file used to install the Coralogix OpenTelemetry Integration: ``` opentelemetry-agent: targetAllocator: enabled: true allocationStrategy: per-node prometheusCR: enabled: true ``` The NVIDIA DCGM Exporter chart creates its own `ServiceMonitor` by default. Do not create a second `ServiceMonitor` for the same exporter. Configure its interval in the NVIDIA DCGM Exporter chart values file: ``` serviceMonitor: enabled: true ``` Target Allocator discovers this `ServiceMonitor`, resolves its endpoints, and assigns each target to one agent. With the `per-node` strategy, an exporter endpoint is assigned to the agent on the same node. ### Validate collection Confirm the Target Allocator was enabled in the rendered agent configuration: ``` helm template coralogix-charts-virtual/otel-integration \ --namespace \ --values values.yaml | grep -n 'targetAllocator' ``` Check the Target Allocator's assigned targets and then search Coralogix Metrics Explorer for `DCGM_FI_DEV_GPU_UTIL` or another metric returned by the exporter: ``` kubectl -n port-forward service/ 8080:8080 curl -s http://localhost:8080/jobs curl -s http://localhost:8080/scrape_configs ``` Each DCGM exporter endpoint must be assigned once. Inspect agent logs for scrape failures. ### Metrics and labels The integration preserves raw DCGM metric names and labels for compatibility. Start with the metrics DCGM Exporter exposes by default. High-value metrics include: * `DCGM_FI_DEV_GPU_UTIL` and `DCGM_FI_DEV_MEM_COPY_UTIL` * `DCGM_FI_DEV_FB_USED` and `DCGM_FI_DEV_FB_FREE` * `DCGM_FI_DEV_GPU_TEMP` and `DCGM_FI_DEV_POWER_USAGE` * `DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION` * `DCGM_FI_DEV_XID_ERRORS` Start with the exporter defaults. Kubernetes pod labels and pod UIDs are disabled by default; keep them disabled unless you need workload correlation. The following is an NVIDIA DCGM Exporter chart value, not a Coralogix OpenTelemetry Integration value. Add it at the root of the values file used to install or upgrade `gpu-helm-charts/dcgm-exporter`: ``` kubernetes: enablePodLabels: true enablePodUID: false podLabelAllowlistRegex: - '^app.kubernetes.io/name$' - '^environment$' ``` Keep `enablePodUID: false`: a pod UID creates a new series when a workload restarts. This guide does not enable process-level metrics or add a custom DCGM counter file. Review PID, process, and extra profiling dimensions before enabling them because they can significantly increase metric cardinality and ingestion cost. ## Collect CUDA workload metrics with OBI [OBI](https://coralogix.com/docs/opentelemetry/instrumentation-options/ebpf-auto-instrumentation/overview.md) (OpenTelemetry eBPF-based instrumentation) instruments the CUDA runtime API in the kernel, so you can see how applications use the GPU (kernel launches, device memory allocations, and host/device transfers), attributed to the pods and services that make the calls. It needs no exporter to scrape and no code changes: deploy the OBI eBPF agent, and CUDA metrics appear for GPU workloads on nodes where the NVIDIA toolkit is present. This path complements DCGM Exporter rather than replacing it. OBI does not report hardware health such as utilization, temperature, power, or framebuffer memory. Use DCGM Exporter for those. ### Enable OBI OBI ships with the same [Coralogix OpenTelemetry Integration Helm chart](https://coralogix.com/docs/opentelemetry/kubernetes-observability/kubernetes-complete-observability-basic-configuration.md). If you already run the integration, enable the eBPF agent in your values file: ``` opentelemetry-ebpf-instrumentation: enabled: true ``` For a first-time deployment (including the Coralogix Helm repository, the `coralogix-keys` secret, and verification steps), follow the [OBI getting-started guide](https://coralogix.com/docs/opentelemetry/instrumentation-options/ebpf-auto-instrumentation/getting-started.md). OBI runs as a DaemonSet, so it already covers every node, including GPU nodes. On GPU nodes where the NVIDIA toolkit is present, CUDA metrics are enabled automatically (`OTEL_EBPF_INSTRUMENT_CUDA=auto`, the default). To force them on or off regardless of toolkit detection, set the OBI `ebpf.instrument_cuda` option; see [Exported metrics](https://coralogix.com/docs/opentelemetry/instrumentation-options/ebpf-auto-instrumentation/exported-metrics.md#enabling-runtime-metrics). OBI attributes CUDA activity to the processes it instruments, so make sure your GPU workloads fall within its [service discovery](https://coralogix.com/docs/opentelemetry/instrumentation-options/ebpf-auto-instrumentation/service-discovery.md) scope. ### CUDA metrics OBI reports CUDA activity under the `gpu.cuda.*` namespace (Prometheus `gpu_cuda_*`). High-value metrics include: * `gpu.cuda.kernel.launch.calls` and `gpu.cuda.graph.launch.calls`: kernel and graph launches * `gpu.cuda.memory.allocations` and `gpu.cuda.memory.free.bytes`: device memory allocated and freed * `gpu.cuda.memory.copies`: host/device transfer sizes, broken down by `cuda.memcpy.kind` * `gpu.cuda.kernel.grid.size` and `gpu.cuda.kernel.block.size`: launch dimensions For the complete list (including stream, event, and synchronization counters, and the release each was introduced in), see [Exported metrics](https://coralogix.com/docs/opentelemetry/instrumentation-options/ebpf-auto-instrumentation/exported-metrics.md#application-metrics). ### Validate OBI collection Confirm the OBI pods are running: ``` kubectl get pods ``` After exercising a GPU workload, search Coralogix Metrics Explorer for `gpu_cuda_kernel_launch_calls_total` or another CUDA metric. If no series appear, confirm the NVIDIA toolkit is present on the node and that the workload falls within OBI's service-discovery scope.