DCGM vs vLLM Metrics: What Should You Monitor?

DCGM and vLLM metrics answer different questions about the same inference service.

NVIDIA DCGM Exporter tells you what the GPU hardware is doing: utilization, memory consumption, power, temperature, clocks, and device errors. vLLM tells you what the inference engine and its users are experiencing: token throughput, request queues, time to first token, inter-token latency, KV cache pressure, and preemptions.

You need both in production. DCGM alone cannot explain whether users are waiting. vLLM alone cannot tell you whether a GPU is throttling, unhealthy, or underutilized.

DCGM and vLLM at a glance

The distinction is not “infrastructure metrics versus useful metrics.” Both are useful. The important distinction is the layer each source observes.

What DCGM can tell you

NVIDIA DCGM Exporter exposes selected GPU telemetry in Prometheus format. In Kubernetes it commonly runs through NVIDIA GPU Operator, with one exporter pod on each GPU node.

### GPU utilization

avg by (node, gpu) (DCGM_FI_DEV_GPU_UTIL)

GPU utilization helps distinguish an accelerator that is busy from one that is idle. It is not a direct measure of useful LLM throughput. A GPU can report high utilization while token throughput falls because the workload shifted toward long prompts, the device is throttling, or a software change made execution less efficient.

Use a short interval to understand bursts and a longer interval for capacity trends. A long average can hide the burst-and-idle pattern of online inference.

### Physical GPU memory

100 * DCGM_FI_DEV_FB_USED
/
(DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)

Framebuffer memory includes model weights, KV cache, CUDA graphs, activations, allocator reserves, and other device allocations. It answers “how full is the device memory?” It does not answer “how full is vLLM's KV cache?”

### Power, temperature, and clocks

avg by (node, gpu) (DCGM_FI_DEV_POWER_USAGE)

max by (node, gpu) (DCGM_FI_DEV_GPU_TEMP)

Power, temperature, and clock metrics help explain why performance changed even when the request mix did not. They can reveal power caps, thermal constraints, or a GPU that behaves differently from its peers.

### Hardware health

DCGM can expose XID, ECC, clock-event, and health information, depending on the collector configuration, GPU, driver, DCGM version, and permissions. Not every configured field is emitted on every system.

Hardware health alerts should remain separate from serving SLO alerts. A device fault may require immediate node isolation even before users experience a visible latency increase.

What vLLM can tell you

The vLLM OpenAI-compatible server exposes production metrics at /metrics. The exact inventory depends on the version, so compare examples with the vLLM production metrics reference and your live endpoint.

### Token throughput

sum by (model_name) (
  rate(vllm:prompt_tokens_total[5m])
)

sum by (model_name) (
  rate(vllm:generation_tokens_total[5m])
)

Tokens are a better unit of LLM work than requests. Prompt tokens represent prefill work, while generation tokens represent decode work. Track them separately because their compute and memory behavior differs.

### User-visible latency

Time to first token, inter-token latency, and end-to-end latency describe different parts of the experience.

histogram_quantile(
  0.95,
  sum by (le, model_name) (
    rate(vllm:time_to_first_token_seconds_bucket[5m])
  )
)

histogram_quantile(
  0.95,
  sum by (le, model_name) (
    rate(vllm:inter_token_latency_seconds_bucket[5m])
  )
)

TTFT is critical for interactive responsiveness. Inter-token latency describes how smoothly output streams after the first token. End-to-end latency also depends strongly on requested output length, so it should not replace the phase-specific views.

### Scheduler demand

sum by (model_name) (vllm:num_requests_running)

sum by (model_name) (vllm:num_requests_waiting)

Running requests show active model execution. Waiting requests show scheduler pressure. A brief queue is not necessarily an incident, but a persistent queue combined with rising TTFT is a strong capacity signal.

### KV cache pressure

100 * max by (pod, model_name) (vllm:kv_cache_usage_perc)

vllm:kv_cache_usage_perc reports the used fraction of vLLM's KV block pool; 1 means 100 percent. It is not GPU memory utilization.

High KV cache usage is often efficient. It becomes concerning when it accompanies queues, latency degradation, or preemptions:

sum by (pod, model_name) (
  rate(vllm:num_preemptions_total[5m])
)

Metric names change over time. Use vllm:kv_cache_usage_perc rather than the deprecated vllm:gpu_cache_usage_perc, and vllm:inter_token_latency_seconds rather than the deprecated vllm:time_per_output_token_seconds, when supported by your deployed version.

GPU memory is not KV cache usage

This is the most common source of confusion when DCGM and vLLM appear on one dashboard.

The two percentages have different denominators. If vLLM reserves a large portion of GPU memory for KV cache, DCGM memory usage may remain high even when the KV cache usage gauge is low. That is expected: reserved or allocated physical memory and occupied logical KV blocks are not equivalent.

Use DCGM to answer whether allocations fit on the GPU. Use vLLM to answer whether active sequences are exhausting the cache capacity available to the scheduler.

How to diagnose common patterns

### High TTFT, high queue, high GPU utilization

The service is probably compute-saturated or underprovisioned for current demand. Confirm the change in prompt throughput and prompt lengths, then consider more replicas, a smaller or quantized model, additional parallelism, batching changes, or traffic shaping.

### High TTFT, high queue, low GPU utilization

Look beyond the accelerator. Possible causes include CPU throttling, tokenizer work, request routing, storage, network delays, an unhealthy replica, or synchronization overhead. Compare pods rather than relying only on a fleet average.

### High KV cache usage, no queue, stable latency

This can be a healthy, efficient service. Do not page solely because the cache is above an arbitrary percentage. Watch whether preemptions begin and whether waiting requests persist.

### High DCGM memory, low KV cache usage

The model, CUDA graphs, allocator reserves, or other runtime allocations may dominate memory. This is not evidence of a KV cache leak. Compare memory after model load, at idle, and under controlled concurrency.

### Falling tokens per second, high GPU utilization

Check whether the workload shifted from decode-heavy to prefill-heavy traffic, prompt or output lengths changed, prefix-cache effectiveness fell, power or clocks changed, or a rollout altered model configuration.

### Normal vLLM metrics, abnormal temperature or XID signals

Treat the hardware signal independently. Drain or isolate a suspect node according to your runbook instead of waiting for request latency to fail.

What should be on the dashboard?

A useful production dashboard starts with the serving SLO and descends toward the hardware.

Keep namespace, pod, model_name, node, and GPU identity available for drill-down. Avoid high-cardinality labels such as request IDs, user IDs, or raw prompts in Prometheus.

Which metrics should trigger alerts?

Alert thresholds should describe sustained impact or a condition that will soon cause impact.

Good vLLM alert candidates include:

• TTFT or ITL exceeding a defined SLO for a sustained period.
• A persistent scheduler queue combined with high KV cache pressure.
• A sustained preemption rate.
• Request failures or abnormal finish reasons.
• Missing vLLM scrape targets.

Good DCGM alert candidates include:

• Hardware health failure, XID, or uncorrectable ECC signals.
• Sustained thermal or power-related throttling.
• Abnormal temperature according to the GPU model and operating policy.
• Missing DCGM Exporter targets.
• A material performance outlier compared with equivalent peer GPUs.

GPU utilization by itself is rarely a good page. High utilization can mean excellent efficiency; low utilization can be normal during low demand. Combine it with serving signals before calling it a capacity incident.

Collection architecture on Kubernetes

Prometheus should scrape the two sources independently:

vLLM pods :8000/metrics ──────────┐
                                  ├── Prometheus ── Grafana and alerts
DCGM Exporter :9400/metrics ──────┘

Use separate Services and ServiceMonitor resources. This separation makes collection failures visible: an application scrape can remain healthy when GPU telemetry is missing, and vice versa.

DCGM Exporter can add Kubernetes workload labels when pod-resources mapping is enabled. Examine labels on the live metrics before attempting PromQL joins. Label availability differs by DCGM Exporter release and deployment mode.

For a complete setup with Kubernetes manifests, PromQL, alerts, and Grafana guidance, read Monitoring vLLM on Kubernetes: Tokens, Latency, KV Cache and GPU Metrics. For DCGM installation details, see How to Monitor NVIDIA GPUs on Kubernetes with DCGM Exporter, Prometheus & Grafana.

The practical answer

Use vLLM metrics to define and measure the serving SLO. Start with token throughput, TTFT, inter-token latency, waiting requests, queue time, KV cache usage, and preemptions.

Use DCGM metrics to explain accelerator capacity and health. Start with GPU utilization, framebuffer memory, power, temperature, clocks, and the hardware error signals supported by your environment.

Neither replaces the other. The fastest route to a correct diagnosis is usually a timeline that places user latency, scheduler pressure, KV cache usage, token throughput, and GPU telemetry next to one another.

References

• vLLM Production Metrics
• vLLM Prometheus and Grafana example
• NVIDIA DCGM Exporter metrics reference
• NVIDIA configuration guide for Prometheus and DCGM Exporter