How Does DRA Work for GPUs in Kubernetes? A Hands-On Guide

Dynamic Resource Allocation, or DRA, gives Kubernetes a richer way to discover, select, configure, and allocate devices such as GPUs. Instead of asking only for an opaque extended resource such as nvidia.com/gpu: 1, a workload can ask for a device through a ResourceClaim and describe properties such as the GPU model, architecture, or memory capacity.

DRA does not create physical GPU capacity. A vendor DRA driver discovers the devices already attached to nodes, publishes their properties to Kubernetes, and prepares the selected device when a pod starts. Some drivers can also perform supported device reconfiguration, such as dynamic MIG partitioning, but that is a separate feature with its own requirements.

This guide uses the current NVIDIA GPU Operator-managed DRA workflow and the resource.k8s.io/v1 API. The commands target Kubernetes 1.34.2 or later, NVIDIA GPU Operator 26.7.0, and NVIDIA driver 580 or later. Check the current NVIDIA installation guide before using these version pins in production.

DRA compared with the traditional device plugin

The NVIDIA device plugin normally advertises an integer extended resource on each node. A pod requests it like this:

resources:
  limits:
    nvidia.com/gpu: 1

That interface is simple, but the resource request itself does not say which model, memory size, or device configuration the workload needs. Placement commonly depends on separate node labels and affinity rules.

DRA moves the device request into a claim:

resources:
  claims:
    - name: gpu

The claim can reference a device class and use Common Expression Language, or CEL, to filter the devices published by the driver.

With the current GPU Operator workflow, a cluster uses either a DRA-oriented GPUCluster resource or the device-plugin-oriented ClusterPolicy resource. Do not deploy both allocation paths onto the same fresh installation and assume they will coordinate.

The four Kubernetes objects behind DRA

The NVIDIA driver currently provides device classes including gpu.nvidia.com for full GPUs, mig.nvidia.com for MIG devices, and vfio.gpu.nvidia.com for VFIO passthrough.

A ResourceClaim is useful when you intentionally manage a claim and its lifetime. A ResourceClaimTemplate is usually the better fit for a Deployment or Job: Kubernetes creates a claim for each pod and deletes that generated claim when the pod is removed.

How allocation works from pod creation to GPU access

1. The NVIDIA DRA kubelet plugin discovers the GPUs on each node.
2. The driver publishes them in ResourceSlice objects with properties such as product name, architecture, UUID, and memory capacity.
3. A workload references a ResourceClaim or ResourceClaimTemplate.
4. The Kubernetes scheduler evaluates the claim against the published devices and finds a device and node that satisfy the complete pod.
5. The scheduler records the allocation in the claim and binds the pod to the selected node.
6. On that node, kubelet asks the NVIDIA DRA plugin to prepare the allocated device.
7. The plugin returns Container Device Interface, or CDI, information to kubelet.
8. The container runtime injects the GPU device, NVIDIA driver libraries, and related environment into the container.
9. When the last consuming pod is gone, kubelet asks the driver to unprepare the device and Kubernetes can allocate it again.

This coordination is why a DRA request can participate directly in scheduling. Kubernetes does not first choose an arbitrary node and then hope the requested GPU exists there.

Prerequisites and preflight checks

The current GPU Operator-managed GPU allocation workflow requires:

• Kubernetes 1.34.2 or later with the resource.k8s.io/v1 APIs.
• NVIDIA driver 580 or later.
• A container runtime with CDI support enabled.
• GPU Operator 26.7.0 or a compatible later version.
• A fresh DRA installation without an existing ClusterPolicy resource.

Check the Kubernetes server and DRA API resources:

kubectl version
kubectl api-resources --api-group=resource.k8s.io

The output should include deviceclasses, resourceclaims, resourceclaimtemplates, and resourceslices using resource.k8s.io/v1.

Check whether the device-plugin-based GPU Operator policy already exists:

kubectl get clusterpolicy

If a ClusterPolicy exists, do not convert that cluster in place using the commands below. NVIDIA documents the GPUCluster path as a new-installation workflow rather than an in-place migration from ClusterPolicy or an older standalone DRA release.

Confirm that Kubernetes can see your GPU nodes and that the nodes are ready:

kubectl get nodes -o wide
kubectl get nodes -l feature.node.kubernetes.io/pci-10de.present=true

The second label is commonly created by Node Feature Discovery. Its absence does not prove that a node has no GPU, so check the labels used in your environment.

Install the NVIDIA DRA driver through GPU Operator

Add and update the NVIDIA Helm repository:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

For an operator-managed NVIDIA driver, install GPU Operator with GPUCluster enabled and ClusterPolicy creation disabled:

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --version=v26.7.0 \
  --namespace gpu-operator \
  --create-namespace \
  --set clusterPolicy.deployCR=false \
  --set gpuCluster.deployCR=true \
  --set driver.nvidiaDriverCRD.enabled=true \
  --set draDriver.computeDomains.enabled=false

The last setting disables ComputeDomain components when the cluster only needs ordinary GPU allocation. ComputeDomains are intended for supported Multi-Node NVLink systems such as GB200 and GB300 NVL platforms.

If driver version 580 or later is already installed on the host, use the pre-installed-driver path instead:

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --version=v26.7.0 \
  --namespace gpu-operator \
  --create-namespace \
  --set clusterPolicy.deployCR=false \
  --set gpuCluster.deployCR=true \
  --set driver.enabled=false \
  --set draDriver.computeDomains.enabled=false

Do not install a separate dra-driver-nvidia-gpu Helm release alongside this managed workflow. GPU Operator creates and manages the DRA components through the singleton GPUCluster resource named gpu-cluster.

Validate the installation

Wait for the GPUCluster resource to report ready:

kubectl get gpucluster gpu-cluster -w

If it remains notReady, inspect its conditions and events:

kubectl describe gpucluster gpu-cluster

List the managed pods:

kubectl get pods -n gpu-operator -o wide

For basic GPU allocation, expect the DRA kubelet plugin, validator, and DCGM Exporter workloads to become ready.

Check the device classes installed by the driver:

kubectl get deviceclass
kubectl get deviceclass gpu.nvidia.com -o yaml

Check that GPU devices have been published:

kubectl get resourceslice
kubectl get resourceslice -o yaml

Each GPU node normally has a resource pool represented by one or more slices. If the list is empty, creating claims will not fix the problem: first verify that the kubelet plugin runs on the GPU node and can discover its devices.

To see a concise list of the NVIDIA slices and their nodes:

kubectl get resourceslice \
  -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,DRIVER:.spec.driver,POOL:.spec.pool.name'

Run a pod with any available full GPU

Create a namespace for the walkthrough:

kubectl create namespace gpu-example

Create a ResourceClaimTemplate and a pod in one file:

cat > single-gpu-pod.yaml <<'EOF'
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: single-gpu
  namespace: gpu-example
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.nvidia.com
---
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
  namespace: gpu-example
spec:
  restartPolicy: Never
  resourceClaims:
    - name: gpu
      resourceClaimTemplateName: single-gpu
  containers:
    - name: workload
      image: ubuntu:22.04
      command: ["bash", "-c"]
      args: ["nvidia-smi -L; sleep 3600"]
      resources:
        claims:
          - name: gpu
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
EOF

kubectl apply -f single-gpu-pod.yaml

The spec.resourceClaims entry attaches the generated claim to the pod. The container's resources.claims entry grants that container access to the claim. The names must match. No nvidia.com/gpu limit is needed. Although the example uses a plain Ubuntu image, the NVIDIA driver's CDI integration injects nvidia-smi and the host driver libraries when it prepares the container.

The toleration is useful when GPU nodes carry the common nvidia.com/gpu:NoSchedule taint. Remove or change it if your cluster uses another taint policy.

Watch the pod, claim, and events:

kubectl get pod,resourceclaim -n gpu-example -w
kubectl get events -n gpu-example --sort-by=.lastTimestamp

When the pod is running, verify the allocated GPU:

kubectl get pod -n gpu-example gpu-pod -o wide
kubectl exec -n gpu-example gpu-pod -- nvidia-smi -L

Inspect the generated claim and its allocation result:

kubectl get resourceclaim -n gpu-example
kubectl get resourceclaim -n gpu-example -o yaml

The generated claim name has a suffix. Its status records the selected driver, resource pool, device, and reservation state.

Select a GPU model with CEL

In a mixed A100, H100, and H200 cluster, first inspect the exact attributes published by your installed driver:

kubectl get resourceslice -o yaml

Then create a claim template that selects an H100 by its published productName:

cat > h100-gpu.yaml <<'EOF'
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: h100-gpu
  namespace: gpu-example
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.nvidia.com
            selectors:
              - cel:
                  expression: |
                    device.attributes['gpu.nvidia.com'].productName.lowerAscii().matches('^.*h100.*$')
EOF

kubectl apply -f h100-gpu.yaml

Reference h100-gpu from a pod by changing resourceClaimTemplateName in the earlier example. If no published device matches the expression, the pod remains pending. The scheduler does not silently substitute a different model.

Select by GPU memory instead of model

A memory requirement can be more portable than a product name. This template asks for a GPU with more than 40 GiB of advertised memory:

cat > large-memory-gpu.yaml <<'EOF'
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: large-memory-gpu
  namespace: gpu-example
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.nvidia.com
            selectors:
              - cel:
                  expression: |
                    device.capacity['gpu.nvidia.com'].memory.isGreaterThan(quantity("40Gi"))
EOF

kubectl apply -f large-memory-gpu.yaml

You can combine attributes and capacity. For example, the following selector requests a Hopper-architecture device with more than 80 GiB:

selectors:
  - cel:
      expression: |
        device.attributes['gpu.nvidia.com'].architecture == 'Hopper'
        &&
        device.capacity['gpu.nvidia.com'].memory.isGreaterThan(quantity("80Gi"))

Use only attribute names and values that appear in your ResourceSlice objects. These fields belong to the driver inventory and can evolve between driver releases.

Request two GPUs for one container

Set count: 2 in a single request when one container needs two matching full GPUs:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: two-gpus
  namespace: gpu-example
spec:
  spec:
    devices:
      requests:
        - name: gpus
          exactly:
            deviceClassName: gpu.nvidia.com
            count: 2

The pod references two-gpus once, and its container receives the matching claim:

spec:
  resourceClaims:
    - name: gpus
      resourceClaimTemplateName: two-gpus
  containers:
    - name: trainer
      image: ubuntu:22.04
      command: ["bash", "-c"]
      args: ["nvidia-smi -L; sleep 3600"]
      resources:
        claims:
          - name: gpus

For tightly coupled training, device count alone is not a topology guarantee. NVLink, NVSwitch, Fabric Manager partitions, network placement, and gang scheduling may impose additional constraints.

Use a ResourceClaimTemplate in a Deployment

Place the claim reference inside the Deployment's pod template. Kubernetes generates a distinct claim for every replica:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-workers
  namespace: gpu-example
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gpu-worker
  template:
    metadata:
      labels:
        app: gpu-worker
    spec:
      resourceClaims:
        - name: gpu
          resourceClaimTemplateName: single-gpu
      containers:
        - name: worker
          image: ubuntu:22.04
          command: ["bash", "-c"]
          args: ["nvidia-smi -L; sleep 3600"]
          resources:
            claims:
              - name: gpu
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

With two replicas, the controller creates two pods and Kubernetes creates a separate generated claim for each pod. Each claim therefore receives its own allocation unless the driver and claim configuration explicitly use a supported sharing mode.

Where MIG and GPU sharing fit

DRA is the allocation framework; it is not itself a sharing mechanism. NVIDIA's driver can expose or configure several resource types and sharing strategies:

• Use gpu.nvidia.com for a full GPU.
• Use mig.nvidia.com for a MIG device with hardware-isolated compute and memory.
• Use a driver-specific claim configuration for supported time-slicing or MPS behavior.
• Use DRA consumable capacity when enabled and supported to let independent claims consume tracked shares of a device.

Static MIG is available when MIG partitions already exist before the DRA GPU kubelet plugin starts. Dynamic MIG and consumable capacity rely on additional feature gates and should be treated as separate rollout projects. Inspect their current feature status and compatibility constraints before enabling them.

Troubleshoot a pending GPU pod

Start with the pod and its events:

kubectl describe pod -n gpu-example gpu-pod
kubectl get events -n gpu-example --sort-by=.lastTimestamp

Then check the claim:

kubectl get resourceclaim -n gpu-example
kubectl describe resourceclaim -n gpu-example <generated-claim-name>
kubectl get resourceclaim -n gpu-example <generated-claim-name> -o yaml

Check whether the requested class exists and whether the driver published any devices:

kubectl get deviceclass
kubectl get resourceslice
kubectl get resourceslice -o yaml

Inspect GPU Operator and DRA workloads:

kubectl get gpucluster gpu-cluster
kubectl describe gpucluster gpu-cluster
kubectl get pods,daemonsets -n gpu-operator -o wide
kubectl logs -n gpu-operator daemonset/nvidia-dra-driver-kubelet-plugin \
  --all-containers=true --tail=200

Common causes include:

• The requested DeviceClass does not exist.
• No ResourceSlice contains a matching device.
• A CEL selector uses an attribute or value the installed driver does not publish.
• Every matching GPU is already allocated.
• Node selectors, affinity, taints, volumes, or other pod requirements exclude the GPU node.
• The DRA kubelet plugin is absent or unhealthy on the selected node.
• CDI is not enabled or correctly configured in the container runtime.
• A ClusterPolicy from the traditional GPU Operator workflow conflicts with the GPUCluster installation.

If the pod schedules but the container cannot see a GPU, inspect the kubelet-plugin logs, container runtime CDI configuration, and pod events. If the pod never schedules, focus first on the claim allocation, ResourceSlices, and the scheduler's event messages.

Clean up the walkthrough

Delete the pod and claim templates:

kubectl delete pod -n gpu-example gpu-pod
kubectl delete resourceclaimtemplate -n gpu-example \
  single-gpu h100-gpu large-memory-gpu two-gpus --ignore-not-found
kubectl delete namespace gpu-example

Before uninstalling GPU Operator, delete GPU workloads and user-created claims, then verify that their devices have been unprepared. Use the normal Helm uninstall path and do not skip the Operator's pre-delete hooks while gpu-cluster exists.

What DRA changes operationally

DRA makes GPU requirements visible to the Kubernetes scheduler as device claims. It gives the scheduler structured inventory instead of only an integer count, makes model and capacity selection part of the workload contract, and lets a vendor driver prepare the chosen device at pod startup.

That additional control also introduces more moving parts. Cluster administrators must monitor the DRA driver, ResourceClaim controller behavior, scheduler capacity, claim cleanup, and published ResourceSlices. Access to DeviceClass and ResourceSlice objects should remain administrative, while workload teams receive namespace-scoped permission to create only the claims and templates they need.

For a production rollout, start with exclusive full-GPU claims, confirm scheduling and cleanup under node maintenance, and add selectors or sharing features only after the basic lifecycle is reliable.

Official references: Kubernetes Dynamic Resource Allocation · NVIDIA GPU Operator DRA installation · NVIDIA full-GPU claim examples · Kubernetes DRA administration guidance

Related reading: GPU Operator Explained · Can You Run A100, H100 and H200 in the Same Kubernetes Cluster? · MIG vs Time-Slicing vs Full GPU

Planning a DRA rollout for a GPU cluster? Talk to gpufleet about driver compatibility, scheduling policy, ResourceClaims, MIG, and observability.