GPU Operator Explained: How NVIDIA GPUs Work in Kubernetes

A GPU server can join a Kubernetes cluster without being ready to run GPU workloads. The hardware needs a working driver, containers need access to the device, and Kubernetes needs a way to discover and allocate it.

The NVIDIA GPU Operator manages the software that connects these layers. Understanding what each component does makes the platform easier to configure and troubleshoot.

What the GPU Operator manages

The Operator automates deployment and management of NVIDIA's GPU software stack. Its components have distinct responsibilities:

The Operator coordinates these components; application containers still provide their own frameworks and application dependencies. See the NVIDIA GPU Operator overview.

How a pod gets a GPU

This guide follows the conventional device-plugin path using the extended resource nvidia.com/gpu.

1. The device plugin discovers GPUs on a node and registers them with the kubelet.
2. Kubernetes advertises the node's GPU capacity and allocatable resources.
3. A pod declares a GPU limit alongside its CPU and memory requirements.
4. The scheduler selects a node that satisfies the pod's resource and placement constraints.
5. The kubelet and device plugin handle device allocation, and the configured container runtime makes the assigned GPU accessible to the container.

The device plugin runs as a DaemonSet on eligible nodes. The Operator manages its deployment, while the scheduler makes pod placement decisions. See the NVIDIA device plugin documentation.

A minimal GPU workload

The following pod requests one GPU resource and runs nvidia-smi to check device visibility. Use a CUDA image compatible with your installed driver; this tag is an example, not a requirement for every cluster.

apiVersion: v1
kind: Pod
metadata:
  name: gpu-check
spec:
  restartPolicy: Never
  containers:
    - name: gpu-check
      image: nvidia/cuda:12.6.3-base-ubuntu22.04
      command: ["nvidia-smi"]
      resources:
        requests:
          cpu: "1"
          memory: "1Gi"
        limits:
          cpu: "1"
          memory: "1Gi"
          nvidia.com/gpu: 1

Save it as gpu-check.yaml, then inspect the result:

kubectl apply -f gpu-check.yaml
kubectl logs gpu-check
kubectl describe pod gpu-check

Kubernetes allows a GPU limit without an explicit GPU request; the request defaults to the limit. If both are supplied, they must match. GPU resource quantities are integers: nvidia.com/gpu: 0.5 does not request half a GPU. Node selectors or affinity can constrain the GPU model when the cluster has mixed hardware. See Kubernetes GPU scheduling.

If the NVIDIA runtime is not the default, your configuration may require runtimeClassName: nvidia under spec, with a matching RuntimeClass installed. Dedicated GPU nodes may also require tolerations. A successful nvidia-smi check confirms device visibility; validate actual CUDA execution with your application or a CUDA test before declaring the workload ready.

Decide who manages the node software

Before installing the Operator, check the supported combination of operating system, Kubernetes release, container runtime, and GPU hardware. Establish whether the cloud provider, node image, or your platform team already manages the driver and toolkit.

For installations with preinstalled drivers, NVIDIA documents driver.enabled=false. When both drivers and the toolkit are already installed, its documented configuration also disables toolkit management with toolkit.enabled=false. Use the settings appropriate to your environment rather than installing duplicate managers.

The Operator also uses Node Feature Discovery. If the cluster already runs it, follow the installation guidance for disabling the bundled NFD deployment. Select and pin a supported chart version, then validate the deployment on a test node pool. See NVIDIA's installation guide.

Where MIG and time-slicing fit

GPU sharing changes what an advertised resource represents. With ordinary full-GPU allocation, one resource corresponds to one device. Sharing requires an explicit configuration.

MIG divides supported GPUs into hardware-isolated instances with dedicated compute and memory resources. Time-slicing advertises multiple shared allocations so workloads can access the same GPU, but it does not provide MIG's memory or fault isolation. A shared allocation does not guarantee a fixed fraction of throughput or GPU memory. See NVIDIA's GPU sharing guide.

Choose the model around workload needs. Our MIG configuration guide and GPU time-slicing guide explore those tradeoffs in more detail.

Troubleshoot one layer at a time

Start with pod events and the status of the GPU software components. Replace the namespace if you installed the Operator elsewhere.

kubectl get pods -n gpu-operator -o wide
kubectl get nodes
kubectl describe pod gpu-check
kubectl get events --sort-by=.metadata.creationTimestamp

Inspect the affected node with kubectl describe node <node-name> and compare its advertised resources with the workload request. Follow NVIDIA's troubleshooting guide for component-specific diagnostics.

Build an operating plan around the Operator

Treat installation as the beginning of platform validation. Run representative training or inference workloads, record their behavior, and decide how your team will evaluate upgrades and investigate failures.

For planning, pair infrastructure health with workload results: job completion time, inference latency, and useful throughput. Our GPU observability guide covers the monitoring side of that work.

If you are choosing GPUs or reviewing an existing cluster, GPU Consultancy can help you assess the architecture and define practical next steps. Talk to gpufleet about your workloads and infrastructure goals.