Can You Run A100, H100 and H200 in the Same Kubernetes Cluster?

Yes. A single Kubernetes cluster can contain A100, H100, and H200 worker nodes at the same time. Kubernetes does not require every node to have identical hardware, and NVIDIA's Kubernetes components can discover and expose different GPU models across the cluster.

But there is an important distinction: the cluster can run all three GPU types, while an individual workload may not be able to use them interchangeably. Kubernetes normally exposes a full NVIDIA GPU through the generic nvidia.com/gpu resource. Without additional scheduling rules, a pod asking for one GPU may land on any compatible GPU node—including the wrong generation for its performance, memory, or cost requirements.

The reliable design is to treat A100, H100, and H200 as separate capacity classes inside one cluster.

What a mixed-GPU cluster looks like

This approach preserves one operational surface for Kubernetes while keeping hardware placement explicit.

Why nvidia.com/gpu: 1 is not enough

After the NVIDIA device plugin is installed, a full A100, H100, and H200 are typically advertised to Kubernetes using the same extended resource name:

resources:
  limits:
    nvidia.com/gpu: 1

That request means "give this container one NVIDIA GPU." It does not mean "give it an H100," "give it at least 100GB of GPU memory," or "choose the cheapest accelerator that meets the SLA."

The default Kubernetes scheduler also does not understand GPU generation, HBM capacity, Tensor Core features, NVLink topology, expected throughput, or provider price. Those requirements must be translated into labels and scheduling policy.

Step 1: Create a node pool for each GPU class

Keep each accelerator type in a dedicated node pool. For example:

• gpu-a100-80gb for mature inference, development, and cost-sensitive training.
• gpu-h100-80gb for high-throughput training and production inference.
• gpu-h200-141gb for models limited by memory capacity or bandwidth.

Separate pools make upgrades, maintenance, autoscaling, quotas, and cost attribution much easier. They also prevent a change intended for one GPU generation—such as a MIG geometry or machine image—from affecting every GPU node.

Include more than the marketing model in the pool definition. An H100 PCIe node and an H100 SXM node have different power envelopes and interconnect characteristics. Useful capacity-class dimensions include:

• GPU model and memory capacity.
• PCIe or SXM form factor.
• Number of GPUs per node.
• NVLink or NVSwitch topology.
• CPU architecture and core count.
• Host memory, local storage, and NIC bandwidth.
• Cloud zone, reservation, and purchasing model.

Step 2: Label GPU nodes consistently

NVIDIA GPU Feature Discovery can create labels describing the installed GPU. Managed Kubernetes platforms may add their own labels as well. Inspect the actual labels before depending on them:

kubectl get nodes --show-labels
kubectl describe node <gpu-node-name>

Provider and operator label names can change, and automatically discovered product strings may include spaces, memory variants, or MIG state. Many teams therefore add a small, stable label vocabulary of their own:

kubectl label node <a100-node> gpufleet.io/gpu-class=a100-80gb
kubectl label node <h100-node> gpufleet.io/gpu-class=h100-80gb
kubectl label node <h200-node> gpufleet.io/gpu-class=h200-141gb

In production, apply these labels through node-pool bootstrap configuration or infrastructure as code. Manually labeling autoscaled nodes will eventually fail because replacement nodes will not inherit the label.

Step 3: Pin workloads to the correct class

For a strict hardware requirement, use a node selector:

apiVersion: v1
kind: Pod
metadata:
  name: h200-inference
spec:
  nodeSelector:
    gpufleet.io/gpu-class: h200-141gb
  containers:
    - name: model-server
      image: example/model-server:1.0
      resources:
        limits:
          nvidia.com/gpu: 1

Use required node affinity when the workload can run on a set of acceptable classes. This example permits either Hopper GPU but excludes A100:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: gpufleet.io/gpu-class
              operator: In
              values:
                - h100-80gb
                - h200-141gb

Preferred affinity is useful for an optimization rather than a requirement—for example, prefer A100 for cost but allow H100 when the A100 queue is full. Use that carefully: fallback to a more expensive GPU can make an availability problem look solved while quietly increasing spend.

Step 4: Protect GPU nodes with taints

Label-based placement gets GPU workloads onto the right nodes. Taints keep unrelated workloads off expensive GPU capacity.

kubectl taint nodes <gpu-node> nvidia.com/gpu=true:NoSchedule

GPU workloads then need a matching toleration:

tolerations:
  - key: nvidia.com/gpu
    operator: Equal
    value: "true"
    effect: NoSchedule

You can also taint by capacity class when access must be restricted more precisely. Remember that a toleration only makes a node eligible; it does not select that node. Pair tolerations with affinity or a selector.

Driver and CUDA compatibility

The container's CUDA user-space libraries, the host NVIDIA driver, the device plugin, and the GPU model all participate in compatibility. A mixed cluster needs a driver branch that supports the newest GPU in the fleet—usually the H200—while still supporting A100 and H100.

That does not mean every node must always run an identical driver. The GPU Operator can target different node groups with different policies, and immutable provider images may have their own driver lifecycle. However, one validated branch across all three pools is simpler when your operating environment allows it.

Before rollout:

1. Verify the chosen driver supports A100, H100, and H200.
2. Verify the CUDA version expected by every important container image.
3. Test framework versions such as PyTorch, TensorFlow, TensorRT-LLM, or vLLM on each eligible GPU class.
4. Roll out one node pool at a time and keep a rollback path.
5. Run an actual CUDA workload, not only nvidia-smi, as the health check.

A container built for Ampere can often execute on Hopper when its software stack is compatible, but it will not automatically benefit from Hopper features such as FP8. Conversely, a binary compiled only for newer GPU architectures may fail on A100. Build images with the required architectures or publish separate, clearly scheduled images.

MIG makes resource names model-specific

A100, H100, and H200 all support Multi-Instance GPU, but their available profiles and memory sizes are not identical. With the GPU Operator's mixed MIG strategy, Kubernetes exposes profile-specific resources such as nvidia.com/mig-1g.10gb rather than only nvidia.com/gpu.

This has two consequences:

• A deployment requesting one MIG profile cannot silently fall back to a different profile.
• The same logical service may need different resource manifests for different GPU models.

Maintain stable MIG node pools for the partition geometries you actually use. Avoid changing MIG mode or geometry just before scheduling a job: reconfiguration is disruptive and may require workloads to be drained or a node to reboot.

Do not mix full-GPU and MIG assumptions in one deployment. A pod requesting nvidia.com/gpu: 1 is not equivalent to a pod requesting a MIG resource, even if both run the same container image.

Can one distributed job mix A100, H100, and H200?

It may be technically possible for some frameworks, but it is usually a poor design for tightly coupled distributed training. Synchronous workers tend to progress at the speed of the slowest rank. Different memory capacities can force a common batch size, and different compute performance can create stragglers. Cross-node networking and collective communication add more variance.

For distributed training, schedule all workers onto a homogeneous pool with consistent GPU count, form factor, interconnect, NIC, and driver stack. Use pod affinity or a training operator to keep the job within the intended topology.

Mixed hardware is more reasonable when work is independent:

• Separate batch jobs consuming a common queue.
• Independent inference replicas behind a load balancer.
• Data preprocessing and evaluation stages with different requirements.
• Different teams or namespaces using different node pools.

Even for inference replicas, measure latency and throughput by GPU type. A load balancer that distributes requests evenly across unequal replicas can create inconsistent tail latency. Weighted routing or separate deployments may be more predictable.

Autoscaling a heterogeneous GPU cluster

Each GPU pool should scale independently. A pending H200 pod must contain enough scheduling information for the cluster autoscaler to know that only the H200 pool can satisfy it. If the pod requests a generic GPU without model affinity, the autoscaler may choose an unintended pool or be unable to make the right cost decision.

GPU nodes also take longer to become useful than ordinary CPU nodes. Provisioning, driver initialization, device-plugin registration, model download, and cache warm-up all contribute to startup time. Plan for this with:

• Minimum warm capacity for latency-sensitive services.
• Longer scale-up expectations than CPU workloads.
• Priority classes for critical GPU pods.
• Pod disruption budgets for inference replicas.
• Controlled scale-down so active training jobs are not interrupted.
• Separate autoscaling limits and budgets for each GPU class.

For queued batch workloads, queue-aware systems such as Kueue can provide better admission control than allowing every job to create unschedulable pods immediately.

Quotas and cost control

Standard Kubernetes quotas can limit named extended resources, including full GPUs and specific MIG profiles. But a quota on nvidia.com/gpu does not distinguish an A100 from an H200 when both use that resource name.

Use namespaces, admission policy, node affinity, and your platform's chargeback labels together. Useful controls include:

• Restricting which namespaces may tolerate premium GPU pools.
• Requiring a GPU class label or affinity rule on every GPU workload.
• Rejecting containers that request a GPU without CPU and memory requests.
• Applying priority and queue quotas by team.
• Reporting cost by node pool, namespace, deployment, and job.

An admission controller such as Kyverno, Gatekeeper, or ValidatingAdmissionPolicy can enforce these rules before a workload reaches the scheduler.

Observability and troubleshooting

Deploy DCGM Exporter through the GPU Operator or your monitoring stack and preserve model and node-pool labels in the metrics pipeline. At minimum, monitor:

• GPU utilization and framebuffer memory use.
• Power draw, temperature, throttling, and hardware errors.
• Per-pod and per-MIG-instance utilization.
• Pending time by requested GPU class.
• Node provisioning and GPU initialization time.
• Application throughput, latency, and cost per useful unit.

If a pod remains pending, check the problem in this order:

1. Does its GPU resource name exist on any node?
2. Does its node selector or required affinity match a node label exactly?
3. Does it tolerate the node's taints?
4. Is sufficient full-GPU or MIG capacity allocatable?
5. Are topology, zone, volume, or pod-affinity rules eliminating the remaining nodes?
6. Can the autoscaler create a matching node, and is that pool below its maximum size?

kubectl describe pod <pod-name> usually shows the scheduler's rejection reasons and is the best place to start.

A safe rollout plan

1. Create one node pool per GPU capacity class.
2. Install or configure the GPU Operator and discovery components.
3. Validate a driver and container matrix across all three GPUs.
4. Add stable labels and taints through infrastructure automation.
5. Define full-GPU and MIG pools separately.
6. Add explicit scheduling rules to every GPU workload.
7. Configure independent autoscaling limits and warm capacity.
8. Add admission policies, quotas, monitoring, and cost attribution.
9. Test node replacement, driver upgrades, MIG recovery, and scale-from-zero behavior.
10. Benchmark each workload and narrow its eligible GPU classes accordingly.

The bottom line

Running A100, H100, and H200 nodes in one Kubernetes cluster is fully reasonable and often desirable. It lets teams keep cost-effective A100 capacity, use H100 as a high-performance general tier, and reserve H200 for memory-intensive workloads.

The key is to expose them as distinct capacity classes, not as a pile of generic GPUs. Separate node pools, explicit placement, compatible drivers, stable MIG profiles, independent autoscaling, and model-aware monitoring turn mixed hardware into a useful fleet instead of a scheduling lottery.

Related reading: A100 vs H100 vs H200 for Kubernetes Workloads · MIG vs Time-Slicing vs Full GPU · GPU Observability

Planning a heterogeneous GPU cluster? Talk to us about GPU node-pool design, scheduling policy, MIG, autoscaling, and observability on your existing provider.