Gang Scheduling for Multi-GPU AI Training on Kubernetes

A distributed training job cannot make useful progress with half of its workers. If an eight-worker PyTorch job receives four GPU pods, those four processes usually wait at rendezvous while holding four expensive GPUs. A second job can claim the remaining GPUs and wait for its own missing workers. The cluster is allocated, every job is blocked, and GPU utilization is close to zero.

Gang scheduling prevents this failure mode by treating a related set of pods as one scheduling unit. The scheduler starts the gang only when it can place the required number of pods. Otherwise, the entire job waits without occupying GPUs.

That atomic start is necessary, but production training needs more than a minAvailable field. The scheduler must account for GPU type, per-node shape, network topology, CPU and memory, queue quota, priorities, preemption, and capacity that may not exist until the cluster autoscaler creates it.

This guide explains those layers, shows practical Kueue and Volcano configurations, and provides a checklist for diagnosing a multi-GPU job that remains pending.

Why the default scheduler can strand GPUs

The default Kubernetes scheduler normally considers pods one at a time. Suppose two teams submit jobs with these requirements:

On a cluster with four free GPUs, Kubernetes might place two workers from each job. Both distributed jobs wait for all four ranks, but each holds half the cluster. Neither can finish and release capacity.

This is not fixed by setting parallelism: 4, a pod anti-affinity rule, or a high PriorityClass. Those settings describe replica count, placement, and ordering; they do not make admission atomic.

A gang scheduler introduces a workload-level decision:

Submit 8-worker job
        ↓
Can the required pod group fit?
   ├── no  → keep the whole job waiting
   └── yes → reserve/admit the group, then place all workers

For fixed-size synchronous training, set the minimum to the complete world size. A lower minimum is appropriate only when the training framework is explicitly elastic and can form a valid rendezvous with fewer workers.

Admission and placement are different problems

Gang scheduling is often used as a broad label for two separate decisions:

1. Workload admission asks whether quota is available for the complete job. Kueue specializes in this layer and keeps a job suspended until its full resource request can be admitted.
2. Pod placement asks whether those resources form a valid shape on actual nodes. Volcano performs gang-aware placement itself. Kueue can combine quota admission with Topology Aware Scheduling and the default scheduler.

The distinction matters. A cluster can have eight GPUs free in aggregate but still be unable to place two pods that each request four GPUs. The free devices may be fragmented one per node, may be the wrong model, or may live in different network domains.

Treat the following as part of the gang's real resource request:

• GPU count and resource name, such as nvidia.com/gpu
• GPU model, memory size, MIG profile, or other accelerator flavor
• CPU, RAM, ephemeral storage, and pod overhead
• node selectors, taints, tolerations, affinity, and topology constraints
• persistent-volume zone constraints
• host ports and any extended network resources

If any one of these cannot fit, adding nominal GPU quota will not make the job schedulable.

Choose a scheduling approach

There are three credible approaches for a new cluster.

Do not install Kueue and Volcano and let both manage the same job accidentally. Establish one owner for admission, one scheduler for placement, and a tested integration path for the workload controller you use.

Admit the complete job with Kueue

Kueue adds queueing and quota management without replacing kube-scheduler. It observes a supported job, builds a workload containing its pod sets, reserves quota for the complete workload, and then unsuspends the job. Unless partial admission is deliberately enabled, it does not admit only part of the workload.

The following example defines one GPU flavor and a cluster queue with quota for 16 GPUs. Labels on your ResourceFlavor must correspond to labels on real nodes. Change the API versions only after checking the Kueue version installed in your cluster.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: h100
spec:
  nodeLabels:
    accelerator: nvidia-h100
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: gpu-training
spec:
  namespaceSelector: {}
  resourceGroups:
    - coveredResources:
        - cpu
        - memory
        - nvidia.com/gpu
      flavors:
        - name: h100
          resources:
            - name: cpu
              nominalQuota: 192
            - name: memory
              nominalQuota: 1Ti
            - name: nvidia.com/gpu
              nominalQuota: 16
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: training
  namespace: ml-team
spec:
  clusterQueue: gpu-training

In a shared cluster, replace the open namespaceSelector with an intentional tenant policy. Also define CPU and memory quota: a workload whose GPU quota fits can still fail admission because an unbounded companion resource does not.

Submit a fixed-size indexed Job to the local queue:

apiVersion: batch/v1
kind: Job
metadata:
  name: torchrun-8x1
  namespace: ml-team
  labels:
    kueue.x-k8s.io/queue-name: training
spec:
  suspend: true
  completionMode: Indexed
  completions: 8
  parallelism: 8
  backoffLimit: 2
  template:
    metadata:
      labels:
        app: torchrun-8x1
    spec:
      restartPolicy: Never
      nodeSelector:
        accelerator: nvidia-h100
      containers:
        - name: trainer
          image: registry.example.com/ml/trainer@sha256:<digest>
          command: ["/opt/train/indexed-job-entrypoint.sh"]
          env:
            - name: JOB_COMPLETION_INDEX
              valueFrom:
                fieldRef:
                  fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']
          resources:
            requests:
              cpu: "12"
              memory: 64Gi
              nvidia.com/gpu: "1"
            limits:
              cpu: "12"
              memory: 64Gi
              nvidia.com/gpu: "1"

The entrypoint must translate JOB_COMPLETION_INDEX into a rank and configure a reachable rendezvous endpoint; those details are framework- and network-specific. Create the required Service or discovery mechanism before treating this as a runnable training manifest. Pin the container by digest and make every worker request the same tested resources.

Kueue should remove spec.suspend only after reserving quota for all eight pod replicas. Verify that transition rather than assuming a pending pod means Kueue is broken:

kubectl -n ml-team get job torchrun-8x1 \
  -o jsonpath='{.spec.suspend}{"\n"}'

kubectl -n ml-team get workloads
kubectl -n ml-team describe workload
kubectl get clusterqueue gpu-training -o yaml

### Make admission topology-aware

Quota alone answers “are eight H100s available to this queue?” It does not necessarily answer “can eight H100s be placed in the required rack or node-pool shape?” Kueue's Topology Aware Scheduling (TAS) evaluates topology domains before admission and is the preferred companion for tightly coupled multi-node jobs.

Model topology with stable node labels such as block, rack, or hostname, then configure a topology-aware resource flavor and request the appropriate placement policy on the job's pod set. Exact TAS annotations and fields have evolved across Kueue releases, so use the documentation for the version you have installed instead of copying an unversioned annotation into production.

Use required placement when crossing the selected topology boundary would make the job invalid. Use preferred placement when locality improves collective performance but the job may run across domains. A hard rack requirement can improve NCCL performance while also increasing queue time, so measure both.

Option 2: schedule a gang with Volcano

Volcano replaces pod-by-pod placement for opted-in workloads. Its VolcanoJob describes tasks, creates a PodGroup, and waits until the group's minimum resource demand can be satisfied.

This example runs four workers with two GPUs each. Because minAvailable equals the total replica count, all four worker pods form the minimum gang.

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: trainer-4x2
  namespace: ml-team
spec:
  schedulerName: volcano
  queue: default
  minAvailable: 4
  maxRetry: 2
  policies:
    - event: PodEvicted
      action: RestartJob
  tasks:
    - name: worker
      replicas: 4
      template:
        metadata:
          labels:
            app: trainer-4x2
        spec:
          restartPolicy: Never
          nodeSelector:
            accelerator: nvidia-h100
          containers:
            - name: trainer
              image: registry.example.com/ml/trainer@sha256:<digest>
              command: ["/opt/train/entrypoint.sh"]
              resources:
                requests:
                  cpu: "24"
                  memory: 128Gi
                  nvidia.com/gpu: "2"
                limits:
                  cpu: "24"
                  memory: 128Gi
                  nvidia.com/gpu: "2"

Volcano automatically creates a PodGroup for a VolcanoJob. minAvailable counts pods, not GPUs: four pods requesting two GPUs each require eight GPUs. Include the job's CPU and memory in capacity planning, because Volcano evaluates the complete pod request.

Inspect the job, its group, and the queue:

kubectl -n ml-team get vcjob,podgroup,pods
kubectl -n ml-team describe vcjob trainer-4x2
kubectl -n ml-team describe podgroup
kubectl get queue default -o yaml

A PodGroup condition such as Unschedulable or NotEnoughResources is more useful than looking only at individual pod events. Confirm that every generated pod has schedulerName: volcano; a missing scheduler name can silently split one logical workload across scheduling systems.

Volcano also supports queues, queue guarantees, priorities, reclaim, and preemption. Configure those policies deliberately. Gang scheduling stops partial starts, but it does not by itself decide which team's 64-GPU job should run first.

Native Kubernetes gang scheduling

Kubernetes 1.37 marks gang scheduling Beta, but the feature is disabled by default. It uses the scheduling.k8s.io/v1beta1 PodGroup API and the GenericWorkload and gang-scheduling configuration in the control plane. The scheduler waits for at least minCount group members and makes an atomic placement decision for the group.

This path is promising when you control the API server, scheduler configuration, and workload-controller integration. Before adopting it, verify all of the following:

• the exact Kubernetes minor version on every control-plane component
• whether the managed Kubernetes provider exposes the required feature gates and API group
• whether your Job or training controller creates and links PodGroups correctly
• how queueing, quota, topology, preemption, and autoscaling will be provided

Beta does not mean enabled, and the native mechanism does not automatically replace the multi-tenant queue policies supplied by Kueue or Volcano.

Get multi-GPU placement right

Atomic admission prevents deadlock; good placement prevents a slow job.

### Prefer the right pod shape

If each node has eight GPUs, choose between one eight-GPU pod per node and eight one-GPU pods intentionally.

• A single multi-GPU pod makes node-local GPU allocation explicit and reduces inter-pod coordination, but requests a large indivisible slot.
• Multiple one-GPU pods can offer finer failure and allocation granularity, but require affinity or topology controls if they must share a node.
• For multi-node training, the number of pods, GPUs per pod, and framework world size must agree.

Do not infer that nvidia.com/gpu: 8 guarantees NVLink connectivity. Kubernetes allocates devices; the NVIDIA device plugin, GPU Operator configuration, node hardware, and topology manager determine the available device topology. Validate placement inside the container with nvidia-smi topo -m and run an NCCL test representative of the training pattern.

### Separate accelerator flavors

Do not put A100, H100, H200, and arbitrary MIG profiles behind one interchangeable quota unless the workload genuinely treats them as equivalent. Label nodes consistently and model each acceptable resource flavor. Mixed GPU generations can create stragglers, change memory limits, and invalidate performance expectations.

For MIG, the requested resource name represents a slice profile rather than a full GPU. The gang size and quota must use that exact extended resource, and the training framework must tolerate the profile's compute, memory, and interconnect characteristics.

### Align CPU, memory, storage, and networking

An H100 worker with too little host memory or too few data-loader CPUs wastes the accelerator even when scheduling is correct. Set realistic requests for every constrained resource. Keep input data close enough to avoid a storage bottleneck, and account for RDMA or secondary-network resources if pods request them.

Network locality matters most during frequent all-reduce operations. Use hard topology requirements only for boundaries that are truly mandatory; use preferences where a cross-rack run is slower but still useful.

Priorities and preemption without thrashing

Large gangs are vulnerable to starvation because small jobs continually fit around them. Queues, fair sharing, and borrowing policies should give large jobs a path to accumulate enough capacity.

Priority can shorten that wait, but pod-level preemption is dangerous if it evicts a few low-priority pods without creating a feasible shape for the whole gang. Prefer workload-aware preemption that verifies the complete job can fit after the selected victims are removed.

Use these guardrails:

• define a small number of meaningful priority classes
• reserve the highest classes for real operational needs
• checkpoint before a job becomes eligible for preemption
• set disruption budgets only after understanding how they interact with batch eviction
• measure useful training time lost to eviction and restart, not just scheduler success

For synchronous training, losing one worker often invalidates the entire run. A controller policy that restarts the full job from the latest checkpoint is usually safer than allowing the remaining ranks to hang.

Make gang scheduling work with cluster autoscaling

A pending gang often represents demand for nodes that do not exist yet. The scheduler and autoscaler must expose and interpret that demand without creating a circular wait.

Test this sequence in a staging cluster:

1. Submit a gang larger than current free capacity but smaller than the node pool's maximum.
2. Confirm the workload remains unadmitted or unscheduled without binding a partial gang.
3. Confirm the autoscaler sees actionable unschedulable demand and selects the correct GPU node group.
4. Confirm new nodes register with the expected GPU, topology, taint, and driver labels.
5. Confirm the complete job starts after the device plugin advertises allocatable GPUs.
6. Confirm empty nodes scale down after the job and its pods are deleted.

Quota is not physical capacity. A Kueue ClusterQueue may admit 32 GPUs even when a cloud quota, node-pool maximum, stockout, or provisioning failure prevents those nodes from appearing. Use admission checks or provisioning integrations when your platform supports them, and alert separately on prolonged admitted-but-pending workloads.

Troubleshoot a gang that stays pending

Diagnose from the outside in: controller, queue, group, pod, node, then cloud capacity.

### 1. Is the workload admitted?

For Kueue, inspect Workload conditions and ClusterQueue usage. If the Job is still suspended, individual pod scheduling is not yet the problem.

kubectl -n ml-team get workloads -o wide
kubectl -n ml-team describe workload <workload-name>
kubectl get clusterqueue gpu-training -o yaml

Look for exhausted flavor quota, an inactive LocalQueue, pending admission checks, or a namespace that the ClusterQueue does not select.

### 2. Is the gang definition complete?

For Volcano, compare minAvailable with the total tasks that must start. For native PodGroups, compare minCount with the number of linked pods. If the controller creates fewer members than the minimum, the group correctly waits forever.

kubectl -n ml-team get podgroup -o yaml
kubectl -n ml-team get pods -o custom-columns=NAME:.metadata.name,SCHEDULER:.spec.schedulerName,PHASE:.status.phase

### 3. Can the requested shape fit any nodes?

Inspect one worker's events and compare its requests with node allocatable capacity:

kubectl -n ml-team describe pod <worker-pod>
kubectl get nodes \
  -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory

Common messages include insufficient nvidia.com/gpu, unmatched node affinity, untolerated taints, volume node-affinity conflicts, and insufficient CPU or memory. Check the per-node request. A 4-GPU pod cannot fit four nodes with one free GPU each.

### 4. Are GPUs advertised correctly?

Compare physical devices with Kubernetes allocatable resources. A node may be Ready before the NVIDIA device plugin is ready.

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'
kubectl -n gpu-operator get pods -o wide

If you use MIG, inspect the actual extended resource names rather than looking only for nvidia.com/gpu.

### 5. Did autoscaling fail?

Inspect autoscaler events and cloud-provider limits. The usual causes are a node-group maximum, regional GPU shortage, account quota, an unsupported pod shape, or a scheduling constraint that no configured node group can satisfy.

### 6. Did all pods start but training still hang?

That is now a distributed-runtime problem. Check rank assignment, world size, rendezvous DNS, NetworkPolicy, NCCL interface selection, shared secrets, clock skew, and whether one worker was OOM-killed. Gang scheduling guarantees initial placement, not successful collective initialization or continued quorum after eviction.

What to monitor

Build workload-level views rather than relying only on GPU utilization.

Alert on duration, not merely state. A large job waiting five minutes may be normal; an admitted job holding 64 GPUs while zero workers train for 20 minutes is expensive and actionable.

Production checklist

Before enabling gang scheduling for production training:

• Make the minimum group size equal the fixed distributed world size.
• Verify resource requests describe the full worker, not only its GPUs.
• Model GPU types and MIG profiles as explicit flavors.
• Test topology constraints with the actual node and network layout.
• Give one system clear ownership of admission and placement.
• Configure queue quotas, fair sharing, priorities, and borrowing deliberately.
• Test workload-level preemption and checkpoint recovery.
• Prove that a pending gang triggers the correct GPU node pool to scale.
• Set deadlines for workloads that cannot be provisioned.
• Monitor admitted-but-not-running GPUs and worker start skew.
• Pin images and pre-pull large training images where startup time matters.
• Run failure drills for node loss, device-plugin delay, and regional capacity shortage.

The practical rule

Gang scheduling should make an incomplete training job cheap to wait and a complete job fast to start. If pending workers hold GPUs, admission is too late. If a fully admitted job cannot fit the node topology, admission is too optimistic. If high-priority jobs repeatedly destroy lower-priority progress, preemption is too granular.

Start with a fixed-size job whose complete resource shape is well understood. Add topology-aware placement, queue policy, autoscaling, and workload-aware preemption one layer at a time. The result is not just fewer pending pods: it is higher useful GPU utilization, predictable queueing, and distributed training jobs that start only when they can actually run.

References

• Kubernetes gang scheduling
• Kueue all-or-nothing scheduling
• Kueue documentation
• Volcano PodGroup
• VolcanoJob
• JobSet overview