supraj.dev

Interview questions & answers

AI on Kubernetes

GPU scheduling, model loading, queueing and inference routing on clusters.

  • 7 questions
  • 1 easy
  • 2 medium
  • 4 hard

7 questions

Question 1EasyGPU basics

How does a Pod get a GPU on Kubernetes?

#

You add GPU nodes to a cluster, but Pods requesting GPUs stay Pending.

Show answer

GPUs are advertised as extended resources by a device plugin running on each node, for example nvidia.com/gpu. A Pod requests them in its resource limits, and the scheduler places it on a node with free GPUs. Without the plugin, the scheduler sees no GPUs.

On NVIDIA hardware, the GPU Operator installs and manages the pieces: driver, container toolkit, device plugin, GPU feature discovery labels and the DCGM metrics exporter. Taint GPU nodes so only Pods that tolerate the taint and need GPUs land there.

Trade-offs

The operator simplifies setup but adds components to upgrade alongside the kernel and driver.

Common mistake

Leaving GPU nodes untainted so ordinary Pods fill them.

Follow-up

How would you check that a Pod can actually see its GPU?

Question 2MediumScheduling

What does Dynamic Resource Allocation add over device plugins?

#

Teams need specific GPU models, shared GPUs and GPUs on the same interconnect.

Show answer

A device plugin exposes an integer count such as "4 GPUs", so you cannot easily ask for a particular model, memory size or topology. Dynamic Resource Allocation, stable since Kubernetes 1.34 with the resource.k8s.io/v1 API, lets drivers publish devices with attributes.

Workloads create ResourceClaims, often from a ResourceClaimTemplate, that select devices from a DeviceClass using CEL expressions on those attributes. The scheduler understands the claims, claims can be shared between Pods, and drivers can report status. It needs a DRA driver for your hardware.

Trade-offs

DRA is far more expressive but newer; check that your GPU vendor's driver and your managed platform support it.

Common mistake

Mixing DRA and the classic device plugin on the same devices without understanding which manages them.

Follow-up

How would you request two GPUs that share a fast interconnect?

Question 3MediumStartup

A 70B model server takes fifteen minutes to become ready. How do you shorten it?

#

Weights are baked into a 140 GB container image pulled on every new node.

Show answer

Measure each stage: image pull, weight download, loading into GPU memory, and warm-up. Then attack the biggest. Keep the serving image small and store weights separately: on a node-local cache, a fast shared volume, or as an OCI artifact mounted with Kubernetes image volumes, which reuse registry tooling for caching and versioning.

Pre-pull images and weights onto GPU nodes, use fast weight formats such as safetensors, and size the startup probe for real load times. For traffic spikes, keep a warm replica rather than relying on cold starts.

Trade-offs

Node-local caches start fast but must be filled and kept in sync with model versions.

Common mistake

A liveness probe that kills the Pod while it is still loading weights.

Follow-up

How would you roll out a new model version without cold-starting every replica?

Question 4HardMulti-tenancy

Ten teams share 32 GPUs. How do you stop one team from hogging them while keeping GPUs busy?

#

One team submits 200 training Jobs on Friday; everyone else waits until Monday.

Show answer

A ResourceQuota on GPU requests caps each namespace but leaves capacity idle when a team is quiet. Kueue adds job queueing on top: each team gets a ClusterQueue with a nominal GPU quota, queues in a cohort can borrow unused quota from each other, and preemption reclaims borrowed capacity when the owner needs it.

Kueue admits whole jobs, so a job starts only when all its resources fit. Add priority classes for urgent work and dashboards showing usage per team so the quotas can be adjusted.

Trade-offs

Borrowing raises utilisation, but preempted jobs lose work unless they checkpoint.

Common mistake

Hard per-team limits with no borrowing, leaving GPUs idle while others wait.

Follow-up

How would you protect a long training run from preemption?

Question 5HardTraffic

Why is round-robin load balancing a poor fit for LLM servers, and what replaces it?

#

One replica is overloaded with long prompts while another sits half idle.

Show answer

Requests vary from a few tokens to thousands, so equal request counts mean very unequal load. Each replica also has its own KV cache: sending a request to a replica that already holds its prompt prefix saves work.

The Gateway API Inference Extension adds model-aware routing to Gateway API implementations. An InferencePool groups model-server Pods, and an endpoint picker chooses a Pod using live metrics such as queue length, KV cache use and loaded LoRA adapters. It can also prioritise more critical traffic.

Trade-offs

Smarter routing improves latency but depends on accurate, fresh metrics from every server.

Common mistake

Using session affinity as a substitute, which overloads replicas holding busy sessions.

Follow-up

How would you roll out a new model version through an inference gateway?

Question 6HardParallelism

How do you serve a model that does not fit on one GPU on Kubernetes?

#

A model's weights and KV cache need more memory than one GPU, and possibly more than one node.

Show answer

Within one node, use tensor parallelism: the serving engine splits each layer across the node's GPUs, which need fast interconnects. If one node is not enough, add pipeline parallelism across nodes, accepting more network traffic and latency. Quantization may let it fit on fewer GPUs, so test that first.

On Kubernetes, a multi-node replica is a group of Pods that must start, scale and fail together. The LeaderWorkerSet API models that: one leader and its workers as a single replica, with all-or-nothing behaviour. Schedule the group on nodes with the right topology, and treat the whole group as unhealthy when any member fails.

Trade-offs

Splitting across nodes makes huge models possible but adds network cost and more ways to fail.

Common mistake

Running the parts of one model as independent Deployments that can restart separately.

Follow-up

How would you roll out a new version of a multi-node model without dropping traffic?

Question 7HardTraining

A distributed training job starts 6 of 8 workers and the rest stay Pending. What went wrong?

#

Two jobs each grabbed part of the GPUs, and now both wait forever.

Show answer

The default scheduler places Pods one by one, so two jobs can each take part of the cluster and wait for each other: a deadlock that wastes the GPUs they hold. Training workers need all-or-nothing placement, often called gang scheduling.

Use a queueing system that admits a job only when all its Pods fit, such as Kueue, with its options to wait for all Pods to become ready and to requeue a job if they do not. Also consider topology, placing workers on nodes with fast interconnects.

Trade-offs

All-or-nothing admission can leave GPUs briefly idle while a large job waits, in exchange for no deadlocks.

Common mistake

Raising timeouts in the training code while the scheduler keeps the job half-placed.

Follow-up

How would you let small jobs use idle GPUs without blocking a large job forever?