supraj.dev

Interview questions & answers

CUDA

GPU threads, streams and attention kernels explained through practical questions.

  • 33 questions
  • 2 easy
  • 19 medium
  • 12 hard
  • 4 related articles

33 questions

Question 1EasyExecution model

Why should an ordinary CUDA kernel avoid depending on another block’s execution order?

#
Show answer

Threads are organized into blocks, and blocks form a grid. Blocks may be scheduled in any order, so an ordinary kernel cannot assume that another block has already produced a value. Threads within a block can cooperate using shared memory and appropriate synchronization.

For cross-block dependencies, split work into ordered kernel launches or use a supported cooperative mechanism with its constraints. More blocks do not mean all blocks run simultaneously. Check indexing, boundary conditions and synchronization correctness before tuning performance.

Question 2MediumStreams

Two CUDA streams access the same buffer. What makes the dependency safe?

#
Show answer

Operations are ordered within one stream; separate streams do not automatically establish the dependency you intend. Record an event after the producer’s work and make the consumer stream wait for it, or use an appropriate synchronization boundary. Preserve the buffer’s lifetime until dependent work completes.

Multiple streams permit overlap when resources and dependencies allow it; they do not guarantee parallel execution. Inspect default-stream semantics and profile actual overlap. Synchronizing the whole device after every operation may be correct but can destroy the concurrency you were trying to gain.

Question 3MediumNsight profiling

GPU utilization is high, but throughput is poor. What should you profile?

#
Show answer

Start with the kernels and transfers consuming wall-clock time. Compare achieved compute and memory throughput, scheduler stalls and occupancy for the expensive stages. Occupancy measures active warps relative to capacity; high occupancy does not prove that those warps are completing useful work efficiently.

Use Nsight Compute for kernel details and correlate with end-to-end latency and useful work per second. Determine whether the dominant stage is constrained by memory, computation or dependencies before tuning. A busy device or a larger batch alone does not establish that the user-facing objective improved.

Question 4MediumSIMT execution

Why can different branches within one CUDA warp reduce efficiency?

#
Show answer

A warp groups 32 threads executing under SIMT. When lanes take different branches, some lanes are inactive while the other path executes. The cost depends on the actual work and compiler/hardware behavior, not just whether the source contains an if statement.

Profile branch efficiency and the expensive paths. Where practical, group similar work or change the algorithm while preserving correctness and memory locality. Threads taking the same branch across a warp are different from divergence inside a warp; do not remove every condition blindly.

Question 5HardAttention kernels

How does FlashAttention reduce memory traffic without replacing attention with an approximation?

#
Show answer

The original algorithm tiles attention computation and uses on-chip memory to avoid repeatedly reading and writing large intermediate matrices in GPU high-bandwidth memory. Its goal is exact attention with less memory traffic, subject to normal floating-point numerical behavior.

Do not confuse this with PagedAttention’s management of KV-cache allocation across requests. Benchmark the supported kernel on your hardware, sequence lengths and dtype. Faster attention alone will not remove retrieval delays, queueing or every generation bottleneck.

Question 6MediumGPU sharing

How do you share one large GPU across several small inference workloads?

#

Six small models each use a fraction of an 80 GB GPU, yet each gets a whole GPU.

Show answer

Multi-Instance GPU (MIG), on supported data-center GPUs such as A100 and H100, splits the GPU into hardware partitions with their own memory and compute, so one workload cannot starve or crash another. Profiles are fixed sizes, so plan the layout.

Time-slicing, which the GPU Operator can configure on Kubernetes, lets several Pods take turns on one GPU. It raises utilisation but gives no memory or fault isolation, so one Pod can exhaust memory for the rest. MPS lets processes share the GPU concurrently with limited isolation.

Trade-offs

MIG gives predictable isolation with less flexibility; time-slicing is flexible but noisy-neighbour prone.

Common mistake

Using time-slicing for tenants that must not affect each other.

Follow-up

How would you choose MIG profiles for a mix of 7B and 13B models?

Question 7MediumPerformance

What are Tensor Cores, and how do you make sure your workload uses them?

#

A float32 training job runs far below the GPU's advertised performance.

Show answer

Tensor Cores are units specialised for matrix multiply-accumulate on small tiles, much faster than regular CUDA cores for that work. They operate on reduced-precision types such as FP16 and BF16, TF32 for float32 inputs, and FP8 or INT8 on newer GPUs, depending on the architecture.

Libraries such as cuBLAS and cuDNN use them automatically when data types and shapes fit. In practice: train with mixed precision, allow TF32 for float32 matrix multiplies in PyTorch (torch.set_float32_matmul_precision("high")), and keep matrix dimensions multiples of 8 or 16. Confirm with a profiler that the main kernels run on Tensor Cores.

Trade-offs

Lower precision gives large speedups but needs a check that accuracy holds.

Common mistake

Keeping strict float32 everywhere and wondering why the GPU looks slow.

Follow-up

How would you confirm in Nsight that matrix kernels are using Tensor Cores?

Question 8HardDistributed

Multi-node training hangs at the first all-reduce. How do you debug NCCL?

#

Single-node runs work. Two nodes start, print rank info and then hang until the timeout.

Show answer

Run with NCCL_DEBUG=INFO to see which network interfaces and transports NCCL selects. A common cause is the wrong interface, such as docker0 or a management NIC; set NCCL_SOCKET_IFNAME to the right one. Check that firewalls allow traffic between nodes and that the high-speed fabric (InfiniBand or EFA) is configured and visible inside the container.

Also confirm every rank uses the same world size and master address, and that versions of NCCL and drivers match across nodes. Set a distributed timeout so hangs fail with a clear error instead of waiting forever.

Trade-offs

Debug logging is verbose; enable it while reproducing, not for every production run.

Common mistake

Increasing the timeout instead of finding which link is blocked.

Follow-up

How would you run a quick all-reduce test outside your training code?

Question 9HardGPU reliability

Your GPU fleet logs Xid errors in the kernel log every day. On-call either ignores them all or drains every node that shows one. How do you build a triage policy instead?

#

Several hundred NVIDIA data-center GPUs on Kubernetes. Recent logs show Xid 13, 31, 63, 79 and 95.

Show answer

Xids are the driver's error reports, and they mean very different things. NVIDIA's Xid catalog gives each one an immediate action, which is the right starting point for automation:

- Xid 13 (graphics engine exception) and Xid 31 (GPU memory page fault): restart the application. These are usually caused by the workload (an illegal address in a kernel, for example), not the hardware, so draining the node punishes the wrong thing. Look for a pattern per job image before per node. - Xid 63 (GPU memory remapping event): ignore as an immediate action; the memory is repairing itself, though a reset is needed for the remap to take effect. - Xid 64 (remapping failure): reset the GPU and contact support. - Xid 94 (contained memory error): restart the application. Xid 95 (uncontained memory error): reset the GPU. - Xid 79 (GPU has fallen off the bus): restart the bare-metal host and contact support. - Xid 119 and 120 (GSP RPC timeout, GSP error): reset the GPU and investigate the software stack.

Turn the table into automation. A node agent maps each Xid to an action: notify the job owner, cordon and drain the node for a GPU reset, or cordon and open a hardware ticket for a host restart. Count repeats per GPU serial number, because a GPU that keeps producing hardware Xids after resets is an RMA candidate, while one that produces Xid 13 only with one image points to a code bug.

Also know that some Xids are side effects of others (the catalog marks several as sympathetic), so act on the first hardware Xid in a burst, not on every line.

Trade-offs

Automatic draining protects jobs from bad hardware but costs capacity on every false positive; per-Xid actions keep both low but must be maintained as the catalog changes.

Common mistake

Draining nodes for application-caused Xids such as 13 and 31, or ignoring Xid 79 because the job restarted elsewhere.

Follow-up

How would you keep a repeatedly failing GPU from being scheduled again after a node reboot clears its taint?

Question 10MediumGPU reliability

A GPU reported a memory remapping event. Jobs on it still run, so the team wants to leave it alone. What is actually pending, and when do you act?

#

A100 or H100 class GPU, long-running training job on the node, DCGM metrics available.

Show answer

Row remapping is the GPU's hardware mechanism for replacing degrading rows of frame-buffer memory with spare rows. It needs no software-visible holes in the address space, but NVIDIA's documentation is clear that the remap requires a GPU reset to take effect. Until then the remap is pending, and the faulty row is still the one in use. Once applied, the remap lasts for the life of the GPU, and the GPU supports up to 512 remappings.

So the right response is a planned reset, not panic and not neglect. Let the current job reach a checkpoint, cordon and drain the node, reset the GPU (or reboot the host), and return it to service. Xid 63 in the catalog is the remapping event itself and its immediate action is ignore; Xid 64, a remapping failure, means reset the GPU and contact support, because there is no spare row left to fix the problem.

Track remapping in telemetry so you see trends: the dcgm-exporter default counters include correctable and uncorrectable row remap totals and a remap-failed field. A GPU accumulating remaps quickly is heading for replacement, and NVIDIA publishes an RMA policy with thresholds for this.

Trade-offs

Resetting promptly costs a drain and a checkpoint restart; waiting saves that cost but leaves a known-degraded row in use.

Common mistake

Assuming that because the job is still running, the memory problem has already been fixed.

Follow-up

How would you schedule pending GPU resets across a fleet without breaking capacity for running training jobs?

Question 11MediumObservability

The FinOps dashboard shows the GPU fleet at 95% utilisation, yet training throughput is low and the ML team says they need more GPUs. Which metrics should the dashboard show instead?

#

The dashboard uses the basic GPU utilisation metric from dcgm-exporter. Profiling metrics are available on these data-center GPUs.

Show answer

Basic GPU utilisation only says that the compute engine was busy for some part of the sample period. A single small kernel running all the time scores close to 100% while most of the GPU sits idle. It is the wrong signal for both cost and capacity.

DCGM's profiling metrics measure much more specifically. Graphics engine activity is the fraction of time any part of the graphics or compute engines was active, which is close to what the basic metric shows. SM activity is the fraction of time at least one warp was active on a multiprocessor, averaged over all multiprocessors, so it exposes kernels that use only a few SMs. SM occupancy is the fraction of resident warps relative to the maximum. Tensor activity is the fraction of cycles the tensor pipe was active, which is what matters for matrix-heavy training and inference. DRAM activity is the fraction of cycles moving data to or from device memory, which shows memory-bound work.

Put SM activity, tensor activity and DRAM activity on the dashboard next to the basic metric, per job. A job at 95% engine activity with low tensor activity is a software problem (small batches, data loading stalls, unfused kernels, communication waits), and buying more GPUs will not fix it. Check the metric names in your exporter version: dcgm-exporter's default counters file has changed names over time, and several profiling fields are commented out by default.

Trade-offs

Profiling metrics are far more informative but cost a little collection overhead and are supported only on data-center GPUs from Volta onwards.

Common mistake

Approving GPU purchases from a utilisation number that only measures whether a kernel was running.

Follow-up

Which per-job metric would you put into a quota or chargeback policy, and why?

Question 12MediumDebugging

A PyTorch job crashes with a CUDA "illegal memory access" error, and the stack trace points at an innocent line such as a loss print. How do you find the real culprit?

#

Custom CUDA extension in the model. The error appears only on some batches.

Show answer

PyTorch's CUDA documentation explains it: by default GPU operations are asynchronous, so an error is not reported until the operation actually executes. The CPU has queued many kernels and moved on; the failure surfaces at the next point that synchronises with the GPU, such as .item() in a print. The stack trace shows where the CPU noticed, not which kernel failed.

Re-run with CUDA_LAUNCH_BLOCKING=1, which forces synchronous computation, so the error is raised at the kernel that caused it. It is slow, so use it only to debug. Then reproduce with the failing batch (save batch indices or the inputs), check index bounds and tensor shapes going into the custom extension, and run the CUDA Compute Sanitizer on the minimal reproduction for out-of-bounds accesses.

After an illegal memory access, the CUDA context is usually unusable: do not try to catch the error and continue in the same process. Restart the worker, and treat repeated occurrences on one GPU with no code change as a possible hardware issue (check the Xid log, where a GPU memory page fault appears as Xid 31).

Trade-offs

Synchronous launches make errors precise but slow everything down; error checks after every launch catch problems early at a smaller cost in development builds.

Common mistake

Debugging the line in the stack trace instead of the kernel that ran before it.

Follow-up

How would you add cheap device-side assertions to a custom kernel for debug builds?

Question 13MediumData movement

Profiling shows the GPU idle while each batch is copied from host memory, then busy while it computes, never both at once. You already pass non_blocking=True. What is missing?

#

PyTorch training with a DataLoader feeding tensors from pageable host memory.

Show answer

Overlap needs three things. First, the host buffer must be pinned (page-locked). NVIDIA's best practices guide says the asynchronous transfer version requires pinned host memory, and overlap with kernels again requires pinned memory. From pageable memory, non_blocking=True cannot do a true asynchronous copy. Second, the copy and the kernel must run in different, non-default streams. Third, the next batch must be ready early enough, which means the loader must stay ahead.

In PyTorch, set pin_memory=True on the DataLoader so batches land in pinned memory, then call .to(device, non_blocking=True). PyTorch's documentation confirms host-to-GPU copies are much faster from pinned memory and that non_blocking=True can overlap transfers with computation. Prefetching the next batch on a side stream completes the pattern.

Do not pin everything. Both NVIDIA and PyTorch warn that overusing pinned memory degrades the system, because pinned pages cannot be swapped and are taken from the host's usable RAM. Pin the batches in flight, not the whole dataset.

Trade-offs

Pinned memory gives faster, overlappable copies but takes RAM away from the operating system and other processes.

Common mistake

Adding non_blocking=True and assuming the copy is now asynchronous, while the source tensor is still in pageable memory.

Follow-up

When the data loader itself is the bottleneck, what do you change before touching GPU code?

Question 14MediumMemory access

A custom kernel reads one field from an array of structs and runs at a fraction of the memory bandwidth you expected. What is going on, and how do you fix it?

#

Each struct holds several floats; each thread reads one field of one struct.

Show answer

Global memory is read in transactions, and a warp is fast when its threads read adjacent addresses that fit in as few transactions as possible. That is coalescing. With an array of structs, neighbouring threads read the same field from neighbouring structs, so their addresses are a struct-size apart: a strided access. NVIDIA's best practices guide gives the arithmetic: a stride of 2 gives 50% load efficiency because half of each transaction is unused, and as the stride grows the effective bandwidth keeps falling.

Fix the data layout, not the arithmetic. Switch to a struct of arrays, one contiguous array per field, so thread i reads element i of the field's array and the warp's reads are contiguous. If you cannot change the layout everywhere, convert once at the boundary of the hot kernel, or stage tiles through shared memory and read them in a coalesced pattern.

Confirm with Nsight Compute: look at the memory workload analysis for sectors per request and achieved bandwidth before and after. This is one of the cheapest large wins in kernel work.

Trade-offs

Struct-of-arrays layouts are faster for GPU kernels but can be awkward for CPU code that handles one object at a time.

Common mistake

Tuning block sizes or unrolling loops when the access pattern is the real limit.

Follow-up

How do shared-memory bank conflicts appear when you stage a transposed tile, and how does padding fix them?

Question 15HardKernel tuning

A teammate capped registers with -maxrregcount to raise occupancy, as a profiler hint suggested. Occupancy went up, and the kernel got slower. Why?

#

Compute-heavy kernel with many live values per thread.

Show answer

Occupancy is the share of possible warps resident on a multiprocessor; it helps hide memory latency by giving the scheduler other warps to run. But NVIDIA's best practices guide is explicit that higher occupancy does not always equal higher performance. The price here was register spilling: with fewer registers per thread, values that no longer fit are spilled to local memory, which lives in device memory and is far slower than registers.

The guide describes the reverse trade-off too: a kernel with lower occupancy gets more registers per thread, which may mean less spilling. A compute-heavy kernel with many live values often runs best at moderate occupancy, with enough independent instructions per thread (instruction-level parallelism) to hide latency instead of more warps.

Tune by measurement. In Nsight Compute, check local memory traffic and spill counts, and the warp stall reasons, before and after. Prefer __launch_bounds__ on the specific kernel over a global -maxrregcount flag, because the flag applies to every kernel in the file, including ones that were fine. Keep the setting that gives the best runtime, not the best occupancy figure.

Trade-offs

More warps hide memory latency; more registers per thread avoid spills and raise instruction-level parallelism. The best point differs per kernel.

Common mistake

Treating occupancy as the goal rather than one input to runtime.

Follow-up

Which Nsight Compute sections would you read first to decide whether a kernel is latency-bound or throughput-bound?

Question 16MediumLaunch overhead

Small-batch inference spends most of its time launching kernels rather than running them. CUDA graphs are proposed. What do they fix, and what breaks when you capture?

#

PyTorch model with many small kernels per step; batch sizes vary between requests.

Show answer

Each kernel launch costs CPU time. When kernels are tiny, the GPU waits for the CPU to issue the next one. A CUDA graph records a sequence of kernels once and replays it with one launch, removing most of that per-kernel overhead.

The constraints come from how replay works, and PyTorch's CUDA semantics page lists them. Capture must happen on a non-default stream. Operations that synchronise the CPU with the GPU, such as .item(), are prohibited during capture. Every replay reads from and writes to the same memory addresses, so inputs must be copied into static tensors before replay. Dynamic shapes are prohibited: the graph assumes every tensor has the same size and layout on every replay. Some of these mistakes raise errors, but others fail silently, which is worse. Warm up on a side stream with a few eager iterations before capturing.

Variable batch sizes are handled by capturing a set of graphs for chosen batch sizes and padding each request batch up to the nearest captured size, which is what serving engines do. The trade-off is memory for each captured graph and some wasted compute on padding.

Trade-offs

Graphs remove launch overhead but cost memory, capture time at start-up and flexibility in shapes and control flow.

Common mistake

Capturing with a CPU sync or a data-dependent shape inside and getting silently wrong results on replay.

Follow-up

How would you choose which batch sizes to capture for a serving workload?

Question 17MediumDrivers

A team ships a container built with a new CUDA 13 toolkit, and it fails on nodes whose driver supports only CUDA 12. Their CUDA 12 images work everywhere. What are your options?

#

Data-center GPUs on Kubernetes. Node driver upgrades are scheduled quarterly by a different team.

Show answer

NVIDIA's compatibility model has three parts. Binary backward compatibility: applications built with older toolkits keep working on newer drivers. Minor version compatibility: an application built with any toolkit in a major family runs on a sufficiently new driver for that family, which NVIDIA lists as driver 525 or later for CUDA 12.x and 580 or later for CUDA 13.x, with limitations. Features that need a newer driver may return cudaErrorCallRequiresNewerDriver, and applications that compile device code to PTX will not work on older drivers. Forward compatibility: the cuda-compat-<major>-<minor> package lets applications built with a newer toolkit run on an older driver across major families.

So the options, in order of preference: upgrade node drivers to a branch that supports CUDA 13; or keep building on CUDA 12 until the fleet is ready; or use the forward compatibility package. That package is supported only on NVIDIA data-center GPUs, select NGC Server Ready RTX SKUs and Jetson. It only provides libraries, so you must point the loader at them (for example with LD_LIBRARY_PATH), and a mismatch shows up as error 803 or 804. NVIDIA also notes older forward compatibility packages are not supported on newer drivers, so remove them when the driver catches up.

Make it a platform rule: publish the supported driver branch per node pool, label nodes with it, and have CI check that an image's CUDA version is supported by the pool it targets.

Trade-offs

Forward compatibility unblocks one team quickly but adds a library layer that must be removed later; fleet driver upgrades are cleaner but slower and riskier.

Common mistake

Bundling a random driver library into the image and hoping it matches the kernel module on the node.

Follow-up

How would you roll out a new driver branch across GPU node pools without breaking running jobs?

Question 18HardInterconnect

A training job scaled from 8 GPUs in one node to 16 GPUs across two nodes, and step time got worse instead of better. How do you show where the bandwidth went?

#

HGX-style nodes with NVLink inside each node and an RDMA network between nodes. Data-parallel training with large gradients.

Show answer

Inside one node, all-reduce traffic travels over NVLink; an H100 SXM's NVLink bandwidth is 900 GB/s. Across two nodes, part of every all-reduce must cross the network, which is usually far slower per GPU. If the gradient all-reduce is not hidden behind compute, step time is set by the slowest link.

Measure it with nccl-tests (all_reduce_perf) on the same nodes, 8 GPUs and then 16. Read bus bandwidth, not algorithm bandwidth. Algorithm bandwidth is size divided by time and falls as you add ranks; bus bandwidth applies a factor of 2(n-1)/n for all-reduce so that it reflects the speed of the actual links, independent of rank count, and can be compared with hardware peak. If 16-GPU bus bandwidth is close to your network's per-GPU bandwidth, the network is the bottleneck and the job needs more overlap or less communication. If it is far below, the network path is misconfigured.

For misconfiguration, run with NCCL_DEBUG=INFO and check the transport NCCL selected for inter-node traffic, set NCCL_IB_HCA (RDMA adapters) or NCCL_SOCKET_IFNAME (IP interfaces) explicitly, and dump the detected topology with NCCL_TOPO_DUMP_FILE to check GPU-to-NIC affinity. Then fix the job: overlap communication with backward compute, use larger buckets, or use hierarchical or sharded strategies that keep most traffic inside the node.

Trade-offs

More GPUs add compute but also communication; past a point, a bigger per-GPU batch or a different parallelism strategy beats adding nodes.

Common mistake

Comparing algorithm bandwidth across different rank counts and concluding that NCCL got slower.

Follow-up

How does placing GPUs and NICs on the same PCIe switch or NUMA node change inter-node bandwidth?

Question 19EasyInterconnect

Months ago someone fixed a hang by adding NCCL_P2P_DISABLE=1 to the base training image. Now multi-GPU jobs are slower than benchmarks. What happened?

#

The variable is set in the Dockerfile; nobody remembers why.

Show answer

NCCL_P2P_DISABLE=1 turns off direct GPU-to-GPU communication, so NCCL routes traffic through host memory instead of NVLink or PCIe peer-to-peer paths. That can hide a broken peer-to-peer path, but it throws away the fastest links inside the node.

NCCL's documentation puts NCCL_P2P_DISABLE, NCCL_P2P_LEVEL, NCCL_IB_DISABLE and NCCL_NET_GDR_LEVEL in its debugging section, with the warning that they are fine for experiments or debugging but should generally not be set for production code. Baking one into a base image turns a debugging step into a permanent slowdown for every job.

Remove it, re-run nccl-tests to compare bus bandwidth with and without it, and if the original hang returns, fix the root cause: IOMMU or ACS settings that break peer-to-peer, a container missing the right devices, or a bad GPU. Keep NCCL tuning variables out of base images; set them per job with a comment and a ticket.

Trade-offs

Disabling a transport is a quick way to isolate a problem but almost always costs bandwidth if left in place.

Common mistake

Leaving debugging environment variables in a shared image where nobody reviews them.

Follow-up

How would you detect configuration drift in environment variables across training images?

Question 20MediumGPU sharing

You need to change the MIG layout on some H100 nodes from seven small instances to two larger ones for a new model. How does that work on Kubernetes, and what disruption should you plan for?

#

NVIDIA GPU Operator with MIG Manager. Some small-instance inference pods are running on those nodes.

Show answer

With the GPU Operator, you change MIG geometry by relabelling the node with nvidia.com/mig.config=<profile>, where the value names a layout in MIG Manager's configuration (the documentation's example, all-1g.10gb, is the all-small-instance layout). MIG Manager applies it with the mig-parted tool and reports progress in the nvidia.com/mig.config.state label, which shows pending while it works and success when done.

Plan for disruption. NVIDIA's documentation says MIG Manager requires that no user workloads are running on the GPUs being configured, and that it first stops all GPU pods, including the device plugin, before applying changes; the node may need to be cordoned, and some changes can require a reboot. So cordon, drain the GPU workloads to other nodes, relabel, wait for success, and uncordon.

Also check how workloads request MIG devices. With the single strategy, each MIG device is exposed as nvidia.com/gpu and all GPUs on the node must use the same profile. With the mixed strategy, each profile has its own resource name, such as nvidia.com/mig-1g.10gb. Changing profiles under the mixed strategy changes resource names, so pod specs that request the old profile will stay Pending. On an H100 SXM, NVIDIA lists up to 7 MIG instances at 10 GB each, which is the small-instance layout you are leaving.

Trade-offs

MIG gives hard isolation but fixed, coarse partitions that are disruptive to change; time-slicing is flexible but gives no isolation.

Common mistake

Relabelling a node with live inference pods and being surprised when they are stopped.

Follow-up

How would you decide the MIG profile mix for a node pool from observed demand?

Question 21MediumGPU sharing

With GPU time-slicing enabled, one team requests four replicas of a shared GPU, expecting four times the compute of teams that request one. Is that what they get, and what else surprises teams on time-sliced GPUs?

#

GPU Operator time-slicing config with several replicas per physical GPU, used for small inference and notebooks.

Show answer

No. NVIDIA's documentation states that a request for more than one time-sliced GPU does not guarantee the pod a proportional amount of compute. Replicas are scheduling tokens that let several pods share one GPU; the work then interleaves. The failRequestsGreaterThanOne option makes such requests fail with an UnexpectedAdmissionError, which is a good default because it removes the false expectation.

Other surprises to warn teams about. There is no memory or fault isolation between replicas, unlike MIG, so one pod can exhaust GPU memory and break its neighbours. dcgm-exporter cannot associate metrics with containers when time-slicing is enabled, so per-team GPU accounting gets harder. The Operator does not watch the time-slicing config map, so after changing it you must restart the device plugin pods, which NVIDIA recommends doing in a maintenance window. With renameByDefault: true, the resource is advertised as nvidia.com/gpu.shared, which makes it obvious in pod specs that the GPU is shared.

Use time-slicing for bursty, low-stakes work such as notebooks and development, set memory limits in the applications themselves, and move anything with an SLO to MIG instances or whole GPUs.

Trade-offs

Time-slicing raises utilisation cheaply but trades away isolation and clean per-container metrics.

Common mistake

Treating time-sliced replicas as fractional GPUs with guaranteed compute and memory.

Follow-up

How would you charge teams fairly for shared GPUs when per-container metrics are not available?

Question 22MediumGPU reliability

New GPU nodes join the cluster and immediately receive training jobs. Twice this month a brand-new node had a bad GPU that crashed a multi-day job. How do you gate new and repaired nodes?

#

GPU Operator installed; DCGM available on nodes.

Show answer

Do not schedule real work on a GPU node until it has passed health checks. Keep new and repaired nodes tainted, run diagnostics, and remove the taint only on success.

DCGM provides levels of diagnostics through dcgmi diag. NVIDIA's guide lists run level 1 as a quick check of seconds covering software deployment, level 2 at about two minutes adding memory and PCIe/NVLink tests, and level 3 at about fifteen minutes adding memory bandwidth, hardware diagnostics and targeted stress and power tests. Use the long level for burn-in of new or repaired nodes, and the quick level as a cheap periodic or pre-job check.

For multi-node training, add a short NCCL test across the new node and a known-good node, because a GPU can pass local diagnostics while its network path is wrong. Record the GPU serial numbers and results, and feed failures into the same RMA tracking as production Xids.

The cost is a delay before a node is usable and some GPU time spent on tests. Compared with losing a multi-day job and the debugging time, that is cheap.

Trade-offs

Longer diagnostics catch more faults but delay capacity; quick checks are cheap enough to run often but miss marginal hardware.

Common mistake

Treating a node as healthy because the driver loaded and the device plugin advertised its GPUs.

Follow-up

How would you rerun diagnostics automatically after a GPU reset triggered by an Xid?

Question 23MediumPerformance

A synchronous data-parallel job runs at the speed of its slowest GPU. Per-rank timing shows one rank consistently slower, on a GPU with no errors. What do you check?

#

64 GPUs. The slow rank is always on the same physical GPU, even across job restarts.

Show answer

In synchronous training, every step waits for the slowest rank at the collective, so one slow GPU slows all 64. If the slowness follows a physical GPU across restarts, suspect the hardware environment rather than the code: clock throttling from temperature or power limits, a degraded PCIe or NVLink link, or memory errors being corrected.

Compare clocks, temperature, power and throttle reasons for that GPU with its neighbours. DCGM exposes fields for SM clock, GPU and memory temperature, power, and throttle time by cause (power, thermal, reliability and others). Note that dcgm-exporter's default counters file includes clocks, temperatures and power but leaves the throttle-reason counters commented out, so you may need to enable them to see why a clock is low. Also check the PCIe replay counter and NVLink error counters, and the correctable error and row remap counts.

A thermal cause usually means airflow or cooling at the server or rack; a power cause may be a power cap set on that node. Fix the environment, then run diagnostics before returning the node. In the meantime, exclude the node from synchronous jobs, because one slow rank costs the whole job far more than one missing GPU.

Trade-offs

Enabling more DCGM fields costs a little collection overhead but turns "this GPU is slow" into a named cause.

Common mistake

Retuning the training code when the slowness follows one physical GPU.

Follow-up

How would you automatically detect stragglers from training metrics before anyone files a ticket?

Question 24HardMemory access

You rewrote a matrix transpose to stage tiles through shared memory so that global reads and writes are coalesced. It got faster, but Nsight Compute reports many shared memory bank conflicts and the kernel is still well below expected bandwidth. What is happening and what is the usual fix?

#

The tile is declared as __shared__ float tile[32][32]. Threads write rows of the tile and read columns.

Show answer

Shared memory is split into banks that can be accessed at the same time. The CUDA Best Practices Guide says there are 32 banks, each 32 bits wide per clock, with successive 32-bit words in successive banks, and that when several addresses in one warp request map to the same bank, "the accesses are serialized", cutting effective bandwidth by the number of separate requests. (All threads reading the same address is a broadcast, not a conflict.)

With a 32 × 32 float tile, each row is 32 words, so every element of a column sits in the same bank. When a warp reads a column, all 32 threads hit one bank: a 32-way conflict, so the read is done one thread at a time.

The fix in the guide is to pad the array with one extra column: __shared__ float tile[32][33]. Now consecutive rows are offset by one bank, so a column read touches 32 different banks. The cost is 32 extra floats per tile.

Then verify: - Re-profile and check that shared memory bank conflicts drop to near zero and that achieved bandwidth rises. - Remember padding changes indexing only for the shared array; global accesses stay as they were. - For other element sizes (for example 8-byte doubles or 2-byte halves) work the bank mapping out again rather than assuming +1 is right.

Trade-offs

Padding removes conflicts at a small shared memory cost, which can slightly lower how many blocks fit per SM.

Common mistake

Fixing global memory coalescing and assuming the shared memory side is free.

Follow-up

How would bank conflicts change if the tile held 64-bit values?

Question 25HardPerformance

A team spent two weeks optimising the arithmetic in a custom fused kernel with no speed-up. Before they continue, how do you decide whether the kernel is limited by memory bandwidth or by compute?

#

The kernel reads two large tensors, applies a few element-wise operations and writes one result. Nsight Compute is available.

Show answer

Use the roofline idea: compare the kernel's arithmetic intensity (useful operations per byte moved to and from memory) with the GPU's ratio of peak compute to peak memory bandwidth. Below that ratio the kernel is memory-bound, and faster arithmetic cannot help; above it, compute is the limit.

For this kernel: a few element-wise operations per element on data read once and written once is a handful of operations per 12 bytes (two 4-byte reads and one write), far below the ratio of any modern data-centre GPU, which is many operations per byte. It is memory-bound, which explains why arithmetic work did nothing.

Check it with measurements: - Effective bandwidth, as the Best Practices Guide defines it: (bytes read + bytes written) / 10^9 / time in seconds, giving GB/s. Compare with the GPU's peak memory bandwidth. Close to peak means you are done optimising this kernel in isolation. - Nsight Compute's Speed Of Light section reports, per unit, achieved throughput as a percentage of the theoretical maximum, and its roofline chart plots the kernel against the memory and compute ceilings.

For memory-bound kernels, the levers are moving fewer bytes: fuse more operations into the same pass, use smaller data types (bf16, fp8), avoid re-reading data, and make sure accesses are coalesced.

Trade-offs

Fusing more work into one pass raises arithmetic intensity but makes kernels harder to maintain and reuse.

Common mistake

Optimising instructions in a kernel that is already close to peak memory bandwidth.

Follow-up

How would you estimate the bytes a fused attention kernel moves compared with an unfused one?

Question 26MediumKernel tuning

A kernel launched with 101 blocks takes almost twice as long as the same kernel with 100 blocks on a GPU with 100 SMs, even though it does only 1% more work. Why, and how do you size grids to avoid it?

#

Each block uses enough registers and shared memory that only one block fits on an SM at a time. Blocks take about the same time each.

Show answer

Blocks are scheduled in waves. With one block per SM and 100 SMs, 100 blocks finish in one wave. The 101st block starts only when an SM frees up and runs alone while 99 SMs sit idle, so the kernel takes two full block-times. This is often called the tail effect or wave quantization.

The Best Practices Guide's grid guidance points the same way: the number of blocks should be larger than the number of multiprocessors so all of them have work, there should be multiple active blocks per multiprocessor, and for scalability the number of blocks per launch should be in the thousands. With many small blocks, the last partial wave is a small fraction of the total.

Ways to fix it: - Use smaller blocks or less shared memory per block so several blocks fit per SM, and launch many more of them. - Size the work so the grid is a multiple of (SMs × blocks per SM), or much larger than it. - For persistent-style kernels, launch exactly one wave of blocks and loop over work items inside the kernel. - Read the SM count at runtime (cudaGetDeviceProperties) instead of hard-coding it, because it differs across GPU models and MIG slices.

Check the result with a profiler's per-SM timeline or occupancy report.

Trade-offs

Larger blocks reduce scheduling overhead and can share more data, but make wave quantization worse; many small blocks smooth it out.

Common mistake

Tuning a grid size on one GPU model and assuming it stays efficient on a GPU with a different SM count or a MIG slice.

Follow-up

How would you handle a kernel whose work per block varies a lot?

Question 27HardSIMT execution

An old reduction kernel that skips __syncthreads() in the last 32 steps, relying on threads of a warp running in lockstep, gives wrong results now and then on newer GPUs. Why, and how should it be written?

#

The kernel uses a volatile shared memory array for the final warp and the non-sync __shfl_down intrinsic in another version.

Show answer

The kernel assumes warp-synchronous execution, meaning it "implicitly assumes threads in the same warp are synchronized at every instruction", as the Volta Tuning Guide puts it. Volta introduced Independent Thread Scheduling among threads in a warp, so that assumption no longer holds: the guide warns that a different set of threads than intended can take part in such code, and results can be wrong intermittently.

Fixes from the guide: - Replace __shfl, __any, __all and __ballot with the *_sync versions, such as __shfl_down_sync(mask, value, offset). The mask states explicitly which lanes must participate; for a full warp it is 0xffffffff. - Where threads exchange data through shared memory within a warp, put __syncwarp() between the write and read steps instead of relying on volatile. - Make sure every non-exited thread in a block reaches each __syncthreads().

A modern warp reduction loops for (offset = 16; offset > 0; offset /= 2) v += __shfl_down_sync(0xffffffff, v, offset);, which needs no shared memory for the last 32 values. Libraries such as CUB already do this correctly.

To find the remaining violations, run compute-sanitizer with its racecheck and synccheck tools, which the guide recommends.

Trade-offs

Explicit synchronisation costs a little but makes the code correct on every architecture; library primitives remove the risk entirely.

Common mistake

Treating volatile shared memory as a substitute for warp synchronisation.

Follow-up

What happens if the mask passed to a sync intrinsic does not include all threads that actually call it?

Question 28MediumAtomics

A histogram kernel with 256 bins uses one global atomicAdd per input element. It is far slower than expected, and a float version of the same kernel gives slightly different totals between runs. Explain both and how you would rewrite it.

#

Inputs are skewed: a few bins receive most of the values.

Show answer

Speed: an atomic is a serialised read-modify-write on one address. The CUDA Programming Guide says atomics "should be used sparingly" because they enforce synchronisation that can impact performance. With skewed inputs, thousands of threads hit the same few bins and queue behind each other.

Rewrite: - Privatise: give each block its own histogram in shared memory, update it with shared memory atomics (much cheaper and contended only within the block), then add each block's histogram to the global one at the end with one atomic per bin per block. - Reduce before updating: within a warp, combine values destined for the same bin (for example with warp-level primitives) so one thread issues the atomic. - For very hot bins, consider several copies per block to spread contention. - Or use a library: CUB provides histogram and reduction primitives tuned for this.

Different float totals: floating-point addition is not associative, so the result depends on the order of additions, and the order in which atomics land changes from run to run. Integer counts are exact; float sums are not reproducible with atomics. If you need bitwise reproducibility, use a deterministic reduction with a fixed order (for example per-block partial sums combined in a fixed tree), and accept some cost.

Trade-offs

Privatised histograms are much faster but use shared memory and an extra merge step; deterministic reductions cost speed for reproducibility.

Common mistake

Expecting bitwise identical float results from code that accumulates with atomics.

Follow-up

How would you make a reduction deterministic across different GPU counts, not just across runs?

Question 29HardMemory

To process a dataset larger than GPU memory, a team switched to cudaMallocManaged. It runs without out-of-memory errors, but much slower than expected, and profiling shows constant page migrations. How would you make it fast, or should they avoid managed memory here?

#

Linux, a recent data-centre GPU. The kernel sweeps through the data several times, and the CPU also touches parts of it between passes.

Show answer

Managed memory lets an application oversubscribe GPU memory: the CUDA Programming Guide says unified memory enables applications to oversubscribe the memory of any individual processor. But pages move on demand when touched, and the guide notes migration can be an expensive operation proportional to the page size. A kernel that sweeps more data than fits evicts pages it will need again on the next pass, and CPU access between passes moves pages back. That is thrashing.

Make it fast: - Prefetch: cudaMemPrefetchAsync migrates data closer to the processor that will use it, ahead of time and asynchronously. Prefetch the next chunk while the current one is processed. - Advise: cudaMemAdviseSetReadMostly for data mostly read (it can be duplicated instead of moved), SetPreferredLocation to keep data where it belongs (a hint, not a guarantee), and SetAccessedBy for data a processor will access often. The guide warns that hints used wrongly can degrade performance, so measure. - Restructure the algorithm into chunks that fit in GPU memory, so each chunk is moved once per pass. - Keep the CPU from touching data the GPU is about to use.

For a well-understood streaming pattern, explicit chunked copies with pinned memory and streams are often faster and more predictable than managed memory. Also check the platform: on Windows and devices below compute capability 6.0, the guide says managed memory cannot exceed physical GPU memory.

Trade-offs

Managed memory simplifies code and allows oversubscription; explicit chunked copies take more code but give predictable performance.

Common mistake

Treating managed memory as free extra GPU memory without prefetching or restructuring access.

Follow-up

How would you size chunks and overlap their transfers with compute?

Question 30HardGPU sharing

Twenty small inference processes each use a few percent of an H100. Someone proposes NVIDIA MPS to run them concurrently. What does MPS give you, what are its limits, and when would you choose MIG instead?

#

The processes belong to different internal teams. One team's service occasionally crashes with GPU faults.

Show answer

MPS lets kernels from several processes share the GPU at the same time. NVIDIA's MPS docs say it is useful when each process does not generate enough work to saturate the GPU, letting the leftover capacity be used by kernels from other processes. That fits twenty small services.

What you get and what you do not: - Memory: MPS clients have fully isolated GPU address spaces, and on Volta and later you can cap a client's pinned device memory with CUDA_MPS_PINNED_DEVICE_MEM_LIMIT (for example 0=1G). - Compute: CUDA_MPS_ACTIVE_THREAD_PERCENTAGE limits the share of threads client contexts can use. - Fault isolation: weak. The docs say a fatal GPU fault from a Volta MPS client is reported to all clients on that GPU, and killing a client without synchronising its GPU work can leave the server and other clients in an undefined state. The crashing team's service would take the others down with it. - Operations: Linux and QNX only; one user per system can have an active MPS server; up to 60 client contexts per device by default; and monitoring attributes client activity to the server, so per-client usage is not visible in nvidia-smi or NVML.

Choose MIG when tenants must not affect each other: hardware partitions with their own memory and compute and separate fault domains, at the cost of fixed slice sizes and fewer slices than twenty. A common pattern is MIG between teams and MPS inside one team's slice for its own trusted processes.

Trade-offs

MPS packs many small processes efficiently with soft limits but shares faults; MIG isolates strongly but in coarse, fixed slices.

Common mistake

Putting untrusted or crash-prone tenants on one MPS server and expecting isolation.

Follow-up

How would you attribute GPU usage to each team when MPS hides per-client activity?

Question 31HardNumerics

You enable FP8 training with Transformer Engine on H100s. The loss matches BF16 for a while, then diverges after a sudden activation spike. What role do scaling factors play, and what would you check?

#

The recipe uses delayed scaling with the hybrid format. The run uses tensor parallelism across eight GPUs.

Show answer

FP8 has a tiny range, so each tensor is multiplied by a scaling factor to fit. The Transformer Engine guide gives the two formats: E4M3 stores values up to ±448 and E5M2 up to ±57344; the hybrid format uses E4M3 in the forward pass (more precision for activations and weights) and E5M2 in the backward pass (more range for gradients).

With delayed scaling, the scale for a tensor comes from the maximum absolute values (amax) recorded over previous iterations. That is cheap, but it means the scale lags. When activations jump suddenly, values above the range the old scale allows overflow or saturate, and if that happens repeatedly the run can diverge.

What to check and change: - The amax history length and how the scale is computed from it: a short history adapts faster to spikes; a long one is smoother but slower to react. - Synchronisation: the guide notes that multi-GPU training needs communication to synchronise scaling factors and amax history. Check it is configured for your parallel layout, so every rank scales the same tensor the same way. - Which layers run in FP8: keep sensitive parts, often the first and last layers and normalisation, in higher precision. - Newer recipes: on Blackwell, MXFP8 gives each block of 32 values its own scale, which lets E4M3 be used everywhere and makes outliers in one block less harmful.

Compare against a BF16 run over the same steps and log per-layer amax values, so you can see which layer spiked first.

Trade-offs

Delayed scaling is cheap but reacts late to spikes; finer-grained or current scaling handles outliers better at more overhead or newer hardware.

Common mistake

Switching a whole model to FP8 without per-layer monitoring of amax and overflow.

Follow-up

How would you decide which layers to keep in BF16?

Question 32MediumDebugging

A custom CUDA kernel gives slightly different outputs from run to run with the same inputs, and sometimes garbage values in a few elements. No errors are reported. Which tools would you use to find the bug, and what do they cover?

#

The kernel uses shared memory tiles and __syncthreads(), and reads from a buffer allocated with cudaMalloc.

Show answer

Use NVIDIA Compute Sanitizer, which runs the program under instrumentation and has four tools: - memcheck: out-of-bounds and misaligned accesses, hardware exceptions and memory leaks. - racecheck: data access hazards in shared memory. The docs say it currently only detects accesses to on-chip shared memory, so races through global memory are not covered. - initcheck: reads of device global memory that was never written by the device or by a CUDA memcpy or memset. Garbage values in a few elements often come from this: cudaMalloc does not zero memory. - synccheck: invalid use of synchronisation primitives such as __syncthreads() and __syncwarp(), for example a barrier inside a branch that not all threads take.

Order of attack for this bug: run initcheck (garbage values), then racecheck (run-to-run differences with shared memory tiles), then synccheck. Fix what they report and rerun.

Practical notes: the tools can need a lot of memory for tracking data, and runs are slower, so use a small input that still reproduces the bug. Memcheck can be made much faster with compile-time instrumentation (-fdevice-sanitize=memcheck). Add the sanitizer to CI for custom kernels on small inputs.

If the tools report nothing, remember that float atomics or reductions in varying order can also cause small differences without any bug.

Trade-offs

Sanitizers find bugs precisely but slow runs heavily; using them on small inputs in CI catches most issues cheaply.

Common mistake

Assuming a missing error message means the kernel has no memory or synchronisation bug.

Follow-up

How would you find a race that only happens through global memory?

Question 33HardPerformance

A custom kernel tuned for A100 runs on H100 only about as much faster as the extra memory bandwidth explains. What Hopper features would you look at to get more, and what would you check first?

#

The kernel is a tiled GEMM-like operation that stages tiles through shared memory with manual loads, sized for A100 limits.

Show answer

The kernel is still shaped for A100. The Hopper Tuning Guide lists what changed: - More shared memory: 228 KB per SM on compute capability 9.0, up from 164 KB on A100. Tile sizes chosen for A100 leave capacity unused; larger tiles can raise data reuse. - Larger L2: 50 MB, up from 40 MB. - Thread block clusters: groups of blocks that can cooperate and access each other's shared memory, with a portable maximum cluster size of 8 (16 with an explicit opt-in). Clusters let neighbouring blocks share loaded tiles instead of each loading their own. - The Tensor Memory Accelerator (TMA): an asynchronous copy engine that moves 1D to 5D tensors between global and shared memory, and between shared memory of SMs in the same cluster, exposed through cuda::memcpy_async with barriers and pipelines. It replaces manual per-thread loads, freeing threads for compute and allowing deeper pipelining.

Check first: - Profile on H100 with Nsight Compute to see what limits the kernel now (memory, shared memory, tensor pipes, latency). - Make sure it is compiled for sm_90, not only for older architectures run through compatibility. - Compare with a library kernel (cuBLAS, CUTLASS) for the same shape. If the library is much faster, using it or a CUTLASS template may be cheaper than hand-porting to TMA and clusters.

Retune on the target hardware rather than reusing A100 launch parameters.

Trade-offs

Using TMA and clusters can unlock large gains but ties the kernel to Hopper-class GPUs and adds complexity; libraries get most of the gain for less work.

Common mistake

Moving a kernel to new hardware without recompiling for its architecture or retuning tile sizes.

Follow-up

How would you keep one code base that runs well on both A100 and H100?