supraj.dev

Interview questions & answers

PyTorch

Tensor memory, strides, views and the hidden cost of layout changes.

  • 25 questions
  • 1 easy
  • 13 medium
  • 11 hard

25 questions

Question 1EasyTensor layout

How can a transposed tensor share storage while showing a different layout?

#
Show answer

A tensor combines storage with metadata such as shape, strides and storage offset. Transpose can change the indexing interpretation without moving the elements. It produces a view, so changes through that view can affect the underlying tensor.

Inspect shape, stride and contiguity when debugging layout-sensitive code. Shared storage saves copies, but aliasing matters when mutating data. Do not describe every shape operation as zero-copy: PyTorch reshape may return a view or allocate a new tensor depending on the layout.

Question 2MediumContiguity

A transpose followed by reshape increases memory use. What would you inspect?

#
Show answer

Check the strides and whether the transposed tensor is contiguous. Reshape may need a copy when the requested layout cannot be represented as a view. Calling contiguous on an already contiguous tensor returns it; otherwise it copies into contiguous storage.

Profile allocations and the downstream operator’s layout requirements before adding contiguous calls everywhere. Keep a compatible view when practical, or make the necessary copy once at a deliberate boundary. A fast metadata operation can still lead to an expensive copy later.

Question 3MediumMemory

Training hits CUDA out of memory midway, not at step one. What do you check?

#

Epoch one runs fine. Around step 4,000 the job dies with an OOM error.

Show answer

Check for memory that accumulates: summing the loss tensor into a running total (total += loss) keeps every step's graph alive; use loss.item() instead. Evaluation without torch.no_grad() also keeps graphs. Variable-length batches can produce a rare, much larger batch that exceeds memory.

Use torch.cuda.memory_summary() or memory snapshots to see whether allocated memory grows or the cache is fragmented. If fragmentation is the issue, the allocator's expandable segments option can help. Reduce peak memory with mixed precision, gradient checkpointing or a smaller micro-batch with gradient accumulation.

Trade-offs

Gradient checkpointing saves memory but recomputes activations, so steps get slower.

Common mistake

Calling torch.cuda.empty_cache() in the loop, which does not free tensors that are still referenced.

Follow-up

How would you cap sequence lengths without hurting the model?

Question 4MediumTraining

When does mixed precision help, and what can go wrong?

#

A training job is memory-bound and slow on recent GPUs.

Show answer

Autocast runs eligible operations in lower precision, which uses less memory and uses tensor cores, while keeping sensitive operations in float32. On hardware with bfloat16 support, bf16 is usually simplest: its exponent range matches float32, so gradients rarely underflow.

With float16, use a gradient scaler, which multiplies the loss so small gradients do not underflow and skips steps that overflow. Watch for NaN or infinite losses, compare a short run against a float32 baseline, and keep custom numerically sensitive code in float32.

Trade-offs

Lower precision speeds training but can change convergence slightly; validate on your metrics, not just loss curves.

Common mistake

Using float16 without a gradient scaler and blaming the model when training diverges.

Follow-up

Which operations would you keep in float32 in a custom loss?

Question 5HardPerformance

GPU utilisation sits around 35% during training. How do you find the bottleneck?

#

An expensive GPU node trains slowly while its CPUs look busy.

Show answer

Profile one stretch of training with torch.profiler and look at the timeline. Gaps between GPU kernels usually mean the input pipeline is too slow: increase DataLoader num_workers, use pin_memory=True and persistent_workers, and move expensive decoding or augmentation off the critical path.

Other causes are forced synchronisation (.item(), printing tensors or moving them to the CPU every step), many tiny kernels from a small batch, and Python overhead, which torch.compile can reduce. Change one thing at a time and measure throughput in samples per second.

Trade-offs

More workers speed loading but use more CPU and memory per process.

Common mistake

Buying a bigger GPU when the input pipeline is the bottleneck.

Follow-up

How would you tell data loading from host-device synchronisation in a profiler trace?

Question 6MediumMemory

nvidia-smi shows your PyTorch training process using 76 GB of an 80 GB GPU, but torch.cuda.memory_allocated() reports 41 GB. A teammate adds torch.cuda.empty_cache() after every step "to free the leak". What is going on, and is that fix right?

#

One training process per GPU. Step time got worse after the empty_cache change, and memory errors did not go away.

Show answer

There is probably no leak. PyTorch uses a caching allocator: when tensors are freed, it keeps their GPU memory in its own pool for reuse instead of returning it to the driver, because calling the driver is slow. The CUDA semantics notes say this unused cached memory "will still show as if used in nvidia-smi".

Two numbers matter: - memory_allocated() and max_memory_allocated(): memory occupied by live tensors. - memory_reserved() and max_memory_reserved(): total memory managed by the caching allocator, live tensors plus the cached pool.

The gap of about 35 GB is mostly cache. empty_cache() releases unused cached memory back to the driver, but the docs say the memory occupied by tensors is not freed, so it does not give PyTorch itself more room. Calling it every step forces the allocator to request memory from the driver again on the next step, which costs time, which is why step time got worse.

When empty_cache() is useful: when another process or library outside PyTorch's allocator must use the memory, for example between two phases that use different frameworks.

If you are hitting out-of-memory errors with a large reserved-but-unallocated gap, the problem is usually fragmentation, not a leak. Look at torch.cuda.memory_summary() and consider allocator settings. A real leak shows up as memory_allocated() rising step after step.

Trade-offs

The cache makes allocation fast at the cost of looking greedy to outside tools; emptying it helps other processes but slows PyTorch down.

Common mistake

Reading nvidia-smi as the amount of memory your tensors use.

Follow-up

How would you tell a real tensor leak from allocator caching using only PyTorch's memory statistics?

Question 7HardMemory

A fine-tune with variable-length batches dies with CUDA out of memory after a few hours. The error says it tried to allocate 2 GiB while several GiB are "reserved by PyTorch but unallocated". Peak allocated memory is well under capacity. What is happening and what do you change?

#

Batches are packed by token count, so tensor shapes change every step. Running on PyTorch 2.15.

Show answer

This is fragmentation. The caching allocator holds memory in segments and splits them into blocks. With shapes changing every step, segments get carved into pieces of different sizes. Over time there is plenty of free memory in total, but no single free block is large enough for the 2 GiB request, so the allocator fails even though "reserved but unallocated" is large.

Options, in order: - Turn on expandable segments: PYTORCH_ALLOC_CONF=expandable_segments:True. The 2.15 docs name PYTORCH_ALLOC_CONF as the variable, with PYTORCH_CUDA_ALLOC_CONF kept as an alias for backward compatibility. With this setting the allocator creates segments it can later expand instead of creating new ones, so it leaves far fewer slivers of unusable memory. The docs mark it experimental and off by default, so test it. - Make shapes less varied: round sequence lengths up to a few buckets, or pack to a fixed token budget with fixed shapes. This also helps torch.compile. - max_split_size_mb stops the allocator splitting blocks above a size. The docs say it can let borderline workloads finish but should be used "as a last resort", for workloads failing with many inactive split blocks, and its performance cost ranges from zero to substantial. - Check torch.cuda.memory_summary() and memory_stats() to confirm: lots of inactive split blocks supports the fragmentation diagnosis.

Avoid fixing it by calling empty_cache() before large allocations; that hides the problem and slows every step.

Trade-offs

Expandable segments usually fix fragmentation with little cost but are marked experimental; fixed shapes waste some padding but make memory and compilation predictable.

Common mistake

Lowering the batch size until the error stops, which wastes GPU capacity without addressing fragmentation.

Follow-up

How would you confirm with a memory snapshot that fragmentation, not a single large tensor, caused the failure?

Question 8MediumMemory

Peak GPU memory jumps by 12 GB at one point in each training step and you cannot tell which code allocates it. How do you find out with PyTorch's own tools, and what will they not show you?

#

Multi-GPU job using NCCL. You can reproduce it in a few steps.

Show answer

Use the memory snapshot tools described in "Understanding CUDA Memory Usage":

1. Call torch.cuda.memory._record_memory_history(max_entries=100000) before the steps you care about. 2. Run a few steps that include the spike. 3. Call torch.cuda.memory._dump_snapshot("snapshot.pickle"), then stop recording. 4. Open the file at pytorch.org/memory_viz. The page says it does not upload snapshot data.

The visualiser shows an active memory timeline of live tensors, a stack trace for each allocation so you can click the 12 GB block and see the line that allocated it, and the allocator state history with segments, block splits and out-of-memory events.

Costs and limits: - Set max_entries. Each entry takes several KB, and snapshot files can reach gigabytes. Trace collection itself is cheap, about 2 microseconds per trace according to the docs, so it can run on real jobs. - Only memory managed by PyTorch's allocator appears. Memory taken directly through CUDA APIs or third-party libraries, such as NCCL's buffers, does not. Pinned host memory needs record_pinned_host_memory=True.

Common culprits a snapshot reveals: logits for a large vocabulary materialised in float32 for the loss, activations kept for backward that could be recomputed, an optimizer step allocating its state on the first step, and gathered parameters in sharded training.

Trade-offs

Detailed history makes the cause obvious but produces large files; short recordings around the spike keep it manageable.

Common mistake

Guessing from model size and parameter counts instead of recording where the memory is actually allocated.

Follow-up

Once you find that the loss computation causes the spike, what are your options to reduce it?

Question 9Mediumtorch.compile

You wrapped a model in torch.compile and the speed-up is close to zero. How do you find out why, and what kinds of code usually cause it?

#

The forward contains logging, a check on a tensor value, and a call to a small third-party function.

Show answer

The usual cause is graph breaks. When torch.compile meets code it cannot trace, it ends the current graph, runs that code in normal Python, and starts a new graph afterwards. Many breaks mean many small graphs, extra overhead and lost fusion, so most of the benefit disappears.

Find them: - TORCH_LOGS="graph_breaks" logs each break with its location in your code and the reason. - tlparse: run with TORCH_TRACE=/tmp/tracedir, then tlparse /tmp/tracedir for a report that answers how many graph breaks and distinct compiled regions there are. - torch.compile(model, fullgraph=True) turns any graph break into an error, which is a good way to keep a hot path clean once you have fixed it.

Typical causes: - Reading a tensor value into Python, such as .item() or if loss > x:, which makes control flow depend on data. - Printing or logging tensors inside the compiled region. - Calls into code the compiler cannot trace, such as some C extensions or unusual Python features.

Fixes: move logging and checks out of the compiled function, keep values as tensors, or use torch.cond for data-dependent branches. Then measure again: time several steps after warm-up, because the first runs include compilation.

Trade-offs

fullgraph=True guarantees one graph but forces you to restructure code; allowing breaks is convenient but can quietly remove the speed-up.

Common mistake

Measuring the first iteration, which includes compile time, and concluding torch.compile made things slower.

Follow-up

How would you compile only the hot part of a model that contains code you cannot change?

Question 10Hardtorch.compile

An inference service compiled with torch.compile is fast in tests but slow and unpredictable in production, where request lengths vary. Logs show a warning about a recompile limit. What is happening?

#

Tests used one fixed sequence length. Production sequence lengths range widely.

Show answer

torch.compile specialises compiled code with guards, such as the input's dtype, device and shape. When a guard fails, for example a new sequence length, it recompiles. Each recompile costs time on a live request. After a limit, torch.compile stops recompiling that function and runs it eagerly with a warning, so part of your traffic loses the speed-up. The relevant settings are torch._dynamo.config.recompile_limit and accumulated_recompile_limit; the troubleshooting guide's example warning shows a limit of 8.

Diagnose: - TORCH_LOGS="recompiles" shows which function recompiled and which guard failed; recompiles_verbose lists all failing guards. - TORCH_LOGS="guards" shows the generated guards. - In a tlparse report, repeated recompiles of one frame appear as [10/0], [10/1], [10/2].

Fix: - Use dynamic shapes. The default dynamic=None detects that a shape changed and compiles a more dynamic kernel on recompile; dynamic=True compiles dynamically from the start. Dynamic kernels can be a little slower than specialised ones. - Or bucket inputs: pad sequence lengths to a few fixed sizes, so there are a few specialised graphs that all stay under the limit. - Warm up every bucket before the replica takes traffic, so compilation does not happen on user requests. - Test with production-like input distributions, not one fixed length.

Trade-offs

Dynamic shapes avoid recompiles at some cost in kernel speed; bucketing keeps specialised kernels but adds padding waste.

Common mistake

Raising the recompile limit until the warning stops, which just moves compilation onto user requests.

Follow-up

How would you choose bucket sizes from real traffic so that padding waste stays low?

Question 11Hardtorch.compile

For small-batch inference you switch torch.compile to mode="reduce-overhead". Latency drops, but GPU memory rises sharply and one model shows no improvement at all. Explain both.

#

Several models in one process. One of them updates a buffer in place during forward.

Show answer

reduce-overhead reduces Python and launch overhead with CUDA graphs: it records the sequence of GPU work once and then replays it with a single launch, which helps most at small batch sizes where launch overhead dominates.

Why memory rises: a captured graph acts on the same virtual addresses every time it replays, so its memory cannot be reused freely. PyTorch's CUDA semantics notes say graphs use private memory pools that stay alive until the graph object and all tensors created during capture go out of scope, and by default each capture gets its own private pool. With several models and several input shapes, these pools add up. The torch.compile docs warn that the reduction of overhead "can come at the cost of more memory usage".

Why one model did not improve: the docs say the overhead reduction is not guaranteed and that, today, it applies only to CUDA-only graphs "which do not mutate inputs". A model that updates a buffer in place can fall outside that. Run with TORCH_LOGS=perf_hints to see why CUDA graphs were skipped.

What to do: - Keep shapes to a small set of buckets so you do not capture a graph per shape. - Remove in-place updates of inputs from the hot path where possible. - Measure memory per model, and use the default mode or max-autotune-no-cudagraphs for models where graphs do not pay off.

Trade-offs

CUDA graphs cut launch overhead and latency but pin memory per captured shape; without them you keep memory flexible and pay launch costs.

Common mistake

Turning on reduce-overhead for every model and every shape, then running out of memory.

Follow-up

When does reduce-overhead stop helping as batch size grows?

Question 12MediumDistributed

A multi-task model trained with DDP uses a different head depending on the task in each batch. Training fails or hangs in backward on some steps. Someone sets find_unused_parameters=True and it works, but slower. Why, and is there a better fix?

#

Each batch contains one task. Different ranks can get different tasks in the same step.

Show answer

DDP registers a hook on every parameter and reduces gradients in buckets as they become ready during backward. If a head was not used in this step, its parameters never get a gradient, so DDP waits for gradients that will not arrive. That is the failure.

find_unused_parameters=True makes DDP traverse the autograd graph from the forward outputs and mark parameters that received no gradient as ready, so the reduction can finish. The cost is that traversal on every iteration; the DDP design notes also say DDP still reduces all buckets, so no communication is saved.

Better options: - Make every parameter participate. For example, run all heads and multiply the unused losses by zero, or keep the heads in one combined module. Cheap heads make this nearly free. - If the set of used and unused parameters is the same every iteration, static_graph=True lets DDP learn the pattern once. The docs require that the graph stays the same, with no iteration-dependent control flow, so it does not fit a model whose heads change per batch. - Put the heads outside DDP, or use a separate process group per task, if they are large. - Make sure all ranks see the same task in a step if the design allows it.

Also check that you did not change the model's parameters after wrapping it; the docs warn that the gradient reduction functions then no longer match the parameters.

Trade-offs

find_unused_parameters is the quickest fix but adds per-step graph traversal; restructuring the model is more work but keeps DDP on its fast path.

Common mistake

Leaving find_unused_parameters=True on permanently in a model where every parameter is actually used.

Follow-up

How does FSDP2 handle parameters that some ranks do not use in a step?

Question 13MediumDistributed

You added gradient accumulation over 8 micro-batches to a DDP job, and each optimizer step is now about 8 times slower than you expected from compute alone. Profiling shows all-reduce after every micro-batch. What is wrong?

#

The code calls loss.backward() for each micro-batch and optimizer.step() every 8th.

Show answer

DDP synchronises gradients on every backward pass. Without telling it otherwise, you pay a full all-reduce for each of the 8 micro-batches, although only the final, accumulated gradient matters.

Use no_sync() for the first 7 micro-batches. Inside it, gradients accumulate locally, and they are synchronised during the first forward-backward pass after leaving the context. The docs add a warning that is easy to miss: the forward pass must be inside the context manager too, "or else gradients will still be synchronized".

``python for i, batch in enumerate(micro_batches): last = (i == len(micro_batches) - 1) ctx = contextlib.nullcontext() if last else model.no_sync() with ctx: loss = model(batch).loss / len(micro_batches) loss.backward() optimizer.step(); optimizer.zero_grad() ``

Also check the loss scaling. Dividing each micro-batch loss by the number of micro-batches gives the mean only if every micro-batch has the same number of tokens; with variable lengths, normalise by the total token count across the accumulation window and across ranks.

FSDP2 has the same idea under a different name: set_requires_gradient_sync(False) accumulates without communication, the documented equivalent of no_sync.

Trade-offs

Skipping sync saves communication but holds full local gradients in memory until the last micro-batch; syncing every step costs bandwidth but changes nothing numerically.

Common mistake

Calling forward outside no_sync and backward inside it, so DDP still all-reduces every micro-batch.

Follow-up

Why can batch 32 and batch 4 with 8 accumulation steps still give different results?

Question 14HardDistributed

A DDP job hangs at the very end of every epoch. All ranks but one have finished their data. The data comes from an IterableDataset that splits files between ranks. What is happening, and how do you fix it?

#

Files have different sizes, so ranks get different numbers of batches. The model uses SyncBatchNorm.

Show answer

DDP's gradient all-reduce is a collective: every rank must join it. When one rank has an extra batch, it runs forward and backward and waits in all-reduce for peers that have already left the loop. It waits until the collective times out.

Fixes: - Make inputs even. With map-style data, DistributedSampler pads or drops so each rank gets the same number of samples. With iterable data, cap every rank at the same number of batches (for example, the global minimum, agreed with an all-reduce at the start of the epoch), or split by records rather than files. - If uneven inputs are inherent, use DDP's join() context manager. It keeps track of ranks that have run out of data, and the Join hooks "shadow the collective communications of non-joined processes", so the remaining ranks can finish "without hanging/erroring". - SyncBatchNorm runs its own collectives in forward. The DDP docs warn that with other collectives like this, throw_on_early_termination must be enabled, which makes all ranks stop with an error when one runs out instead of hanging.

Also watch for duplicated data: an IterableDataset is replicated in every DataLoader worker on every rank, so sharding must account for both the rank and the worker ID.

Trade-offs

Dropping extra batches is simple and wastes a little data; join() uses all data but adds bookkeeping and does not cover every custom collective.

Common mistake

Raising the NCCL timeout so the hang lasts longer before it fails.

Follow-up

How would you make a training loop fail fast, with a clear error, when ranks disagree on the number of steps?

Question 15HardDistributed

A colleague applied FSDP2 by calling fully_shard(model) once on the root. The job runs, but it is slow and peak memory is close to an unsharded model. What went wrong, and how should FSDP2 be applied?

#

A decoder-only transformer with 48 blocks. The optimizer was built before fully_shard was called.

Show answer

Each fully_shard call creates one communication group: the parameters of that module that no earlier call has claimed. The group is all-gathered in one collective before forward and reduce-scattered after backward. Wrapping only the root puts every parameter in one group. The fully_shard docs describe the result: all communication happens "as two large blocking operations with no overlap with compute", and they call this "almost never what you want". Gathering everything at once also means the whole unsharded model sits in memory during forward and backward, which explains the peak.

FSDP2 has no bucket_cap_mb; group boundaries come only from which modules you wrap. So apply it bottom-up:

``python for block in model.layers: fully_shard(block) fully_shard(model) # root picks up embeddings, final norm, head optimizer = torch.optim.AdamW(model.parameters(), lr=lr) ``

Now FSDP gathers one block at a time and can prefetch the next block's parameters while the current one computes. By default reshard_after_forward frees non-root parameters after forward and keeps the root's, which the docs recommend.

Build the optimizer after fully_shard. It converts model.parameters() from plain tensors to DTensors in place, and the optimizer must run on those DTensor parameters.

After fixing the wrapping, check that step time scales with the number of GPUs, and use explicit prefetching (set_modules_to_forward_prefetch) only if profiling shows exposed communication.

Trade-offs

Finer wrapping gives more overlap and lower peak memory but more, smaller collectives; very fine wrapping can make collectives too small to use bandwidth well.

Common mistake

Treating FSDP2 as a drop-in wrapper on the root module, as if it bucketed communication automatically.

Follow-up

Where would you apply fully_shard in a mixture-of-experts model, and why?

Question 16HardDistributed

A multimodal model trained with FSDP2 hangs in backward only on steps where some ranks get text-only batches and others get image batches. Why, and what FSDP2 setting addresses it?

#

The vision encoder is skipped for text-only batches. Each encoder block is wrapped with fully_shard.

Show answer

On a text-only rank, the vision encoder's parameters get no gradient. FSDP2 reduce-scatters gradients per communication group, so the image ranks enter a reduce-scatter for the vision blocks that the text ranks never call. The collectives no longer match across ranks, and the job hangs.

FSDP2 has a setting for this: set_reduce_scatter_unused_params(True) on the affected modules. The docs say it includes zero gradients for parameters that did not receive a gradient in backward, and that this "is needed when different ranks use different parameters due to conditional control flow (e.g. multi-modal models, mixture of experts)". They compare it to DDP's find_unused_parameters. They also note that parameters with an explicit grad_dtype=None need a non-None reduce_dtype, so the zero gradients have the same dtype on every rank; otherwise backward raises an error.

Alternatives: - Make the computation uniform: run the vision encoder on a dummy input with its output multiplied by zero, so every rank touches the same parameters. Simple, but wastes compute. - Arrange batches so all ranks see the same modality in a step.

Whichever you choose, add a test that runs a few steps with deliberately mixed modalities across ranks, because these hangs often appear only with a particular data order.

Trade-offs

Reducing zero gradients costs communication on every step; uniform batching avoids it but constrains the data pipeline.

Common mistake

Debugging this as an NCCL network problem because the symptom is a collective timeout.

Follow-up

What changes for a mixture-of-experts layer where each rank routes tokens to different experts every step?

Question 17MediumData loading

Host RAM grows steadily during training until the node starts swapping. It scales with num_workers. The dataset object holds a Python list of 50 million file paths. What is going on?

#

Linux, fork start method, num_workers=16.

Show answer

Worker processes start with the parent's memory shared through copy-on-write. But every time a worker touches a Python object, it updates that object's reference count, which is a write. The page holding the object is then copied into the worker. Over many iterations each worker ends up with its own copy of the list.

The PyTorch data docs describe exactly this: after several iterations, workers "will consume the same amount of CPU memory as the parent process for all Python objects in the parent process which are accessed from the worker processes", so total memory is roughly the number of workers times the size of the parent. Large lists of file names are their example.

Fixes: - Store the index in non-refcounted structures, which the docs recommend: a NumPy array (for example, of fixed-width strings or offsets into one byte buffer), a Pandas or PyArrow table, or a memory-mapped file. - Keep only what each sample needs, for example integer IDs, and resolve paths in the worker. - Re-check num_workers: more workers than the input pipeline needs only multiplies memory.

This is not a leak in your code and it will not show up in GPU memory tools, which is why it is often misdiagnosed.

Trade-offs

NumPy or Arrow indexes avoid the copies but make the dataset code less convenient than plain Python lists and dicts.

Common mistake

Hunting for a memory leak in the training loop when the growth comes from copy-on-access in the workers.

Follow-up

How would you measure per-worker memory to confirm this before rewriting the dataset?

Question 18MediumData loading

After you set num_workers=4 on a DataLoader over a streaming IterableDataset, metrics suggest the model sees each example four times per epoch. Why, and how do you fix it, also for multi-GPU training?

#

The dataset reads records from a list of shard files in order. Training runs on 8 GPUs with DDP.

Show answer

With multi-process loading, each worker process gets a replica of the dataset object. For map-style datasets that is fine: the sampler in the main process hands out indices. For iterable-style datasets, each replica runs its own iterator from the start. The PyTorch data docs warn that "naive multi-process loading will often result in duplicated data".

Fix it in the dataset's __iter__: - Call torch.utils.data.get_worker_info() to get the worker ID and number of workers, and read only the shards assigned to this worker, for example shards[worker_id::num_workers]. - Do the same for distributed ranks: split by rank first, then by worker, so each (rank, worker) pair reads a distinct slice. With 8 ranks and 4 workers, that is 32 slices. - Use enough shards that every slice gets data, and similar shard sizes, or some ranks will run out of data earlier than others.

Also note that drop_last applies per worker replica for iterable datasets, and len(dataloader) is only a heuristic for them.

Verify by logging example IDs for one epoch and counting duplicates.

Trade-offs

Sharding by file is simple but needs many evenly sized files; sharding by record is even but harder with streaming formats.

Common mistake

Fixing duplication across workers but forgetting to shard across distributed ranks, or the other way round.

Follow-up

How would you make the split resumable from a checkpoint mid-epoch?

Question 19MediumData loading

You notice that random crops look identical across DataLoader workers, and an experiment cannot be reproduced even with the same seed. The augmentation code uses NumPy's random functions. What is wrong?

#

num_workers=8, torch.manual_seed set at start. Augmentations call np.random.

Show answer

PyTorch seeds each worker's own generator to base_seed + worker_id, but the data docs warn that "seeds for other libraries may be duplicated upon initializing workers, causing each worker to return identical random numbers". Forked workers inherit the parent's NumPy and Python random state, so every worker produces the same sequence.

Use the pattern from PyTorch's reproducibility notes:

```python def seed_worker(worker_id): worker_seed = torch.initial_seed() % 2**32 numpy.random.seed(worker_seed) random.seed(worker_seed)

g = torch.Generator() g.manual_seed(0) DataLoader(ds, num_workers=8, worker_init_fn=seed_worker, generator=g) ```

The worker_init_fn gives each worker a different seed derived from the PyTorch seed, and the generator makes the base seed, and the shuffle order, reproducible from run to run.

Alternatives: use PyTorch's own random functions in augmentations, or create a per-worker numpy.random.Generator seeded from get_worker_info().seed.

The duplicate crops reduced the augmentation's diversity by up to the number of workers, so it is worth re-running affected experiments.

Trade-offs

Explicit seeding makes runs reproducible but needs care when resuming mid-epoch; leaving seeds alone looks random and may not be.

Common mistake

Assuming torch.manual_seed controls every source of randomness in the data pipeline.

Follow-up

What else, beyond seeds, can make two runs on the same data differ?

Question 20HardReproducibility

Two runs of the same training script with the same seed produce different loss curves after a few hundred steps. A regulator asks you to make results reproducible. What can PyTorch guarantee, and how do you get as close as possible?

#

Single node, 8 GPUs, cuDNN benchmark mode on, some scatter and index operations in the model.

Show answer

Set expectations first. PyTorch's reproducibility notes say "completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms", and results may differ between CPU and GPU even with identical seeds. What you can get is reproducibility on a fixed software and hardware setup.

Steps: - Seed everything: torch.manual_seed, Python random, NumPy, and DataLoader workers with a worker_init_fn and a seeded generator. - Turn off cuDNN benchmarking: with torch.backends.cudnn.benchmark = False, cuDNN selects algorithms deterministically, possibly at a performance cost. - Call torch.use_deterministic_algorithms(True). It uses deterministic implementations where they exist and raises an error for operations known to be nondeterministic without an alternative. The docs list examples such as index_add_ on CUDA and torch.histc on CUDA. warn_only=True turns the errors into warnings, which is useful to find all the offenders in one run. - Replace or rewrite flagged operations, for example with a deterministic formulation of a scatter. - Pin the environment: PyTorch version, CUDA and cuDNN versions, driver, GPU type and number of GPUs, since the reduction order in collectives depends on the topology.

Then prove it: run twice and compare parameters bitwise after N steps, and keep that as a CI test.

Deterministic mode usually costs throughput, so you might use it for audit and debugging runs rather than every run.

Trade-offs

Determinism makes debugging and audits possible but can slow training and rule out some fast kernels.

Common mistake

Promising bitwise reproducibility across different GPU types or PyTorch versions.

Follow-up

How would you investigate a divergence between two runs if you cannot make the whole run deterministic?

Question 21MediumProfiling

A teammate's benchmark shows a new attention implementation is 100 times faster than the old one. The end-to-end step time did not change. What is wrong with the benchmark, and how should it be measured?

#

The benchmark times one call with time.time() before and after the function.

Show answer

GPU operations in PyTorch are queued and return immediately. time.time() around the call measures how long it took to enqueue the kernels, not how long they ran. The CUDA semantics notes state that time measurements without synchronisation are not accurate.

Measure properly: - Either call torch.cuda.synchronize() before starting and before stopping the timer, or use CUDA events:

``python start = torch.cuda.Event(enable_timing=True) end = torch.cuda.Event(enable_timing=True) start.record(); out = attn(q, k, v); end.record() torch.cuda.synchronize() ms = start.elapsed_time(end) ``

- Warm up first. The first calls include compilation, autotuning, memory allocation and cache effects. - Run many iterations and report a median and spread, not one number. - Use realistic shapes and dtypes, and include the backward pass if it matters for training.

For whole training steps, use torch.profiler with a schedule: wait and warmup steps before active recording steps, and on_trace_ready with tensorboard_trace_handler to save traces. Turn on record_shapes or with_stack only when needed; the docs say they add overhead, and record_shapes keeps temporary references to tensors.

If the kernel really is faster but the step time did not change, attention was not the bottleneck; the profiler timeline shows what is.

Trade-offs

Careful benchmarks take longer to write but avoid decisions based on enqueue time; quick timers are fine for CPU code and misleading for GPU code.

Common mistake

Timing GPU code with wall-clock timers and no synchronisation.

Follow-up

How would you check that the new kernel gives numerically equivalent results before switching?

Question 22MediumNumerics

A float32 model on A100 GPUs runs much slower than a colleague's identical model, and the two also give slightly different results. The difference is one line in the colleague's setup. What is it likely to be?

#

Both use float32 weights and no autocast. Both run a recent PyTorch release.

Show answer

Probably TF32. On Ampere and later GPUs, float32 matmuls and convolutions can use TF32 tensor cores, which keep the float32 range but use fewer mantissa bits internally. It is much faster and slightly less precise.

PyTorch's defaults: the CUDA semantics notes say the matmul TF32 flag torch.backends.cuda.matmul.allow_tf32 defaults to False in PyTorch 1.12 and later, while cuDNN's TF32 flag defaults to True. So float32 matmuls run in full precision unless someone opts in. Your colleague probably set torch.set_float32_matmul_precision("high") or the TF32 flag.

The docs also describe a finer-grained API available after PyTorch 2.9: torch.backends.cuda.matmul.fp32_precision and the cuDNN equivalents accept "ieee" (full FP32) or "tf32", per backend and per operator. The docs ask you not to mix the old and new settings.

What to do: - Decide per workload: TF32 is usually fine for training deep networks; keep IEEE FP32 where you need tight numerical agreement, such as some scientific code or tests comparing against references. - Set the precision explicitly in code, not by relying on defaults, and record it with the run. - Remember the setting also affects modules built on matmuls, such as nn.Linear, and that mixed precision with bf16 autocast is often a bigger speed-up than TF32.

Trade-offs

TF32 gives large speed-ups for float32 work at a small precision cost; IEEE FP32 is exact to the standard and slower.

Common mistake

Comparing two runs for performance or numerics without recording float32 matmul precision settings.

Follow-up

How would you test whether TF32 changes the final quality of your model?

Question 23HardNumerics

You must serve and lightly fine-tune a model that was pretrained in bfloat16 on GPUs that support only float16 tensor cores. In float16 the outputs contain NaNs and the GradScaler scale keeps falling below 1. What is going on?

#

Older GPUs without bf16 support. Fine-tuning uses autocast with float16 and a GradScaler.

Show answer

bfloat16 has the same exponent range as float32; float16 has a much smaller range, with a maximum of 65504. A model trained in bf16 can have activations and gradients that never had to fit in fp16. Cast to fp16, large values overflow to infinity and turn into NaNs.

PyTorch's AMP docs describe this case: AMP with fp16 may not work for every model, and "most bf16-pretrained models cannot operate in the fp16 numerical range of max 65504 and will cause gradients to overflow instead of underflow". The GradScaler then lowers its scale below 1 to bring gradients into range. GradScaler exists to stop small fp16 gradients from flushing to zero, not to fix overflow, so a scale below 1 is a sign the model does not fit fp16.

Options: - Best: use hardware with bf16 support for this model. - Keep the overflowing parts in float32: find the layers whose activations overflow (often attention scores, residual streams or final logits) and run them in fp32. Autocast already runs some ops, such as softmax, layer norm and log, in float32 on CUDA, but that may not cover your hot spots. - For inference, consider weight-only quantization that computes in a safe dtype, and validate quality. - Measure activation ranges layer by layer on real inputs before choosing.

Whatever you choose, run your quality evals; a model that stops producing NaNs can still be worse.

Trade-offs

Mixing fp32 into hot layers fixes overflow at a cost in memory and speed; changing hardware is cleanest but may not be possible soon.

Common mistake

Treating fp16 and bf16 as interchangeable because both are 16-bit.

Follow-up

How would you find which layer overflows first without reading the whole model?

Question 24HardMemory

You add activation checkpointing to a LoRA fine-tune to save memory. Memory drops, but the LoRA adapters inside checkpointed blocks stop learning: their gradients are None or zero. What is the likely cause?

#

The base model and embeddings are frozen. Checkpointing was added with torch.utils.checkpoint.checkpoint(block, hidden) and no other arguments, following an old example.

Show answer

Likely the reentrant implementation. PyTorch has two variants, chosen with use_reentrant, and the docs say it should be passed explicitly; a future release will raise an error if it is omitted. The reentrant variant has a requirement that bites LoRA setups: "At least one input and output must have requires_grad=True for the reentrant variant." Otherwise, the checkpointed part gets no gradients.

In a LoRA fine-tune, the embeddings are frozen, so the hidden states entering the first checkpointed block do not require grad. Under reentrant checkpointing, the adapters inside that block receive no gradients, even though they are trainable.

Fix: pass use_reentrant=False, which the docs recommend. The non-reentrant variant does not need inputs that require grad, records the autograd graph during forward, supports torch.autograd.grad, handles tensors inside nested structures and keyword arguments, and stops recomputing once the needed activations are rebuilt. A common older workaround was to make the input embeddings require grad; with the non-reentrant variant you do not need it.

Also check: - preserve_rng_state defaults to True, so dropout gives the same mask in recomputation. Turning it off saves some time but breaks that. - If recomputation costs too much, selective activation checkpointing lets you save expensive operations, such as matmuls, and recompute cheap ones, through a policy passed with context_fn (non-reentrant only).

Verify by checking that adapter gradients are non-zero after one step.

Trade-offs

Checkpointing saves activation memory at the cost of recomputing forward work in backward; selective checkpointing recovers some speed for some memory.

Common mistake

Copying checkpointing code without use_reentrant and assuming trainable parameters will get gradients.

Follow-up

How would you choose which operations to save under selective activation checkpointing?

Question 25HardDeployment

You exported a model with torch.export and compiled it with AOTInductor for a C++ service. It works with the example input, but fails on any other batch size, and one branch in the model raises an error at export time. How do you fix both?

#

The example input had batch size 1. The forward has if x.shape[0] > 2: and calls a helper with Python features the tracer does not support.

Show answer

Two properties of torch.export explain this. First, it aims to capture a full graph and "will error out when something untraceable is reached", unlike torch.compile, which falls back to Python at a graph break. Second, by default it assumes all input shapes are static and specialises the program to the example shapes, so the exported program will not run on other shapes, even if they are valid in eager mode.

Shapes: declare the dynamic dimensions with dynamic_shapes, for example Dim("batch") for dimension 0. Dim.DYNAMIC raises an error if a dimension turns out to be static, and Dim.AUTO gives a best-effort dynamic export without that error. Expect symbolic dimensions to be assumed not equal to 0 or 1; the docs describe this "0/1 specialization" and say it does not necessarily mean the program fails for those sizes, but test them.

Branches: data- or shape-dependent control flow like if x.shape[0] > 2 must be rewritten with control flow operators such as torch.cond. For untraceable Python in helpers, the docs suggest trying non-strict export with strict=False; draft-export lists the graph breaks you will hit.

Then compile with torch._inductor.aoti_compile_and_package() into a PT2 package and load it with aoti_load_package() in Python or the C++ runtime. The AOTInductor docs say inference inputs should match the size, dtype and stride used at export; set AOTI_RUNTIME_CHECK_INPUTS=1 while testing so mismatches raise an error instead of misbehaving.

Test the package on batch sizes 1, 2 and several larger ones before shipping.

Trade-offs

Dynamic dimensions make one artifact serve many shapes, at some cost in kernel specialisation; multiple static exports are faster per shape and more artifacts to manage.

Common mistake

Exporting with a single example input and assuming the artifact generalises like the eager model.

Follow-up

How would you version and validate exported artifacts in a release pipeline?