supraj.dev

Interview questions & answers

LLM training, fine-tuning & mathematics

Pretraining objectives, LoRA and QLoRA, distillation, losses, gradients, PCA and adaptation quality.

  • 45 questions
  • 1 easy
  • 21 medium
  • 23 hard
  • 1 related article

45 questions

Question 1HardParameter-efficient fine-tuning

What changes between LoRA and QLoRA in trainable weights, memory and quality?

#
Show answer

LoRA keeps the pretrained weight matrix frozen and learns a low-rank update through small trainable adapter matrices. QLoRA builds on this idea by storing the frozen base model in quantized form, commonly 4-bit, while computing gradients for the adapters. Quantization reduces base-model weight residency; it does not remove the need for activations, optimizer state, temporary buffers and sufficient memory for the chosen sequence length.

The original QLoRA research demonstrated fine-tuning a 65B model on a single 48GB GPU under a particular setup, not that every 70B model fits any single GPU. Select rank, precision, batch size and context based on measured peak memory and a held-out quality set.

Trade-offs

QLoRA saves VRAM but brings quantization-related quality and throughput considerations.

Common mistake

Interpreting a research benchmark as a universal hardware guarantee.

Follow-up

Why might QLoRA still run out of memory when the quantized base weights fit?

Question 2MediumPretraining objectives

How does masked-language-model pretraining teach contextual representations?

#
Show answer

Masked-language modeling hides selected tokens from an input and trains the model to predict their identities from surrounding visible context. This encourages bidirectional representations useful for tagging, classification and extracting information. BERT popularized this objective, often with a mixture of mask, random substitution and unchanged selections in its training recipe.

Masked pretraining differs from left-to-right text generation: a bidirectional encoder can inspect tokens on either side of a masked position and is not by itself an autoregressive chat decoder. In modern systems, masked, denoising and causal objectives can be combined or adapted for different tasks.

Trade-offs

Bidirectional context can help understanding tasks, while causal objectives align directly with free-form generation.

Common mistake

Calling masked-language modeling the universal training method for every LLM.

Follow-up

Why is an MLM encoder natural for classification but not directly a causal next-token chat generator?

Question 3MediumPretraining objectives

How does causal next-token training differ from masked-token training?

#
Show answer

A causal LM predicts each next token using only the preceding positions, commonly optimizing a sum of next-token cross-entropy losses. An MLM predicts selected hidden tokens while using allowed context on both sides. The attention masks and losses reflect these different objectives, creating different strengths for generation versus representation tasks.

For a model-selection exercise, ask whether you need to produce long continuations, embed documents or classify inputs. These are not mutually exclusive downstream abilities, and some architectures use span corruption or mixtures of objectives. State the intended capability before choosing an objective.

Trade-offs

Causal training supports direct generation; bidirectional representations often help token-level understanding.

Common mistake

Assuming causal pretraining means a model cannot perform classification.

Follow-up

Why does a causal decoder use triangular attention masking during training?

Question 4MediumPretraining objectives

What was BERT next-sentence prediction, and is it required for modern LLMs?

#
Show answer

In original BERT pretraining, next-sentence prediction (NSP) was a binary objective that asked whether a second text span followed the first in the sampled corpus or was a random segment. It was trained alongside masked-token prediction to support sequence-pair understanding.

NSP is a historical objective, not a required ingredient of LLMs or proof of discourse reasoning. Later BERT-style training recipes experimented with removing or changing NSP. In an interview, describe what the label meant, why researchers introduced it, and why a modern retrieval or chat model should be evaluated on its actual downstream task instead.

Trade-offs

Extra training objectives can encourage useful representations but may also bias training toward artificial sampling artifacts.

Common mistake

Claiming next-sentence prediction is mandatory for GPT-style generation.

Follow-up

What experiment would reveal whether adding NSP improves a document-pair classifier?

Question 5HardFine-tuning reliability

How do you detect and mitigate catastrophic forgetting after fine-tuning?

#
Show answer

Forgetting occurs when an adaptation improves the target task but reduces performance on previously learned or important capabilities. Preserve a baseline evaluation suite covering general instructions, safety boundaries, supported languages and the new domain. Compare before/after performance by segment, not just aggregate loss.

Possible mitigations include mixing representative general data into supervised fine-tuning, smaller updates or learning rates, regularization, parameter-efficient adapters and restricting adaptation scope. None guarantees zero forgetting. Select a strategy based on permitted training data, privacy constraints and measured regressions, then keep the earlier checkpoint for rollback.

Trade-offs

Less aggressive adaptation can preserve broad capability but limit gains on the specialized task.

Common mistake

Assuming training loss on the new dataset proves nothing important was forgotten.

Follow-up

How would you detect a Telugu-language regression after adapting an English support model?

Question 6MediumModel compression

What is knowledge distillation, and how would you verify a smaller student model?

#
Show answer

Distillation trains a student to approximate behavior of a larger teacher using soft output distributions, generated target responses, intermediate representations or combinations with ground-truth labels. Temperature-scaled soft labels can convey relative preferences among alternative classes, not only the correct class.

The goal is a cheaper or faster model that preserves the capabilities important for a given workload. Train on representative examples, then evaluate the student independently for task quality, distribution shifts, unsafe behavior, throughput and latency. It may copy the teacher’s errors, so do not treat imitation as validation.

Trade-offs

Smaller students improve serving economics, but lose capacity and may regress on rare or long-context tasks.

Common mistake

Saying distillation always requires soft labels only, or that a student is guaranteed to match teacher quality.

Follow-up

Which safety and rare-case tests would you keep even if the student matches the teacher on average accuracy?

Question 7MediumTraining quality

How do you recognize and reduce overfitting when adapting a language model?

#
Show answer

Overfitting happens when the model improves on its training examples without generalizing to independent data. Signs include falling training loss alongside rising validation loss, poorer performance on unseen topics or disproportionate memorization of distinctive passages.

Hold out examples at a document, user or time boundary that prevents near-duplicate leakage. Consider regularization, fewer update steps, smaller adapter capacity, early stopping and more representative data. For language tasks, inspect behavior on a held-out suite rather than one scalar perplexity: factual accuracy, refusals, multilingual quality and leakage risk can change independently.

Trade-offs

Strong regularization may reduce memorization and overfit while also limiting performance gains.

Common mistake

Using test data repeatedly for hyperparameter selection until it is no longer a real holdout.

Follow-up

Why is a random row-level split dangerous when many training rows come from the same document?

Question 8HardLoss functions

Why is cross-entropy a natural loss for next-token prediction?

#
Show answer

Given logits over the vocabulary at each position and the correct next-token ID, cross-entropy penalizes low probability assigned to the observed target. For a one-hot target, the token loss is minus log of the predicted probability of that token. Averaging token losses trains the model to improve next-token likelihood under the chosen training distribution.

In code, use a numerically stable cross-entropy implementation on logits rather than taking logarithms of manually normalized probabilities. Mask padding or ignored positions correctly. A low cross-entropy value does not automatically certify factuality, harmlessness or good performance on a downstream instruction-following rubric.

Trade-offs

Token likelihood is differentiable and scalable, but is only a proxy for real task usefulness.

Common mistake

Comparing loss values without accounting for tokenizer, evaluation distribution or padding.

Follow-up

How is perplexity related to average token cross-entropy, and what does it fail to measure?

Question 9HardBackpropagation

How are token embedding vectors updated by backpropagation?

#
Show answer

The embedding lookup selects rows of a trainable matrix for token IDs. The forward pass uses those vectors in subsequent layers; during backward propagation, the chain rule sends gradients from the loss through the network to the selected embedding rows. Tokens repeated in a batch contribute accumulated gradient information for their row.

The optimizer applies updates to permitted parameters. Frozen embeddings do not update, even though gradients continue through later trainable layers. Sparse-gradient optimizations are possible for some embedding modules, but depend on model and optimizer support. Inspect gradients and parameter-freeze settings before assuming a fine-tune changed vocabulary representations.

Trade-offs

Training token embeddings can improve domain adaptation while increasing trainable memory and forgetting exposure.

Common mistake

Assuming every embedding row receives the same update on every batch.

Follow-up

What happens to an embedding row when its token does not appear in a minibatch?

Question 10HardBackpropagation

Where does the Jacobian appear in transformer backpropagation, and must it be materialized?

#
Show answer

A Jacobian describes how each component of a vector-valued function changes with respect to each input component. Backpropagation composes local derivatives using the chain rule, typically as vector-Jacobian products (VJPs) in reverse-mode automatic differentiation.

Frameworks such as PyTorch do not usually build an enormous dense Jacobian for every transformer layer. Instead, autograd records the relevant computation and propagates products with upstream gradients. This is essential because model states and parameter vectors are huge. Understand shapes and VJPs before writing out matrix derivatives for layer normalization, attention or embeddings.

Trade-offs

Reverse-mode autodiff is efficient for scalar objectives but requires retaining or recomputing intermediate states.

Common mistake

Claiming practical backprop explicitly constructs one full Jacobian matrix for the model.

Follow-up

Why is reverse-mode autodiff efficient for a scalar loss with millions of parameters?

Question 11HardLinear algebra

How do eigenvectors and eigenvalues explain PCA, and when is PCA useful with embeddings?

#
Show answer

PCA centers data and finds orthogonal directions that explain the greatest variance. The eigenvectors of the covariance matrix indicate principal directions, and their eigenvalues quantify variance along those directions. Projecting data onto selected components reduces dimensionality while discarding some variance.

For embedding visualization, PCA can provide a fast projection or preprocessing stage. It is not the standard mechanism that every LLM uses to compress input tokens, and retained variance is not identical to retained semantic or retrieval quality. Measure nearest-neighbor recall or downstream task performance after any projection.

Trade-offs

Lower-dimensional vectors can save storage and computation, but may lose low-variance features critical to the task.

Common mistake

Presenting PCA as an obligatory part of a transformer tokenization pipeline.

Follow-up

Why might a PCA projection preserve 95% variance yet harm retrieval of rare legal clauses?

Question 12HardLoss functions

What does KL divergence measure, and why does its direction matter?

#
Show answer

KL divergence measures how one probability distribution differs from a reference: D_KL(P || Q) = sum over x of P(x) log(P(x)/Q(x)). It is nonnegative and asymmetric; swapping P and Q generally changes the value. It is not a distance metric and can become unbounded if the target distribution assigns zero probability where the reference has mass.

In language-model work, KL penalties can constrain updates relative to a reference policy, and KL-style targets appear in distillation. State which distribution is the reference and how probabilities are represented. Match framework conventions for log probabilities, target distributions and reduction to avoid silently optimizing the wrong direction.

Trade-offs

KL regularization can limit model drift but may also restrict useful adaptation.

Common mistake

Describing KL as symmetric or interchangeable with cross-entropy in every implementation.

Follow-up

Why can KL(P || Q) and KL(Q || P) favor different behavior when the model misses a rare event?

Question 13MediumActivation functions

What is the ReLU derivative, and are modern LLMs limited to ReLU?

#
Show answer

ReLU(x) is max(0, x). Its derivative is 1 for positive inputs and 0 for negative inputs; at zero it is mathematically undefined and implementations choose a subgradient convention, often zero. Its simple nonlinearity allows positive activations to propagate gradients without the saturation behavior of some older functions.

This does not eliminate all vanishing gradients and can create inactive or dead units. Many current transformers use other activations and gated feed-forward variants such as GELU or SwiGLU. In an interview, show the derivative accurately, then explain why activation selection and residual/normalization design both matter.

Trade-offs

ReLU is cheap to evaluate; smoother or gated activations may improve training depending on model design.

Common mistake

Claiming ReLU always prevents vanishing gradients or is the only activation used in LLMs.

Follow-up

How could a unit remain permanently inactive after a large negative update?

Question 14MediumBackpropagation

How does the chain rule connect a transformer loss to a parameter update?

#
Show answer

Each layer computes a function of earlier activations. The chain rule composes derivatives of these functions, allowing a loss gradient to propagate backward through the output projection, attention, feed-forward blocks and embeddings. Reverse-mode autodiff computes the required gradient products, while an optimizer such as Adam uses those gradients and its own state to update trainable parameters.

A concrete example: the cross-entropy gradient with respect to output logits flows through the output projection into the final hidden state, then through earlier layers. Gradient clipping, mixed precision and activation checkpointing affect stability and memory without changing the underlying derivative rule.

Trade-offs

Checkpointing recomputes activations to save memory but adds training computation.

Common mistake

Treating backpropagation and gradient descent as the same operation.

Follow-up

What is the distinction between computing a gradient and applying an optimizer step?

Question 15MediumParameter-efficient fine-tuning

Why can PEFT reduce adaptation cost, and why does it not guarantee no forgetting?

#
Show answer

Parameter-efficient fine-tuning updates a small set of parameters while keeping most of the base model frozen. LoRA adapters are a common choice; other methods can adapt prompts or selected components. This reduces optimizer-state and gradient memory relative to full-weight fine-tuning and lets teams preserve a shared base checkpoint.

A frozen base checkpoint does not guarantee unchanged behavior while an adapter is active. The adapter can shift outputs and degrade other capabilities. Evaluate the adapted model against both the new task and a frozen general-capability baseline, and retain a path to disable or replace the adapter.

Trade-offs

PEFT makes updates cheaper but can constrain adaptation capacity and still cause behavioral regressions.

Common mistake

Claiming that frozen pretrained weights make catastrophic forgetting impossible.

Follow-up

How would you measure a domain adapter’s effect on multilingual and safety performance?

Question 16EasyTraining operations

What is a hyperparameter, and how would you tune one without leaking test data?

#
Show answer

Hyperparameters are choices made outside the learned parameter set, such as learning rate, batch size, weight decay, warmup schedule, LoRA rank or generation settings. Learned weights are optimized from data; hyperparameters govern how that process runs or how outputs are sampled.

Define a search budget and a validation metric before experiments. Tune on training/validation splits, keep a genuinely untouched test set, record seeds and versions, and compare cost and variance. For deployment hyperparameters like decoding temperature, optimize task quality separately from training loss.

Trade-offs

More tuning experiments may improve performance but consume GPU budget and increase validation overfitting risk.

Common mistake

Optimizing the final held-out test set by repeatedly adjusting configuration against it.

Follow-up

What operational symptom would lead you to lower the learning rate instead of increasing the batch size?

Question 17HardDistributed training

How much GPU memory does full fine-tuning of a 70B model need, and how do you fit it?

#

You have a node with eight 80 GB GPUs and want to fully fine-tune a 70B-parameter model.

Show answer

With mixed-precision training and the Adam optimizer, a common estimate is about 16 bytes per parameter: half-precision weights and gradients plus float32 master weights and two Adam moment buffers. For 70B parameters that is roughly 1.1 TB before activations, far more than 8 × 80 GB = 640 GB.

To fit it you shard: ZeRO-style or PyTorch FSDP spread weights, gradients and optimizer states across GPUs and nodes, and activation checkpointing trades compute for activation memory. Or change the approach: LoRA or QLoRA train small adapters on a frozen, possibly 4-bit base model, which can fit on far fewer GPUs.

Trade-offs

Full fine-tuning gives the most control but needs multi-node sharding; adapters cost far less but change the model less.

Common mistake

Estimating memory from the weights alone and running out at the first optimizer step.

Follow-up

How would you estimate activation memory for a given batch size and sequence length?

Question 18MediumAlignment

What is DPO, and how is it different from RLHF with PPO?

#

You have pairs of preferred and rejected answers from user feedback.

Show answer

Classic RLHF trains a reward model on preference pairs, then optimises the language model against it with reinforcement learning such as PPO, while a penalty keeps it close to the original. That means several models and a complex, sometimes unstable loop.

Direct Preference Optimization trains on the preference pairs directly with a classification-style loss, without a separate reward model or RL loop. It is simpler and often cheaper, and it fits well when you already have good preference pairs. Quality still depends on how clean and representative those pairs are.

Trade-offs

DPO is simpler but learns only from the fixed pairs; RL methods can keep improving with fresh samples and a reward signal.

Common mistake

Training on noisy thumbs-up/down data without checking label quality.

Follow-up

How would you build preference pairs from production feedback?

Question 19MediumData

How would you turn 50,000 support conversations into a fine-tuning dataset?

#

The raw export includes agent mistakes, customer PII, duplicates and conversations that were escalated.

Show answer

Filter for quality first: keep conversations that were resolved well, drop ones with wrong answers or escalations unless you want to teach escalation, and remove near-duplicates so common cases do not dominate. Remove or mask personal data before anything leaves the source system.

Format each example in the chat structure and template the target model expects, with the assistant turns as the parts to learn. Split train and evaluation sets by time or customer, not randomly, so similar conversations do not leak into both. Have experts review a sample, and keep the dataset versioned so you can trace a model back to its data.

Trade-offs

Strict filtering shrinks the dataset but raises quality; a smaller clean set often beats a large noisy one.

Common mistake

Training on raw transcripts, including the human agents' mistakes and customers' personal data.

Follow-up

How would you check that the evaluation set does not overlap the training set?

Question 20MediumAdaptation choice

A ticket-triage feature works well with a long prompt on a large hosted model, but it costs too much and is slow at 2 million calls a day. Someone says "fine-tune it". When is that the right call, and what would you need first?

#

The task is narrow: read a ticket, pick one of 40 queues, extract four fields as JSON. The prompt carries 30 examples and a page of rules.

Show answer

This is the case where fine-tuning usually earns its cost. The behaviour is narrow and stable (a fixed label set and a fixed output shape), the volume is high, and today you pay for the same 30 examples and page of rules on every one of 2 million calls. A fine-tuned smaller model can learn that behaviour into its weights, so the prompt shrinks to the ticket itself and you can serve a model that is much cheaper and faster per call.

Before training, get three things. First, a labelled evaluation set from real tickets, split by time so the test is newer than the training data, with per-queue accuracy and field-level exact match. Second, the current prompted large model's score on that set: that is the bar the fine-tune must meet. Third, a training set, which you can often bootstrap by having the large model label historical tickets and having people review a sample, especially the rare queues.

Then compare like with like: the fine-tuned small model against the prompted large model and against the same small model prompted, on accuracy, cost per thousand tickets and latency. Plan for change: if queues are added monthly, you need a retraining pipeline and a way to route unknown cases to a fallback. If the label set changes weekly or the answer depends on fresh facts, keep prompting or retrieval instead.

Trade-offs

A fine-tuned small model is far cheaper per call but adds a training pipeline, data upkeep and a release process; the prompted large model changes in minutes.

Common mistake

Fine-tuning before an evaluation set and a prompted baseline exist, so nobody can say whether the new model is better.

Follow-up

How would you handle a new queue that appears after the model is trained?

Question 21HardLoRA

Your LoRA fine-tune on attention projections only is clearly worse than a full fine-tune on the same data. Which knobs do you change first, and what do you expect each to do?

#

Rank 8, adapters on q_proj and v_proj, learning rate copied from a full fine-tuning recipe.

Show answer

Check three things in order.

Where the adapters are. Adapters on only two attention projections give LoRA very little to work with. QLoRA applied adapters to all linear layers, and Hugging Face PEFT supports this directly: target_modules="all-linear" selects every linear layer except the output layer, which is easier than naming modules that differ by architecture. Adding the MLP projections is usually the first thing to try.

The learning rate. Adapters start from zero and are small, so they normally need a higher learning rate than full fine-tuning. TRL's SFT docs suggest about 1e-4 for adapters, while its SFTConfig default (meant for full fine-tuning) is 2e-5. A copied full-fine-tune learning rate often leaves LoRA under-trained.

Rank and scaling. Classic LoRA scales the update by lora_alpha / r, so raising r without changing alpha shrinks each update. PEFT's rsLoRA option (use_rslora=True) scales by lora_alpha / sqrt(r), which the docs say stabilises adapters and makes higher ranks more useful. DoRA (use_dora=True) can help at low ranks at the cost of more overhead.

Keep expectations honest. In "LoRA Learns Less and Forgets Less" (Biderman et al.), LoRA substantially underperformed full fine-tuning on code and maths in standard low-rank settings, and full fine-tuning learned weight changes with rank 10 to 100 times higher than typical LoRA configurations. If the gap remains after these fixes, the task may need more capacity than an adapter gives.

Trade-offs

More target modules and higher rank close the gap to full fine-tuning but use more memory and may forget more of the base model.

Common mistake

Raising the rank while keeping alpha fixed under classic scaling, which makes each update smaller, then concluding that rank does not help.

Follow-up

How would you tell whether the remaining gap is a capacity problem or a data problem?

Question 22HardLoRA

You need a model that is much better at your company's programming language but must not lose general chat and reasoning ability. Do you pick LoRA or full fine-tuning?

#

About 20 billion tokens of internal code are available, plus a smaller instruction set. Both quality on the new language and general ability matter.

Show answer

This is the exact trade-off studied in "LoRA Learns Less and Forgets Less" (Biderman et al.). They compared LoRA and full fine-tuning on programming and mathematics, with both continued pretraining (20 billion unstructured tokens) and instruction fine-tuning (about 100,000 prompt-response pairs). In standard low-rank settings, LoRA substantially underperformed full fine-tuning on the target domain. But LoRA better kept the base model's performance on tasks outside that domain, and mitigated forgetting more than common regularisers such as weight decay and dropout.

So the choice depends on which failure costs more. If deep skill in the new language is the point and you have this much domain data, full fine-tuning (or continued pretraining) is more likely to get there, and you must actively protect general ability: mix general data into training, keep a general-capability eval suite, and stop or roll back on regression. If general ability is critical and the gain needed is modest, LoRA is the safer start and cheaper to run.

Either way, decide with measurements, not belief. Build two eval suites before training: one for the internal language (compile and test pass rate on held-out tasks) and one for general ability. Run a small pilot of both methods and plot domain gain against general loss.

Trade-offs

Full fine-tuning learns more of the new domain but forgets more and costs more; LoRA forgets less and is cheaper but may plateau below the target.

Common mistake

Assuming LoRA is a free, equivalent shortcut to full fine-tuning, or assuming LoRA cannot cause forgetting at all.

Follow-up

How much general data would you mix into continued pretraining, and how would you choose that ratio?

Question 23MediumDeployment

You trained a QLoRA adapter on a 4-bit base model, merged it into the BF16 base for serving, and the served model scores a little differently from your training evals. Is that a bug, and how should you ship QLoRA adapters?

#

Evals during training ran on the 4-bit base plus adapter. Production runs the merged BF16 model on vLLM.

Show answer

It is expected rather than a bug. In QLoRA the frozen base is stored in 4-bit NormalFloat and gradients flow through that quantized model into the adapters. The adapter therefore learned to correct the outputs of the quantized base. When you merge it into the original BF16 weights, it is applied to slightly different base weights, so behaviour shifts a little. Usually the shift is small, but it is a different model from the one you evaluated.

So treat the artifact you serve as the thing you evaluate. Run the release eval on the exact merged (or adapter-on-base) model and serving stack you will deploy, with the production chat template and decoding settings, not on the training-time setup.

Two practical choices. You can serve the adapter unmerged on top of the base, which keeps one base for many adapters; or you can merge with PEFT's merge_and_unload(), which removes adapter latency (and note that it returns a new model rather than changing it in place, so use the returned object). If the quantization gap matters for your task, PEFT documents LoftQ initialisation, which initialises LoRA weights to reduce quantization error when training on a quantized base.

Trade-offs

Serving unmerged adapters keeps one shared base and fast swaps; merging gives simpler, faster inference per model but one deployment per fine-tune.

Common mistake

Signing off a release from training-time evals on a different base precision and template from production.

Follow-up

When would you serve many adapters on one base instead of merging each one?

Question 24MediumSFT data

After supervised fine-tuning on support chats, the model sometimes writes the customer's side of the conversation or repeats the system prompt. What in the training setup causes that?

#

The data is in conversational messages format with system, user and assistant turns, trained with TRL SFTTrainer defaults.

Show answer

The model was probably trained to predict every token in the conversation, including system and user turns, so it learned to produce them. With TRL's SFTTrainer, the loss target depends on the dataset type. For prompt-completion datasets the default is to compute loss on the completion only. For language-modelling datasets, which include a plain messages conversation, the default is loss on the full sequence.

Fix it by training on assistant tokens only. In TRL that is assistant_only_loss=True on a conversational dataset. The docs warn that this needs a chat template with {% generation %} and {% endgeneration %} markers around assistant content; TRL patches templates for some known families, but for other models you must check the template, otherwise the mask cannot be built. Alternatively, convert the data to prompt-completion form, where everything before the final assistant reply is the prompt.

Verify before a long run: tokenise a few examples, print which tokens have labels (TRL uses -100 to ignore positions), and confirm that only assistant text, including its end-of-turn token, is learned.

Trade-offs

Training on the full sequence uses more signal and can help with very small datasets; masking keeps the model in the assistant role, which is what chat fine-tunes need.

Common mistake

Assuming the trainer masks prompts automatically for every dataset format.

Follow-up

Would you also mask earlier assistant turns in a multi-turn example, and why?

Question 25MediumSFT data

Your fine-tuned model answers correctly and then keeps going: it invents a new user turn, or generates until max_tokens. Training loss looked fine. What do you check?

#

A base model was fine-tuned with a chat template. The tokenizer had no pad token, so the training script used the EOS token for padding.

Show answer

The model never learned when to stop. Check three things.

The end-of-turn token. Chat models stop when they produce the template's end-of-turn token, which is not always the tokenizer's default EOS. TRL's SFT docs give exactly this case: for some base models with a predefined chat template, you must align the EOS token with the template (for example setting eos_token="<|im_end|>" for Qwen2.5 base models) so responses terminate correctly. If training ends assistant turns with one token and serving stops on another, generation runs on.

Padding with EOS. When a tokenizer has no pad token, TRL uses the EOS token for padding, and padding positions are masked out of the loss. If your own preprocessing masks by token ID rather than by position, it can also mask the real EOS at the end of every answer, so the model is never trained to emit it.

Truncation. If long examples are cut at max_length (TRL SFT defaults to 1,024 tokens, keeping the start), the end of the answer and its stop token are lost for those examples.

The quick test: tokenise one training example, look at the last labelled tokens, and confirm the end-of-turn token is there with a real label. At serving time, make sure the server's stop tokens match the template.

Trade-offs

Adding a dedicated pad token is cleanest but may need an embedding resize; reusing EOS for padding is common but needs position-based masking.

Common mistake

Masking padding by token ID when the pad token is the same as EOS, which hides every real stop token from the loss.

Follow-up

How would you catch this in an automated check before a training run starts?

Question 26MediumSFT data

A fine-tune for writing long incident reports produces reports that stop abruptly around the same length. The data has full reports. Where would you look?

#

Training used TRL SFTTrainer with mostly default settings. Many target reports are 1,500 to 3,000 tokens.

Show answer

Look at truncation. TRL's SFTConfig has max_length defaulting to 1,024 tokens, and sequences longer than that are truncated keeping the start. With a prompt plus a 2,000-token report, the trainer keeps the first 1,024 tokens and drops the rest, including the end of the report and its stop token. The model sees many reports that seem to break off and learns that pattern, and it never sees how good long reports finish.

Fix it by measuring the token length distribution of your formatted training data (with the real chat template), then setting max_length to cover it, or None for no truncation if memory allows. If memory is the limit, use gradient checkpointing (on by default in TRL's SFTConfig), a smaller micro-batch with gradient accumulation, or padding-free training. Do not keep long examples truncated just to fit.

Add a pre-training check that counts how many examples would be truncated and fails the run above a small threshold. The same default exists in TRL's DPO config, where truncation can cut the chosen and rejected answers differently.

Trade-offs

A longer maximum length keeps whole examples but raises memory per step; dropping or splitting very long examples is cheaper but changes what the model learns about length.

Common mistake

Trusting library defaults for sequence length without measuring your own token lengths.

Follow-up

How would truncation in a preference dataset bias DPO toward shorter or longer answers?

Question 27HardEfficiency

Turning on sequence packing made fine-tuning twice as fast, but quality dropped on short examples. What can packing do wrong, and how do you pack safely?

#

Most examples are 100 to 300 tokens; sequence length is 4,096. The run used a simple concatenate-and-cut packer.

Show answer

Packing puts several examples into one fixed-length sequence so little compute is wasted on padding. Two things can go wrong.

Cross-contamination. If packed examples share one causal attention mask, tokens of the second example can attend to the first, which is unrelated. The model learns from fake context and its attention patterns no longer match inference, where each request starts fresh.

Cut examples. A naive packer concatenates everything and cuts every 4,096 tokens, so some examples are split across two sequences: the second half has no prompt, and the first half has no end.

Pack with boundaries. TRL's SFT trainer offers packing=True with a packing_strategy: the default bfd (best-fit decreasing) fits whole sequences into bins and truncates overflow, bfd_split splits overflow sequences, and wrapped is the aggressive mode that cuts mid-sequence. With bfd, TRL turns on padding-free mode, which flattens sequences and, according to the docs, is currently supported only with FlashAttention 2 or 3, which can keep the flattened examples separate. If your attention kernel cannot handle sequence boundaries, do not pack.

Then verify with an A/B on a held-out set, particularly short examples, before keeping the speed-up.

Trade-offs

Packing can cut training time substantially on short data; boundary-aware packing needs supported attention kernels, and naive packing trades speed for subtle quality loss.

Common mistake

Measuring the speed-up from packing but not re-running the quality eval.

Follow-up

How does packing interact with gradient accumulation and per-token loss averaging?

Question 28HardTraining loop

With batch 32 on 8 GPUs the run converges one way; with batch 4 and 8 gradient-accumulation steps it converges differently, even though the effective batch is the same. Why can they differ for causal LM fine-tuning?

#

Examples vary a lot in length, and padding is masked out of the loss.

Show answer

For token-level training the correct loss over a step is the sum of the token losses across all micro-batches, divided by the total number of non-padding tokens in those micro-batches. A naive accumulation loop averages within each micro-batch and then averages those averages. With varying lengths, that is a different number: tokens in short micro-batches get more weight than tokens in long ones. Hugging Face described exactly this in "Fixing Gradient Accumulation" (16 October 2024): the default loss functions in model code were not built for accumulation, so Trainer results with accumulation did not match full-batch training, and they fixed the Trainer.

The same issue exists across devices: each GPU averaging over its own tokens before the gradient all-reduce gives a different result from averaging over all tokens. TRL configs expose average_tokens_across_devices (default True) for this.

In your own loop, count the non-padding label tokens across all micro-batches (and ranks) for the step first, sum the token losses, and divide by that total once. Then test: run one step both ways with fixed seeds and compare the gradients.

Trade-offs

Exact token averaging needs the step token count up front (one extra pass or all-reduce); per-example averaging is simpler but weights examples, not tokens.

Common mistake

Dividing each micro-batch loss by the number of accumulation steps and assuming it equals the full-batch loss.

Follow-up

Is it always wrong to weight by example instead of by token? When might you want that?

Question 29HardPreference tuning

After a DPO run, reward margins on the training dashboard look great, but people prefer the new model less and its answers got longer. How do you read the DPO metrics and fix the run?

#

TRL DPOTrainer with default beta, preference pairs where the chosen answer is usually longer, no SFT on the chosen answers first.

Show answer

Know what the dashboard shows. In TRL, rewards/chosen and rewards/rejected are implicit rewards: beta times the log-probability ratio between the policy and the reference model. rewards/margins is their average difference, and rewards/accuracies is how often chosen beats rejected. A large margin means the model separates the training pairs, not that its free-form answers are better. TRL's docs also note that DPO usually widens the margin by pushing down the likelihood of rejected answers rather than raising chosen ones, so check logps/chosen too: if it falls, the model may be drifting away from good answers as well.

The length growth is a data signal. If chosen answers are usually longer, the cheapest way to fit the preferences is to be longer. Fixes: balance pair lengths in the data, use a length-normalised loss (TRL's sigmoid_norm follows SimPO's normalisation by token count), and judge with length-controlled comparisons.

Other levers: beta (default 0.1) controls how far the policy can move from the reference, and higher means less deviation; run SFT on in-distribution answers first so the reference model is close to the data; and use a held-out human or judged win-rate, not training margins, as the stop signal.

Trade-offs

Higher beta keeps the model safer and closer to the reference but learns the preferences less; length normalisation removes a bias but can under-reward detail that is genuinely needed.

Common mistake

Treating rising reward margins or accuracies as proof the model improved.

Follow-up

How would you build a preference dataset that does not reward length by accident?

Question 30MediumPreference tuning

Your product logs thumbs-up and thumbs-down on single answers, not side-by-side comparisons. The team wants to "do DPO". Which preference method fits the data you actually have?

#

About 200,000 rated answers, mostly thumbs-up; very few prompts have both a good and a bad answer.

Show answer

DPO needs pairs: one prompt with a chosen and a rejected answer. Thumbs on single answers are not pairs, and building pairs by matching prompts throws most of the data away or invents comparisons that never happened.

KTO (Ethayarajh et al.) is designed for this shape. It needs only a binary signal of whether an output is desirable or undesirable, and the paper reports that it matches or exceeds preference-based methods at scales from 1B to 30B parameters. That makes it a natural first choice for thumbs data. Account for imbalance: if most ratings are positive, the small set of negatives carries a lot of the learning, so check those for quality.

If you later collect real comparisons, DPO is the standard pairwise method: it solves the RLHF objective with a simple classification loss, with no separate reward model and no sampling during training. ORPO is worth knowing when you want one stage instead of SFT followed by preference tuning: it is reference-model-free and adds an odds-ratio preference term during SFT.

Whatever the method, the feedback is biased (people rate when annoyed or delighted, and rarely otherwise), so clean the data, remove ratings that reflect tool outages rather than answer quality, and judge the result on a held-out eval.

Trade-offs

Binary feedback is cheap and plentiful but noisier per example; pairwise comparisons are more informative but cost more to collect.

Common mistake

Forcing unpaired thumbs data into fake pairs so a familiar method can be used.

Follow-up

How would you collect a small set of true pairwise comparisons cheaply to check the KTO model?

Question 31HardReinforcement learning

You want to improve a model at writing SQL that must pass test queries. Why would you consider GRPO with a test-based reward instead of DPO, and what goes wrong in practice?

#

There are 20,000 tasks, each with a database fixture and expected results. No human preference data exists.

Show answer

When correctness can be checked by a program, you do not need human preferences: sample answers, run them, and reward the ones that pass. That is reinforcement learning with a verifiable reward. GRPO (Group Relative Policy Optimization, from the DeepSeekMath paper) is a variant of PPO built for this. Instead of training a separate value model, it samples several outputs for the same prompt and uses their average reward as the baseline, so each answer is rewarded for being better than its siblings. The paper presents it as reducing PPO's memory use, and it regularises by adding the KL divergence to the reference policy directly to the loss.

What goes wrong: - Reward hacking. The model finds answers that pass weak checks without being right, for example queries that return the expected rows only for this fixture. Use several fixtures per task, hidden tests and occasional human review. - No learning signal. If all sampled answers for a prompt fail (or all pass), the group has no spread and that prompt teaches nothing. Choose tasks the model solves sometimes, and add harder ones over time. - Cost. Each step generates several samples and executes them, so rollout speed, not the gradient step, often sets the pace. - Drift. Keep a general eval and watch the KL term so the model does not lose other skills.

DPO fits when you have pairs of better and worse answers; here a test runner gives a better signal than pairs.

Trade-offs

Verifiable rewards scale without human labelling but only reward what the checks measure; preference data captures quality the checks miss but costs people time.

Common mistake

Trusting the training reward curve without auditing what the passing answers actually do.

Follow-up

How would you detect reward hacking automatically during the run?

Question 32HardReinforcement learning

During RLHF, the reward model score keeps rising but human ratings of samples peaked halfway through and are now falling. What is happening, and how do you choose when to stop?

#

PPO against a learned reward model with a KL penalty. Checkpoints are saved every 200 steps.

Show answer

This is reward model overoptimization, an example of Goodhart's law. The reward model is a learned approximation of human preferences; once the policy is pushed hard against it, the policy finds outputs the proxy scores highly but people do not like (padding, flattery, confident tone, odd phrasings). Gao et al. studied this with a fixed "gold" reward model standing in for humans and a proxy trained on its labels: optimising the proxy too much hurt gold performance, and the shape of that curve changed smoothly with reward model size.

What to do: - Choose the checkpoint on held-out human or gold evaluation, not on training reward. You already have checkpoints every 200 steps, so pick the one near the human-rating peak. - Watch the KL divergence from the reference model. Quality often peaks at moderate KL; large KL means the policy has moved far from where the reward model was trained to be reliable. - Improve the proxy: a larger reward model, fresh preference labels on samples from the current policy, and targeted tests for known hacks such as length. - Keep a fixed set of prompts with human-rated samples for every checkpoint, so you see the turn early.

Trade-offs

Stopping early leaves some reward on the table but protects real quality; a stronger KL penalty limits hacking but also limits improvement.

Common mistake

Picking the final checkpoint because it has the highest reward score.

Follow-up

How would you refresh the reward model during training without destabilising it?

Question 33MediumDistributed training

An 8B full fine-tune does not fit with plain DDP on 8 GPUs. Someone jumps straight to ZeRO stage 3 with CPU offload. Is that the right first move?

#

Eight 80 GB GPUs in one node with NVLink. The run is limited by memory, not by time yet.

Show answer

Start with the smallest change that fits, because each step up costs speed.

DDP keeps a full copy of the weights, gradients and optimizer state on every GPU. DeepSpeed's ZeRO tutorial describes the stages as partitioning more of that state across processes: stage 1 partitions optimizer states, stage 2 also partitions the 16-bit gradients, and stage 3 also partitions the 16-bit parameters. Optimizer state is usually the largest piece (Adam keeps two moments per parameter, plus FP32 master weights in mixed precision), so stage 1 or 2 is often enough for a model of this size on 80 GB GPUs. Stage 3, like PyTorch FSDP's full sharding, must gather parameters before each layer's forward and backward, which adds communication; inside one NVLink node that is usually affordable.

Offload is the last step. ZeRO-Offload moves optimizer and gradient states to CPU memory, and ZeRO-Infinity can move full model state to CPU or NVMe. That frees GPU memory but sends data over PCIe and runs the optimizer on the CPU, which can slow steps a lot.

Also check activations: long sequences may be the real problem, which activation checkpointing addresses at a compute cost. Measure peak memory and step time for each option, and pick the fastest one that fits.

Trade-offs

Higher ZeRO stages and offload fit larger models on fewer GPUs at the cost of step time; buying more GPUs is faster but costs money.

Common mistake

Enabling maximum sharding plus CPU offload by default and paying a large slowdown that the model size did not require.

Follow-up

When does ZeRO stage 3 across nodes become slower than adding tensor parallelism?

Question 34HardDistributed training

You moved a training job to PyTorch FSDP2 (fully_shard). It fits in memory but is slower than expected, and the loss curve is slightly worse than the old FSDP1 run. Which FSDP2 settings would you look at?

#

Default fully_shard on every transformer block and the root module, BF16 parameters via MixedPrecisionPolicy(param_dtype=torch.bfloat16).

Show answer

FSDP2 shards each parameter on dimension 0 across the data-parallel ranks, as DTensors. Two settings matter here.

reshard_after_forward. With True, FSDP frees the gathered parameters after forward and all-gathers them again in backward: less memory, more communication. With False, it keeps them from forward to backward and avoids the second all-gather. The docs suggest False for the root module, because its parameters are needed again right away in backward. An integer gives a middle ground, resharding to a smaller group at the cost of more memory than True. If you have memory headroom, try False for the root and measure the step time.

The gradient reduction dtype. MixedPrecisionPolicy has param_dtype (the dtype of unsharded parameters used for compute and all-gather) and reduce_dtype (the dtype for gradient reduce-scatter or all-reduce). Reducing gradients in BF16 across many ranks loses precision compared with FP32 reduction, which can show up as a slightly worse curve. Set reduce_dtype=torch.float32 and compare. The 2.15 docs also note that with reduce_dtype=None, gradients now follow the parameter's grad_dtype rather than param_dtype.

Change one setting at a time, and compare against the old run on throughput (tokens per second) and loss at the same step.

Trade-offs

Keeping parameters gathered saves communication but costs memory; FP32 gradient reduction is more accurate but sends twice the bytes of BF16.

Common mistake

Comparing an FSDP2 run with an FSDP1 run whose mixed-precision settings were different, and blaming the framework.

Follow-up

When would you add CPUOffloadPolicy, and what does it do to the optimizer step?

Question 35HardParallelism

A 70B pretraining job on 64 GPUs (8 nodes of 8) uses tensor parallelism 16 and data parallelism 4. GPU throughput is poor. What layout would you try, and why?

#

Each node has NVLink between its 8 GPUs and InfiniBand between nodes.

Show answer

Tensor parallelism of 16 spans two nodes, so the all-reduces inside every layer, in forward and backward, cross the slower inter-node network. That is the usual cause of poor throughput in this layout.

The Megatron-LM paper (Narayanan et al.) gives the guidance: on servers with g GPUs, tensor model parallelism should generally be used up to degree g, and pipeline parallelism is then used to scale across servers. Here that means TP of 8 inside each node, then pipeline and data parallelism across nodes. Their combined approach reached 502 petaFLOP/s on 3,072 GPUs for a 1-trillion-parameter model, 52% of theoretical peak per GPU, showing how well the layered layout can work.

A starting point: TP 8 (inside a node), then choose pipeline stages so the model fits (for example PP 2), and use the remaining factor for data parallelism (DP 4 here), possibly with sharded optimizer state. Pipeline parallelism adds idle time (the bubble), so use enough micro-batches.

Then measure. Compare tokens per second and model FLOPs utilisation for two or three layouts on a short run, and check the profiler for time in communication.

Trade-offs

Tensor parallelism inside a node is fast but limited to the node size; pipeline parallelism scales across nodes but adds bubble time and scheduling complexity.

Common mistake

Choosing parallel degrees only so the model fits, without matching tensor parallelism to the fast interconnect domain.

Follow-up

How would sequence or context parallelism change this plan for 128K-token training?

Question 36HardParallelism

A job uses 8 pipeline stages and 8 micro-batches per step. Profiling shows GPUs idle a large part of each step. Quantify the problem and say how you would reduce it.

#

A simple pipeline schedule (no interleaving); global batch size is fixed by the training recipe.

Show answer

In a pipeline, each stage waits at the start of a step for the first micro-batch to arrive and goes idle at the end while the last ones drain. The Megatron-LM paper gives the bubble time as a fraction of ideal compute time: (p - 1) / m, where p is the number of pipeline stages and m is the number of micro-batches per batch. With p = 8 and m = 8, that is 7/8: the pipeline spends almost as long idle as it spends computing, so you lose close to half of the GPU time.

Ways to shrink it: - More micro-batches. With the global batch fixed, use a smaller micro-batch size so m grows. With m = 64 the fraction is 7/64, about 11%. Very small micro-batches can lower per-kernel efficiency, so measure. - Fewer stages. If memory allows, use fewer pipeline stages and more tensor or data parallelism. - Interleaved schedule. Megatron-LM's interleaved schedule gives each device several smaller model chunks, which reduces the bubble by the number of chunks per device, v, but increases communication by v as well. The paper reports this schedule improving throughput by more than 10%.

Use the profiler to confirm the idle time is the bubble, not slow communication or a slow stage that is unbalanced.

Trade-offs

More micro-batches shrink the bubble but each is smaller and less efficient; interleaving cuts the bubble but adds communication.

Common mistake

Adding pipeline stages to fit a model without raising the micro-batch count, so most of the extra GPUs sit idle.

Follow-up

How do you balance layers across stages when the first and last stages also hold the embedding and output layers?

Question 37MediumCheckpointing

After a crash, the run resumed from the last checkpoint, but loss jumped and then the learning rate looked wrong on the dashboard. What must a checkpoint contain for a true resume?

#

The custom training loop saved only model.state_dict() every 1,000 steps.

Show answer

Saving only the weights restarts training from those weights with a fresh optimizer and schedule, which is not a resume.

A proper checkpoint holds: - Model weights. - Optimizer state. Adam's moment estimates take many steps to build; starting from zero causes a loss jump. - Learning-rate scheduler state, including the step count, or warmup restarts and the schedule shifts. - Gradient scaler state if you use FP16. - Data position: which samples or shards were consumed and the sampler's shuffle seed, so the run neither repeats nor skips data. - Random number generator states, if you need reproducibility. - The global step and the config, including world size and batch settings.

Tools help. Hugging Face Trainer's resume_from_checkpoint restores model, optimizer and scheduler states, and has an ignore_data_skip option (off by default) that controls whether it skips the batches already seen. For sharded models, PyTorch Distributed Checkpoint provides get_state_dict and set_state_dict helpers that return and load both model and optimizer state, and it writes multiple files per checkpoint, at least one per rank, in parallel.

Test it: kill a short run on purpose, resume, and check that the loss and learning-rate curves continue smoothly.

Trade-offs

Full checkpoints are several times larger than weights alone and slower to write, but anything less turns a crash into a partial restart.

Common mistake

Never testing resume until the first real crash, weeks into a run.

Follow-up

How do you resume the data loader exactly when it streams from many shards with multiple workers?

Question 38HardCheckpointing

You lost two of sixteen nodes and replacements will take days. Can you resume the run on fourteen nodes from a checkpoint written on sixteen, and what else changes?

#

FSDP-sharded model and optimizer, checkpoints written with PyTorch Distributed Checkpoint (DCP).

Show answer

Loading is the easier part. PyTorch Distributed Checkpoint supports load-time resharding, which the docs describe as saving in one cluster topology and loading into another. DCP stores tensors with enough metadata that each new rank can read the pieces it needs. (A plain torch.save of each rank's shard would not reshard this way.) DCP loads in place, so you first build the model and optimizer with the new sharding, then load into them.

The recipe is the harder part. With fewer GPUs, keep the global batch size the same if you can, by raising gradient accumulation or per-GPU micro-batch, so the learning-rate schedule and optimizer dynamics still match. If the global batch has to change, the learning rate may need adjusting and you should record the change. Check that the parallel layout still divides cleanly (14 nodes may not fit a tensor-by-pipeline-by-data layout that assumed 16), and that the data sampler continues from the right position with the new number of data-parallel ranks.

Expect each step to be slower, and update the end date. When the nodes come back, you can reshard again to sixteen the same way.

Trade-offs

Resuming smaller keeps the run moving but takes longer and may change the recipe; waiting keeps the recipe exact but leaves the cluster idle.

Common mistake

Changing the global batch size silently when the world size changes, then comparing loss curves across the gap as if nothing happened.

Follow-up

How would you design checkpoints so a resume on a different parallel layout is routine rather than an emergency?

Question 39HardReliability

You are planning a two-month training run on a few thousand GPUs. Leadership asks how often it will be interrupted and what "good" looks like. What do you tell them and how do you design for it?

#

Checkpoints currently take 20 minutes to write synchronously, and restarts are manual.

Show answer

Expect frequent interruptions, mostly from hardware. The Llama 3 paper reports a 54-day snapshot of its 405B pre-training with 466 job interruptions: 47 planned and 419 unexpected. About 78% of the unexpected ones were attributed to confirmed hardware issues, with GPU issues alone accounting for 58.7% of unexpected interruptions. Even so, Meta reports more than 90% effective training time. That is a reasonable "good" target to quote, with the context that it took a lot of automation.

Design for failure: - Automatic restart. torchrun's --max-restarts stops and restarts all workers when any fail, but recovery depends on your script loading its last checkpoint; the docs say to make sure you checkpoint your progress. Add automation that removes bad nodes and brings in spares. - Cheaper, more frequent checkpoints. With 20-minute synchronous saves you checkpoint rarely and lose hours per failure. Use sharded checkpoints written by every rank in parallel, and asynchronous saving (PyTorch DCP has async_save, which stages the state and writes it in a background thread). - Fast diagnosis. Meta used PyTorch's NCCL flight recorder, which keeps collective metadata and stack traces in a ring buffer and can dump them on a watchdog or heartbeat timeout, to find which rank hung. - Health checks before rejoining a node.

Report effective training time weekly: time making progress divided by wall-clock time.

Trade-offs

Frequent checkpoints reduce lost work but cost I/O and storage; hot spares cost idle GPUs but cut recovery time.

Common mistake

Planning the schedule as if the run will go uninterrupted, then losing days to manual restarts.

Follow-up

How would you pick the checkpoint interval from the failure rate and the save time?

Question 40HardStability

Three weeks into pretraining, the loss suddenly spikes and does not come back down. What do you do in the next hour, and how do you avoid repeats?

#

Checkpoints every 500 steps; gradient norms jumped just before the spike; the data loader is deterministic.

Show answer

First, stop wasting compute: pause the run and keep the checkpoints from before the spike.

The PaLM paper describes a practical recovery. Their largest model had loss spikes that did not line up with obviously bad data. They restarted from a checkpoint roughly 100 steps before the spike and skipped roughly 200 to 500 data batches, covering the batches seen before and during the spike; the spike did not recur at that point. They also found that re-running the same batches from an earlier checkpoint did not reproduce the spike, which suggests an interaction between the data and the model state at that moment, not simply corrupt data.

So: rewind to the last good checkpoint before the gradient norm started rising, skip the window of batches around the spike, and resume while watching gradient norm and loss closely. Your deterministic data loader makes skipping exact.

To reduce repeats, log per-step gradient norms and alert on jumps; keep gradient clipping; review the learning-rate schedule if spikes cluster at high learning rates; and inspect the skipped batches anyway, since sometimes they do contain broken or duplicated data. Record each intervention in the run log so the final model's history is clear.

Trade-offs

Rewinding and skipping is fast and cheap but drops some data; lowering the learning rate is more stable but slows learning for the rest of the run.

Common mistake

Letting a diverged run continue in the hope it recovers, burning days of GPU time.

Follow-up

How would you automate detection and rewind so a spike at 3 a.m. does not wait for a person?

Question 41HardEvaluation

You fine-tuned an aligned chat model on harmless internal data. The new model now answers requests it used to refuse. How is that possible, and what do you add to the release process?

#

The training data is customer-support chats with no harmful content. Only task metrics were checked before release.

Show answer

Fine-tuning can weaken safety behaviour even with harmless data. Qi et al. ("Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!") showed two things. With adversarial data, they broke GPT-3.5 Turbo's safety guardrails by fine-tuning on only 10 harmful examples, at a cost of less than $0.20 through the fine-tuning API. More relevant here, fine-tuning on benign, commonly used datasets also degraded safety alignment, to a lesser degree. Safety training is a learned behaviour, and ordinary task training can wash part of it out.

Add safety to the release gate: - Run a safety evaluation on every fine-tuned checkpoint: requests the base model refused, borderline requests it handled carefully, and your own policy cases. Compare refusal and harm rates with the base model, not just task scores. - Mix safety-relevant examples (good refusals and good careful answers) into the training data. - Keep runtime safeguards that do not depend on the fine-tuned weights, such as input and output classifiers and tool permissions. - If you let customers fine-tune, treat their training data as untrusted input and screen it.

Block release on a safety regression just as you would on a task regression.

Trade-offs

Safety evals and mixed-in safety data add cost to every fine-tune, but a released regression is far more expensive.

Common mistake

Assuming that harmless training data cannot make a model less safe.

Follow-up

How would you build a safety eval set that keeps working as your policies change?

Question 42MediumData quality

Your fine-tuned model scores far above the base model on your internal benchmark, but the gain does not show up in production. How do you check for contamination, and why does deduplication matter even without it?

#

The training set was assembled from tickets, wiki pages and old evaluation spreadsheets; the benchmark was written from the same wiki.

Show answer

Suspect leakage first. If benchmark questions or their source passages are in the training data, the score measures memory, not skill. Check by searching the training set for each test item and its reference answer: exact matches, long shared word sequences (n-gram overlap) and near-duplicates by embedding similarity. Old evaluation spreadsheets and the wiki that the benchmark was written from are both likely sources. Remove the overlaps, retrain or at least re-score on a clean subset, and build future benchmarks from material that the training pipeline is blocked from reading.

Deduplication matters beyond contamination. Lee et al. ("Deduplicating Training Data Makes Language Models Better") found heavy duplication in common datasets, such as a single 61-word sentence repeated over 60,000 times in C4. Models trained on deduplicated data emitted memorised text ten times less often, and train-validation overlap affected over 4% of the validation set of standard datasets. In your data, duplicated tickets and templated wiki pages make the model over-learn a few patterns and inflate scores on look-alike test items.

Prevention is a pipeline step: deduplicate within training data, deduplicate training against every eval set, and log what was removed.

Trade-offs

Aggressive near-duplicate removal can drop legitimate variations; light deduplication leaves memorisation and inflated scores.

Common mistake

Splitting train and test randomly from data that contains many near-duplicates, so the test set is mostly copies.

Follow-up

How would you detect contamination for a benchmark that was paraphrased rather than copied?

Question 43HardData quality

To save labelling cost, the team plans to fine-tune each new model version on answers generated by the previous version, round after round. What risks do you raise, and how would you make synthetic data safe to use?

#

Human-written examples are expensive; generated answers are cheap and look fluent.

Show answer

The main risk is model collapse. Shumailov et al. ("The Curse of Recursion") found that using model-generated content in training causes irreversible defects, with the tails of the original distribution disappearing. Each round over-represents what the model already produces often and loses rare cases, unusual phrasing and edge conditions. In a product, the tails are often where customers struggle most. The same paper argues that data from genuine human interactions becomes more valuable as generated text spreads.

Other risks: errors in generated answers are learned as truth; style narrows; and evaluation becomes circular if the same model family writes and grades the data.

Making synthetic data safe: - Keep a fixed core of human-written or human-verified examples in every round, and never replace it with generated data. - Verify generated examples with something independent: tests, rules, retrieval against sources, or human review of a sample. - Generate for coverage on purpose: target rare intents and edge cases, and track the distribution of topics and lengths each round. - Evaluate on a held-out human set that no generated data touches. - Check the terms of any model used to generate training data.

Trade-offs

Synthetic data is cheap and fills gaps quickly, but without verification and a human anchor it narrows the model over rounds.

Common mistake

Measuring each round only on generated test data, which hides the loss of rare cases.

Follow-up

What metric would show you that the tails are shrinking before users notice?

Question 44MediumEvaluation

Validation loss improved steadily through the fine-tune, so the team shipped the final checkpoint. Users say answers got worse. What should the evaluation of a fine-tune include besides loss?

#

Validation data came from a random split of the training set; no generation-based eval was run.

Show answer

Validation loss tells you the model got better at predicting reference tokens from a random split of the same data. That is useful for spotting overfitting, but it misses what users see: whole answers generated with sampling, in the production template, on new requests.

A fine-tune evaluation should include: - Generation-based task metrics on a held-out set split by time or by customer, generated with the production chat template, system prompt and decoding settings. Score correctness, format and refusal behaviour, with automated checks where possible and a judged or human sample. - A comparison against baselines: the base model with a good prompt, and the current production model. A fine-tune that beats its own earlier checkpoints can still lose to a prompted model. - Regression suites for general ability, safety and other languages or tasks the product relies on. - Slices: per intent, length and customer segment, because an average gain can hide a broken slice. - Checkpoint selection on these metrics, not on the last step. Lower loss late in training can come with blander or more overconfident answers.

Then ship gradually (shadow or a small percentage of traffic) and compare user signals before a full rollout.

Trade-offs

Generation-based evals cost more time and judging effort than loss, but loss alone cannot tell you whether the product improved.

Common mistake

Using the final checkpoint and a random split of the training data as the only evidence of quality.

Follow-up

How would you choose between two checkpoints that win on different slices?

Question 45MediumCost

Finance wants a budget for full fine-tuning an open 70B model on 5 billion tokens. How do you produce an estimate you can defend?

#

You can rent H100 nodes by the hour. Nobody on the team has trained a model this size before.

Show answer

Do not estimate from the GPU's peak FLOPS. Real runs reach a fraction of peak; even the highly tuned Megatron-LM setup reported 52% of theoretical peak per GPU at very large scale, and a first attempt is usually lower. Measure instead.

Run a pilot on the target layout (for example two or four nodes) with your real sequence lengths, parallelism and checkpointing for an hour or two. Record tokens processed per second for the whole job after warm-up. Then: GPU-hours for one pass = (5 billion tokens / measured tokens per second) / 3,600 x number of GPUs. Multiply by the number of epochs, and by the hourly price.

Then add the costs people forget: - Hyperparameter search and failed or restarted runs; budget several full or partial runs, not one. - Lost time from interruptions and restarts (checkpoint save and reload time, lost steps since the last checkpoint). - Evaluation runs on each candidate checkpoint. - Storage for checkpoints, which for a 70B model with optimizer state are very large. - Idle time while nodes are reserved but not training.

Present a range with these assumptions written down, and compare it with the LoRA or QLoRA option measured the same way, which usually costs far less and may meet the goal.

Trade-offs

A pilot costs a few node-hours up front but replaces a guess with a measurement; skipping it risks a budget off by a large factor.

Common mistake

Budgeting one clean run at peak FLOPS and ignoring sweeps, failures and evaluation.

Follow-up

Which pilot measurements would also tell you whether the parallel layout is well chosen?