supraj.dev

Interview questions & answers

Transformers, attention & architecture

Q/K/V attention, encoders and decoders, positional signals, MoE, embeddings and model design maths.

  • 27 questions
  • 1 easy
  • 16 medium
  • 10 hard
  • 2 related articles

27 questions

Question 1MediumSelf-attention

How does self-attention decide which tokens contribute to a token representation?

#
Show answer

Each token’s hidden state is projected into a query, key and value vector. A query is compared with the keys of tokens it is allowed to attend to; normalized similarity scores become weights on those tokens’ value vectors. The resulting weighted combination is projected and passed to later layers.

In a causal decoder, a mask prevents positions from attending to future tokens. In an encoder, bidirectional attention is typical. Attention does not directly search for human-understandable words: it learns numerical relationships useful for the training objective. Explain the Q/K/V flow and the mask before drawing conclusions about what a particular head means.

Trade-offs

Full attention offers flexible context mixing but standard dense attention has quadratic score work in sequence length.

Common mistake

Assuming each head always corresponds to one fixed linguistic rule.

Follow-up

What changes between a bidirectional encoder attention mask and a causal decoder mask?

Question 2EasySequence architectures

What is a sequence-to-sequence task, and how can encoder-decoder models solve it?

#
Show answer

Sequence-to-sequence means transforming an input sequence into an output sequence, often with a different length. Translation, summarization and speech transcription are examples. In an encoder-decoder model, the encoder represents the source input, and the decoder autoregressively generates output tokens while attending to the encoded source.

Seq2Seq describes a problem and a family of designs, not a synonym for an RNN. Older systems used recurrent encoder-decoders; transformer architectures such as T5 also implement Seq2Seq. Decoder-only models can solve the same tasks with a prompt, though the computation and control flow differ.

Trade-offs

Encoder-decoder systems separate input understanding from output generation; decoder-only systems can reuse a unified prompt interface.

Common mistake

Treating all Seq2Seq systems as recurrent or assuming input and output lengths must match.

Follow-up

Which design would you test for translating a very long document while preserving cross-paragraph terminology?

Question 3MediumRepresentations

What is a token embedding, and how is it learned?

#
Show answer

An embedding layer is a table mapping a token ID to a dense vector. Parameters can be randomly initialized and updated by backpropagation, or initialized from a pretrained checkpoint and adapted to a new task. These vectors are combined with positional information and processed by layers whose representations become increasingly contextual.

Distinguish a static input embedding from a contextual hidden state and from a separate sentence/document embedding used in retrieval. The vector for the same token ID begins the same within one checkpoint, while its later hidden representation changes with surrounding context.

Trade-offs

Large embedding tables increase capacity and memory; vocabulary design and weight tying affect cost.

Common mistake

Assuming embedding vectors automatically encode an immutable dictionary definition.

Follow-up

Why might the same word in two sentences have identical lookup vectors but different final hidden states?

Question 4MediumSequence architectures

Why did transformers displace many recurrent encoder-decoder designs?

#
Show answer

Recurrent networks propagate a hidden state step by step, creating sequential dependencies across input positions during training. Self-attention lets a transformer process the positions of an input sequence in parallel during training, with direct connections across distant positions and learned or constructed position information.

The trade-off is not a universal victory: standard attention has quadratic score computation with long context, and autoregressive transformer generation is still token-by-token. Modern kernels, attention variants and caching address different bottlenecks. Use the actual sequence length, latency target and task to justify an architecture.

Trade-offs

Attention helps model longer relationships, but memory and computation for long sequences still need explicit planning.

Common mistake

Claiming that every transformer operation, including next-token generation, happens in parallel.

Follow-up

Which part of transformer training is parallel, and why does a decoder still generate sequentially?

Question 5MediumPosition encoding

Why do transformers need positional information, and what approaches exist?

#
Show answer

Self-attention over an unordered set of embeddings has no inherent concept of token order. Models therefore inject position information through absolute positional vectors, sinusoidal encodings, relative position biases or methods such as rotary position embeddings (RoPE), which transform query/key representations based on position.

The choice affects length generalization, handling of relative distances and efficient inference. Position handling must match the checkpoint’s training configuration; swapping encodings in an existing model is not a harmless runtime knob. Evaluate long-context behavior rather than assuming one scheme guarantees perfect extrapolation.

Trade-offs

Absolute learned positions can be simple, while relative methods may better represent distance patterns, with model-specific caveats.

Common mistake

Thinking token order is automatically preserved by the attention matrix without any positional signal.

Follow-up

What might break if you extend the maximum sequence length without validating RoPE-related behavior?

Question 6MediumSelf-attention

What does multi-head attention add beyond one attention head?

#
Show answer

Instead of producing one query/key/value representation, a multi-head layer uses multiple learned projections into subspaces. Each head independently computes attention-weighted values; the results are concatenated and projected into the model dimension. This gives the network multiple routes for combining information at a layer.

Heads may learn different patterns, but labels such as syntax-head or meaning-head are interpretations, not guaranteed roles. More heads do not automatically mean better accuracy; projection dimensions, attention implementation and cache layout also matter. Grouped-query and multi-query variants deliberately share some key/value heads to reduce inference memory.

Trade-offs

Multiple heads can increase expressiveness; fewer KV heads can reduce cache memory with quality trade-offs.

Common mistake

Assuming each attention head is manually assigned a linguistic function.

Follow-up

Why might an inference model use fewer KV heads than query heads?

Question 7MediumAttention mathematics

What exactly does softmax do to attention scores?

#
Show answer

For a given query, attention first obtains compatibility scores for eligible keys. Softmax exponentiates and normalizes these scores so the weights are nonnegative and sum to one across the attended positions. The weights determine how strongly each value vector contributes to the new representation.

Implementations use numerically stable softmax by subtracting the maximum score, and masked positions are excluded before normalization. A softmax weight is not automatically an interpretable probability that a statement is true; it is a normalized coefficient inside a learned computation.

Trade-offs

Dense normalized attention is expressive; masking, precision and long-sequence kernels introduce implementation challenges.

Common mistake

Treating attention weights as calibrated confidence in facts.

Follow-up

Why do implementations subtract the maximum score before exponentiation?

Question 8HardAttention mathematics

Derive scaled dot-product attention from Q, K and V. Why divide by sqrt(dk)?

#
Show answer

For query matrix Q, key matrix K and value matrix V, compute compatibility scores as Q times transposed K. Divide by the square root of the key dimension, apply the attention mask, normalize across keys with softmax and multiply by V. In notation: Attention(Q, K, V) = softmax((QKᵀ / √dₖ) + mask) V.

Scaling helps prevent large dot products from pushing softmax into saturated regimes with tiny gradients when vector dimension grows. With sequence length n, the dense score matrix has n-by-n entries, explaining the memory and computational pressure of long contexts. Fused attention kernels can avoid materializing the full score matrix without changing the mathematical result.

Trade-offs

Exact dense attention preserves all pairwise interactions, while efficient implementations target memory movement and computation cost.

Common mistake

Forgetting to apply the mask before softmax, or confusing dk (per-head key width) with the number of tokens.

Follow-up

How would causal masking and a fused attention kernel change execution without changing this equation?

Question 9HardModel architectures

How does a mixture-of-experts transformer reduce active compute without shrinking all model weights?

#
Show answer

A sparse MoE replaces some dense feed-forward layers with a router and multiple expert sub-networks. For each token, the router chooses a small subset of experts and combines their outputs. This can increase total parameter capacity while keeping active per-token computation lower than activating all experts.

Sparse execution does not mean inactive experts disappear from memory: the full expert pool must still be stored or distributed across devices. Routing balance, token dispatch, capacity limits, expert placement and inter-GPU communication become production bottlenecks. Evaluate throughput and quality using the actual load and topology, not just total or active parameter counts.

Trade-offs

MoE increases parameter capacity for a compute budget but adds routing, memory residency and communication complexity.

Common mistake

Assuming a 10% active-parameter ratio automatically means 90% less total VRAM.

Follow-up

Why can a model with few active experts still require substantial aggregate GPU memory?

Question 10HardOutput layers

What is adaptive softmax, and is it the default output layer for modern LLM serving?

#
Show answer

Adaptive softmax is a hierarchical approximation that partitions a large output vocabulary into frequent tokens and less-frequent clusters. Because rare-token clusters do not always need full computation, it can reduce training cost for some large-vocabulary language models.

It is a technique, not a universal feature of current production LLMs. Many modern decoder models use a conventional vocabulary projection with optimized matrix operations and decoding kernels. The benefit depends on vocabulary distribution, hardware and task; hierarchical approximation can introduce extra control flow and complicate full-vocabulary scoring.

Trade-offs

Hierarchical outputs can reduce some work, but require special handling and may not outperform optimized dense kernels.

Common mistake

Assuming adaptive softmax is always enabled or that it is equivalent to top-k generation sampling.

Follow-up

How would you benchmark adaptive softmax against a fused dense output projection on a GPU?

Question 11HardOptimization

Why are residual connections and normalization important for training deep transformers?

#
Show answer

Deep networks propagate gradients through many composed layers. Residual connections provide a direct information and gradient pathway around each transformation, and normalization stabilizes activation scales. Initialization, residual scaling and optimizer settings also influence whether training remains stable as depth increases.

Self-attention by itself does not guarantee freedom from vanishing or exploding gradients. Compare a model with and without the residual/normalization path, monitoring gradient norms and loss curves. Modern architectures differ in pre-norm versus post-norm placement and their preferred training settings.

Trade-offs

Normalization and residual design affect convergence and training stability but may require tuning for depth and precision.

Common mistake

Saying that self-attention alone solves the vanishing-gradient problem.

Follow-up

What failure signals would suggest gradient instability rather than a poor dataset?

Question 12MediumSequence architectures

What distinguishes encoder-only, decoder-only and encoder-decoder transformers?

#
Show answer

An encoder-only model typically allows tokens to attend bidirectionally and yields contextual representations useful for tagging, retrieval or classification. A decoder-only model applies a causal mask and is trained for next-token prediction, making it a natural generator. An encoder-decoder model encodes source tokens, then uses causal decoding plus cross-attention to source representations.

These are common configurations rather than rules that restrict downstream tasks: decoder models can classify via constrained generation, and encoders can be part of larger generative systems. Explain the attention masks and training objectives before selecting a model for summarization or search.

Trade-offs

Encoder-decoder can separate source and target computation; decoder-only offers a unified generation interface.

Common mistake

Claiming every decoder must always use an encoder output or that encoders cannot support question answering.

Follow-up

Which attention block allows a decoder token to access the complete encoded source sequence?

Question 13MediumAttention

Why does long context cost so much, and which techniques reduce it?

#

Doubling prompt length more than doubles prefill time and fills GPU memory.

Show answer

In standard attention, every token attends to every earlier token, so prefill compute grows roughly with the square of sequence length. During generation, the KV cache stores keys and values for every token and layer, so memory grows linearly with context and with the number of concurrent requests.

FlashAttention computes exact attention with fewer memory reads and writes, which speeds it up. Grouped-query attention shares key and value heads across query heads, shrinking the KV cache. Serving engines add paged KV memory and prefix caching to reuse shared prompts.

Trade-offs

Sharing KV heads saves memory but changes the model; it is a training-time choice, not a serving switch.

Common mistake

Assuming a longer context window is free once the model supports it.

Follow-up

How would you estimate KV cache memory for a given model and context length?

Question 14MediumMultimodal

How does a vision-language model turn an image into something the language model can read?

#

A team sends high-resolution screenshots to a multimodal model and is surprised by token costs and missed small text.

Show answer

A common design splits the image into fixed-size patches and encodes them with a vision transformer, so each patch becomes an embedding. A small projection layer maps those embeddings into the language model's embedding space, and they enter the sequence alongside text tokens; the language model then attends over both.

That explains the practical behaviour: more image tiles or higher resolution means more tokens, so cost and latency grow, and images are often resized, which can blur small text. Crop to the region that matters, check the provider's image-size rules, and test reading of small details directly.

Trade-offs

Higher resolution improves detail but multiplies image tokens and cost.

Common mistake

Sending full-page screenshots when only one table matters.

Follow-up

How would you estimate the token cost of an image before sending it?

Question 15HardPositional encoding

Why do positional encodings matter when extending a model's context length?

#

A model trained on 8K tokens degrades badly when given 32K tokens.

Show answer

Rotary position embeddings (RoPE) encode position by rotating query and key vectors by angles that depend on position. Beyond the trained length, the model meets rotation angles it never learned, and attention patterns break down.

Context-extension methods rescale positions so longer inputs map into ranges the model knows, for example position interpolation, or YaRN, which scales different frequency components differently. They usually need a short fine-tune on long sequences. Always test retrieval across the whole long window, not just perplexity.

Trade-offs

Extension is cheaper than retraining but can slightly reduce quality on short inputs.

Common mistake

Raising the maximum length in config and assuming the model can use it.

Follow-up

How would you test whether a model really uses information in the middle of a long context?

Question 16HardAttention variants

You are choosing between open models for a long-context product. One uses multi-head attention, one grouped-query attention, one multi-head latent attention. How does that choice show up in serving cost, and what does each give up?

#

Similar benchmark quality on your evals. Serving cost is dominated by KV cache memory at long context.

Show answer

In multi-head attention (MHA) every query head has its own key and value head, so the KV cache per token scales with the full head count. Multi-query attention (MQA) shares one key/value head across all query heads. The GQA paper (Ainslie et al.) notes that MQA drastically speeds up decoder inference but can degrade quality. Grouped-query attention (GQA) sits between them, with an intermediate number of key/value heads, and the paper reports that GQA achieves quality close to MHA with speed comparable to MQA. It also showed existing MHA checkpoints can be uptrained to this form using 5% of the original pre-training compute.

The effect is direct. Llama 3 70B has 64 query heads but 8 key/value heads, so its KV cache is one eighth of what full MHA would need at the same size. That ratio translates into eight times more concurrent long sequences per GPU, or the same sequences on fewer GPUs.

Multi-head latent attention (MLA), introduced with DeepSeek-V2, goes further by compressing keys and values into a latent vector. DeepSeek reported a KV cache 93.3% smaller than their earlier 67B model and 5.76 times its peak generation throughput. The trade-off is engine support and kernels: MLA needs dedicated attention implementations, so check that your serving engine and hardware run it efficiently before counting the saving.

So for a long-context product, compute KV bytes per token for each candidate from its config, multiply by your expected tokens in flight, and compare cost per conversation, not just benchmark scores.

Trade-offs

Fewer key/value heads cut memory and bandwidth per token but can cost some quality; latent compression saves more but depends on specialised kernels.

Common mistake

Comparing models by parameter count and ignoring the attention layout that sets KV cache size.

Follow-up

How would you calculate KV bytes per token from a model config file, and which fields can mislead you?

Question 17MediumAttention variants

A model uses sliding-window attention in some layers. What does that buy you, what does it cost, and which serving optimisations need extra care with it?

#

You are adding prefix caching and an FP8 KV cache to a deployment of such a model.

Show answer

In a sliding-window attention layer, each token attends only to a fixed number of recent tokens instead of the whole history, so the attention cost and the KV state for that layer stop growing with sequence length. Mistral 7B, for example, combined grouped-query attention for faster inference with sliding-window attention to handle long sequences at reduced inference cost. Information from further back still flows, but indirectly, through stacked layers, and models often mix windowed layers with full-attention layers to keep some direct long-range access.

The cost is that a windowed layer cannot look up a distant detail in one step, so tasks that need exact recall from far back depend on the full-attention layers or on information having been carried forward. Evaluate long-context retrieval tasks specifically, not only perplexity.

Serving details change too. Windowed layers keep only recent KV entries, so the engine manages their cache differently from full-attention layers. vLLM's quantized KV cache documentation notes that some layer types, such as sliding-window layers, are more sensitive to KV cache quantization, and offers --kv-cache-dtype-skip-layers to keep them in the native type. Test prefix caching and quantization on this model rather than assuming results from a full-attention model carry over.

Trade-offs

Windowed layers cap memory and compute per token but weaken direct long-range recall; mixing in full-attention layers restores some of it at their cost.

Common mistake

Assuming a model with a very long advertised context can attend directly to every token in every layer.

Follow-up

How would you design an eval that separates direct long-range recall from information carried forward through layers?

Question 18HardLong context

For an always-on assistant, an engineer keeps only the most recent N tokens of KV cache and drops the rest. Quality collapses as soon as the first tokens fall out of the window, even though they are just a system preamble. Why, and what is the fix?

#

A model trained with full attention, served with a rolling window of KV cache.

Show answer

This is the attention sink effect described in the StreamingLLM paper (Xiao et al.). Models trained with full attention learn to put strong attention on the initial tokens, as a kind of sink, even when those tokens are not semantically important. The usual intuition is that softmax weights must sum to one, so attention that has nowhere useful to go lands on the first tokens, which every later token can see. The paper shows that window attention, which caches only the most recent keys and values, fails when the text length exceeds the cache size: once those initial tokens are evicted, the attention distribution breaks.

The fix is simple: keep the KV states of the first few tokens permanently, plus the recent window. The paper reports this largely recovers the performance of window attention, works without fine-tuning, has been tested on streams of up to 4 million tokens and more, and is up to 22.2 times faster than a sliding-window baseline that recomputes the KV cache.

Be clear about what it does not do. Streaming with sinks keeps the model stable on endless input, but it does not let the model remember facts from the evicted middle. For memory across a long session you still need summarisation, retrieval or an explicit memory store.

Trade-offs

Sink-plus-window caching keeps memory constant and quality stable for streams, but anything outside the window is forgotten unless stored elsewhere.

Common mistake

Treating stable output on long streams as evidence that the model remembers the whole stream.

Follow-up

How would you combine a sink-plus-window cache with a summarised memory for a multi-day agent session?

Question 19MediumModel architectures

A vendor pitches a state-space (Mamba-style) or hybrid model for a long-document workload because it "scales linearly". What do you take from that claim, and what do you test before adopting it?

#

Current stack serves a transformer with vLLM. Workload: long contracts, extraction and question answering.

Show answer

The claim has a real basis. The Mamba paper (Gu and Dao) makes state-space model parameters functions of the input, so the model can choose what to keep or forget along the sequence. It reports linear scaling in sequence length, about 5 times higher inference throughput than Transformers, and that Mamba-3B outperforms Transformers of the same size and matches Transformers twice its size. Instead of a KV cache that grows with every token, such a model carries a fixed-size state, which is very attractive for long inputs.

The same paper names the weakness its design targets: earlier subquadratic models could not do content-based reasoning well. A fixed-size state must compress the past, so exact recall of a specific clause from far back is the thing to test hardest. This is one reason many recent models are hybrids that interleave state-space layers with some attention layers.

Before adopting, test on your tasks: exact extraction of details at various depths of long contracts, multi-hop questions across sections, and a long-context benchmark with tasks beyond needle-in-a-haystack. Then check the operational side: does your serving engine support the architecture with prefix caching, quantization and the batching you rely on, and how do throughput and memory compare at your real lengths? Architectures with less mature tooling can lose in production what they win on paper.

Trade-offs

A fixed-size state removes KV cache growth and speeds long inputs, but compresses the past and has less mature serving support than transformers.

Common mistake

Adopting an architecture on a throughput chart without testing exact long-range recall on your own documents.

Follow-up

Which serving optimisations built for transformers (prefix caching, KV offloading) do not map directly onto a state-space model?

Question 20MediumNormalization

You are training a small transformer from scratch. With the layer norm placed after each residual addition (Post-LN), training diverges unless you use a long learning-rate warm-up. A colleague moves the norm inside the residual branch (Pre-LN) and the warm-up becomes unnecessary. Why does placement matter so much?

#

Same model size, optimizer and peak learning rate in both runs.

Show answer

In Post-LN, the normalisation sits on the main path after each residual addition, so the residual stream itself is renormalised at every layer. Xiong et al. ("On Layer Normalization in the Transformer Architecture") show that at initialisation "the expected gradients of the parameters near the output layer are large". A high learning rate on those large gradients destabilises training, which is why Post-LN needs a warm-up: start small, let the model leave the bad region, then raise the rate.

In Pre-LN, each sub-layer normalises its own input, and the residual stream passes through unchanged. The paper's analysis finds that the gradients are well-behaved at initialisation, and in their experiments Pre-LN without warm-up reaches comparable results with significantly less training time.

Practical points: - Pre-normalisation is widely used in large decoder-only models, which are deep stacks where easy training matters. - Pre-LN needs a final normalisation before the output head, since the residual stream is never normalised on the main path. - Placement is still an active design choice. Gemma 2, for example, normalises both the input and the output of each attention and feed-forward sub-layer with RMSNorm "to stabilize training".

Trade-offs

Pre-LN trains more easily without warm-up; Post-LN keeps the original design but needs careful warm-up and learning-rate tuning to train at all.

Common mistake

Treating a long warm-up as a tuning detail when the real cause of divergence is where the normalisation sits.

Follow-up

What would you monitor during the first thousand steps to catch an unstable run early?

Question 21MediumNormalization

Most recent open LLMs use RMSNorm instead of LayerNorm. What exactly does RMSNorm remove, why does that not hurt, and what should you watch for when porting code between the two?

#

You are converting a model implementation and reusing kernels from another codebase.

Show answer

LayerNorm re-centres each vector (subtracts the mean) and re-scales it (divides by the standard deviation), then applies a learned gain and bias. RMSNorm drops the re-centring: it divides by the root mean square of the vector and applies a learned gain, usually without a bias.

Zhang and Sennrich's RMSNorm paper hypothesises that "re-centering invariance in LayerNorm is dispensable", keeps re-scaling invariance, and reports that RMSNorm reduces running time by 7% to 64% on different models while achieving comparable performance. Fewer reductions per vector make it cheaper, which matters because normalisation runs twice per layer for every token.

When porting: - The formulas are not interchangeable. Loading RMSNorm weights into a LayerNorm module, or the reverse, silently changes the outputs. - Check the epsilon value and where it is added; it differs between implementations and can change outputs in low precision. - Check how the gain is parameterised in each implementation before copying weights across. - Compute the norm in float32 even when activations are bf16; the sum of squares can lose precision.

Test a port by comparing hidden states layer by layer against the reference implementation, not just final outputs.

Trade-offs

RMSNorm is cheaper and works as well in practice; LayerNorm's centring can matter for some architectures and is the default in older models you may need to match exactly.

Common mistake

Swapping the norm type while converting checkpoints and assuming the learned weights still mean the same thing.

Follow-up

Where are the norms placed in the model you work with, and how would you check that a converted checkpoint matches?

Question 22MediumFeed-forward layers

Llama 3 8B has a model dimension of 4,096 but a feed-forward dimension of 14,336, which is 3.5 times, not the 4 times from the original transformer. Why do models with SwiGLU use these odd sizes?

#

You are sizing a new model and want to match the parameter count of a standard design.

Show answer

The original feed-forward block has two matrices: up-project to d_ff, apply a nonlinearity, down-project. Gated variants such as SwiGLU have three: two up-projections, one passed through the activation and multiplied element-wise with the other, then a down-projection.

Shazeer's "GLU Variants Improve Transformer" paper tests these variants and finds that some improve quality over ReLU or GELU. Because they have three matrices instead of two, the paper reduces the hidden size by a factor of 2/3 "to keep the number of parameters and the amount of computation constant"; in its experiments d_ff goes from 3,072 to 2,048 for a model dimension of 768.

Apply that to a model dimension of 4,096: the standard 4 times gives 16,384, and two-thirds of that is about 10,923. Llama 3's table lists 14,336 for the 8B model and 28,672 for 70B (model dimension 8,192), both 3.5 times, with SwiGLU as the activation. So the 2/3 rule is a starting point, not a law: Llama 3 uses a larger value than the rule gives, and designers also round to sizes that split evenly across tensor parallel ranks and suit GPU kernels.

When sizing your own model, count parameters per block as 3 x d_model x d_ff for the feed-forward part, and keep d_ff divisible by your tensor parallel degree and by the tile sizes your kernels prefer.

Trade-offs

A larger feed-forward block adds capacity and compute per token; a smaller one leaves more budget for depth, attention or vocabulary.

Common mistake

Copying 4 x d_model for a SwiGLU layer and ending up with a model about 50% larger in its feed-forward blocks than planned.

Follow-up

How does the feed-forward block's share of parameters change in a mixture-of-experts model?

Question 23HardMixture of experts

In a mixture-of-experts training run, a few experts receive most of the tokens, others are almost never chosen, and throughput drops because of overflowing experts. How do MoE models keep routing balanced, and what are the trade-offs of each method?

#

Top-k routing with expert parallelism across GPUs. Training in bf16.

Show answer

Unbalanced routing hurts twice. Hot experts become bottlenecks under expert parallelism, and the DeepSeek-V3 report notes that an unbalanced load can lead to "routing collapse", where the router keeps choosing the same experts and the rest stop learning.

Capacity factor. The Switch Transformer gives each expert a fixed buffer: tokens per batch divided by the number of experts, times a capacity factor. A factor above 1.0 adds slack for imbalance. Tokens beyond capacity are dropped: the expert computation is skipped and the token passes to the next layer through the residual connection. A higher factor means fewer drops but "wasted computation and memory". The paper reports that keeping drop rates low matters for scaling, typically under 1% in their runs.

Auxiliary load-balancing loss. Switch adds a loss per MoE layer based on the fraction of tokens sent to each expert and the router probability assigned to it, which pushes towards uniform use. The weight needs tuning: DeepSeek-V3 notes that "too large an auxiliary loss will impair the model performance".

Auxiliary-loss-free bias. DeepSeek-V3 adds a per-expert bias to the routing scores, used only to choose the top-k experts, not to weight their outputs. After each step, the bias goes down for overloaded experts and up for underloaded ones. They report balanced load through training, better performance than loss-based balancing, and no dropped tokens in training or inference.

Also check numerics: Switch found router instability in low precision and casts the router input to float32.

Trade-offs

Capacity limits bound memory but drop tokens; auxiliary losses balance load at some cost to quality; bias-based balancing avoids the loss but adds a controller with its own update speed to tune.

Common mistake

Monitoring only the average loss and missing that a few experts are doing all the work.

Follow-up

Which metrics per expert would you put on the training dashboard?

Question 24HardStability

You are reimplementing Gemma 2 for a custom inference engine and use a standard fused attention kernel. Outputs drift from the reference implementation, especially on long prompts. What architectural detail is easy to miss?

#

Weights load correctly, short prompts look fine, and the reference uses eager attention.

Show answer

Logit soft-capping. The Gemma 2 report says it caps logits in each attention layer and in the final layer with logits = soft_cap * tanh(logits / soft_cap), so they stay between minus and plus soft_cap. It sets soft_cap to 50.0 for self-attention and 30.0 for the final layer.

A standard fused attention kernel computes softmax(QK^T / sqrt(d)) directly. If it has no hook to apply the tanh cap to the scores first, attention is no longer the same function. Because tanh is nearly linear for small values, the cap changes little until scores get large, so tests on easy inputs can pass while other inputs drift.

Check for the rest of Gemma 2's details too: the report describes alternating local sliding-window attention (4,096 tokens) and global attention layers, GQA, and RMSNorm on both the input and output of each sub-layer. Any of these, implemented the usual way, gives a model that runs but drifts.

What to do: - Use a kernel that supports a soft-cap parameter, or apply the cap in an unfused path for these layers. - Apply the final-layer cap to output logits before sampling. - Compare against the reference layer by layer on long inputs, with tolerances suited to your precision.

More generally, read the model's paper and configuration for non-standard pieces before porting it to a generic engine.

Trade-offs

Soft-capping helps stability in training but needs kernel support at inference; dropping it for speed changes the model.

Common mistake

Validating a port only on short prompts, where capped and uncapped attention look the same.

Follow-up

How would you build a layer-by-layer comparison harness for a model port?

Question 25MediumEmbeddings

Your 2B-parameter on-device model spends a large share of its parameters on the token embedding and output layers. Someone suggests tying them. What does tying mean, why does it work, and when would you not do it?

#

Large multilingual vocabulary, tight memory budget on phones.

Show answer

The input embedding maps token IDs to vectors (vocabulary x d_model). The output layer maps the final hidden state back to vocabulary logits, a matrix of the same shape transposed. Tying uses one matrix for both.

Press and Wolf's "Using the Output Embedding to Improve Language Models" shows that the output matrix is itself a valid word embedding, recommends tying, and reports a significant reduction in perplexity on several language models, and that tying can shrink translation models to under half their size without hurting performance.

Why it matters for small models: the vocabulary matrix does not shrink with the rest of the model. Gemma 2's report lists about 590 million embedding parameters for its 2B model, against about 2.0 billion non-embedding parameters, because it uses a 256k-entry vocabulary built for many languages. Storing a second matrix of that size for the output would be a large extra cost on a phone.

When not to tie: - Larger models where the matrix is a small share of total parameters, and an untied output layer gives the model a little more freedom. - When the input and output need different treatment, for example quantising the output layer differently or adding special tokens to one side.

In implementation, make sure the tie survives: checkpoint conversion, LoRA wrappers and some sharding setups can silently untie weights and double memory.

Trade-offs

Tying saves a whole vocabulary matrix and can regularise small models; untying adds parameters and flexibility.

Common mistake

Assuming a 2B model's parameters are mostly in its transformer layers, when the vocabulary matrix is a large share.

Follow-up

How would vocabulary size affect your choice of tokenizer for an on-device model?

Question 26HardTraining objectives

A paper claims that training a model to predict several future tokens at once improves code generation and makes inference up to three times faster. How does multi-token prediction work, and what would you check before adopting it?

#

You train code models in the 1B to 10B range and serve them with speculative decoding.

Show answer

In Gloeckle et al.'s "Better & Faster Large Language Models via Multi-token Prediction", the model predicts the next n tokens at each position, all at once, using a shared trunk and n independent output heads. Training adds those extra prediction losses on top of the shared representation.

What they report: - Benefits grow with scale: in their experiments multi-token prediction models are worse than the baseline at small sizes but better at scale, with 13B models solving around 15% more code problems on average. - The extra heads can act as a draft for self-speculative decoding. Models trained with 4-token prediction are up to 3 times faster at inference, even with large batch sizes. - They argue it reduces the mismatch between teacher-forced training and autoregressive generation.

DeepSeek-V3 also uses a multi-token prediction objective, with sequential modules that keep the complete causal chain at each prediction depth, so the idea is used in production-scale models.

Before adopting it: - Check your scale. The small-model result is a warning: gains are not universal. - Evaluate on your tasks; their strongest results are on code. - Measure serving speed in your engine: the speed-up depends on how many drafted tokens are accepted, which depends on your traffic. - Count the cost: the extra heads add parameters, although the paper reports no overhead in training time for its setup.

Trade-offs

Multi-token prediction can add quality and a built-in draft model at scale, but adds heads to maintain and can hurt small models.

Common mistake

Expecting the paper's inference speed-up without measuring acceptance rates on your own traffic.

Follow-up

How does self-speculative decoding with extra heads compare with using a separate small draft model?

Question 27MediumInference

Batched generation with a decoder-only model works for the longest prompt in each batch, but shorter prompts produce garbage continuations such as repeated tokens. Single-prompt generation is fine. What is wrong?

#

Hugging Face Transformers, a tokenizer with the default right padding, prompts of different lengths in one batch.

Show answer

With right padding, a short prompt is followed by pad tokens, so the model is asked to continue from padding rather than from the end of the prompt. The Transformers generation guide says LLMs "aren't trained to continue generation from padding tokens", which is why it sets padding_side="left" for decoder-only models. Its own example shows the failure: a right-padded "1, 2, 3" continues as "1, 2, 33333333333".

Fix: - Load the tokenizer with padding_side="left" for generation. Pad tokens then come before the prompt, and every sequence ends with its real last token where generation begins. - Pass the attention_mask returned by the tokenizer, so attention ignores the pad positions. - Set a pad token if the model has none (the guide's example uses the end-of-sequence token).

The guide adds that right padding is worse for state-space and hybrid models, because their recurrent state keeps updating over trailing pad tokens, so the state no longer matches the end of the prompt.

Training is different: padding side matters less there because you mask the loss on pad positions. Keep separate tokenizer settings for training and generation, or set padding side explicitly at each call site.

Trade-offs

Left padding is right for generation; padding every prompt to the longest one in the batch still wastes compute on pad positions.

Common mistake

Running batched evaluations with right padding and blaming the model for worse scores than single-prompt runs.

Follow-up

Why can batching also change results slightly even with correct padding?