supraj.dev

Interview questions & answers

LLM fundamentals

Tokens, context, sampling and output contracts. Explain what a model can and cannot guarantee.

  • 29 questions
  • 12 easy
  • 15 medium
  • 2 hard
  • 4 related articles

29 questions

Question 1EasyProduction reasoning

Why does a short-looking conversation exceed the context budget?

#

A chat application rejects a request after many tool calls, even though the last user message is tiny.

Show answer

Inspect tokenized system instructions, conversation history, tool definitions and tool results. Measure with the model-compatible tokenizer or provider usage. Reserve room for output and prune irrelevant context deliberately. Summaries should retain decisions and source links; test what information they lose.

Trade-offs

Summaries reduce context size but can omit constraints; selective retrieval adds a separate relevance decision.

Common mistake

Equating words, characters and tokens, or forgetting tool output.

Follow-up

Which information would you preserve before compressing an incident conversation?

A useful sequence to explain
  1. Instructions + history + tools
  2. Tokenization
  3. Input budget + output reserve
Question 2EasyProduction reasoning

The model sounds certain about an invented API. How do you respond?

#

A generated integration example names a method that does not exist in the installed SDK.

Show answer

Verify the installed SDK and primary documentation. Ask for an answer grounded in the relevant interface, then compile or run a small bounded example. Require explicit uncertainty when evidence is absent. Capture the failure as a test; lowering randomness alone does not establish truth.

Trade-offs

Grounding and verification add work, but make unsupported claims easier to catch.

Common mistake

Treating confidence of wording as calibrated confidence.

Follow-up

What would you test before allowing generated code to run against a real system?

Question 3MediumProduction reasoning

The JSON is valid, but the business decision is wrong. What does structured output buy you?

#

An expense classifier emits the required schema but assigns an ineligible category.

Show answer

Validate the schema, then check domain rules and supporting evidence outside the model. Handle refusal, truncation and missing required information explicitly. Evaluate classification on representative cases. A constrained format can improve parsing reliability; it cannot prove that field values are correct.

Trade-offs

Tighter schemas simplify integration but require versioning and careful handling of unsupported cases.

Common mistake

Using schema conformance as the only quality test.

Follow-up

How would you represent insufficient evidence without inventing a category?

Question 4MediumProduction reasoning

Your support assistant gets the tone right but misses fresh facts. What would you change?

#

The team proposes fine-tuning so the assistant can answer questions about policies updated every week.

Show answer

First establish a baseline with clear instructions and representative cases. Ground changing facts in current authorized sources. Consider training only for a measured persistent behavior gap and validate on held-out cases. Explain how policy updates would reach the answer and how incorrect evidence would be detected.

Trade-offs

Retrieval supports changing facts but adds operations; training requires curated data and another release process.

Common mistake

Assuming model training is a reliable weekly document synchronization mechanism.

Follow-up

What evidence would justify training rather than better context or instructions?

Question 5HardProduction reasoning

Temperature is zero. Why should you still test repeated runs?

#

A workflow makes occasional different tool choices despite unchanged user input and decoding settings.

Show answer

Record model version, exact inputs, tool schemas and runtime configuration. Check retrieved context and tool results for drift, then run repeated cases across the deployment environment. Define acceptable variation at the task-outcome level. Use application validation and bounded tool execution even when outputs appear repeatable.

Trade-offs

Strict reproducibility constraints can limit deployment flexibility; outcome-based checks tolerate harmless wording differences.

Common mistake

Promising deterministic correctness from one decoding setting.

Follow-up

How would you compare a new model version without requiring identical wording?

Question 6EasyModel foundations

What makes a language model an LLM, and what can its pretraining actually teach it?

#
Show answer

A large language model is a neural model trained on broad text or code datasets to predict or reconstruct tokens. Transformer-based autoregressive models are common, but neither a fixed parameter threshold nor one architecture defines every LLM. Training learns statistical patterns, associations and useful representations; it does not attach a source-of-truth database to every claim.

In an interview, distinguish the pretrained model from the application around it: instructions, retrieval, tools, evaluation and deployment controls determine whether it is useful and reliable for a particular task.

Trade-offs

Larger models may improve capability but often need more memory, latency and budget; compare on your actual task.

Common mistake

Equating fluent language with verified facts or defining every LLM as a model with billions of parameters.

Follow-up

What changes if the application needs today’s company policy rather than general pretrained knowledge?

Question 7EasyTokenization

How is a sentence converted into tokens, and why does the tokenizer matter?

#
Show answer

A tokenizer maps text into token IDs using a vocabulary and segmentation rules. Common schemes use subwords or byte-level units; one visible word may become several tokens and punctuation or whitespace may have their own IDs. An embedding lookup then maps each ID to a vector that the network can process.

Token counts affect context limits, cost and throughput. Different tokenizers divide the same text differently, especially code, non-English scripts and rare names. For production systems, count with the model’s actual tokenizer rather than assuming one token equals one word.

Trade-offs

A larger vocabulary may shorten some inputs but increases embedding and output projection size.

Common mistake

Assuming every tokenizer splits on spaces, or that a token is always a full word.

Follow-up

Why might Telugu text or source code have a different tokens-per-character ratio from English?

Question 8MediumContext windows

What is an LLM context window, and what needs to fit inside it?

#
Show answer

The context window is the maximum sequence budget supported by a particular model/configuration. Depending on the provider, it may cover system and user instructions, conversation history, retrieved chunks, tool outputs and some or all generated output. The usable budget can be lower than the headline maximum because of reserved output and application overhead.

A larger context does not guarantee every relevant detail is attended to or recalled accurately. For a long RAG chat, maintain a budget, retain high-value evidence, retrieve only relevant passages and measure accuracy across different context positions.

Trade-offs

More context may improve available evidence but raises prefill cost and can distract from the relevant passages.

Common mistake

Treating the context window as persistent user memory or assuming the full advertised limit is always usable for input.

Follow-up

How would you keep a 20-turn support conversation within a fixed token budget without losing critical policy evidence?

Question 9MediumDecoding

How does beam search differ from greedy decoding, and when would you avoid it?

#
Show answer

Greedy decoding picks the highest-scoring next token at every step, keeping one candidate sequence. Beam search keeps multiple partial sequences, expands each at the next step and retains the highest-scoring alternatives. It can improve sequence-level search for constrained outputs such as translation, especially with length handling and repetition controls.

Beam search is not automatically more creative or more coherent for open-ended chat: high-probability sequences can be repetitive and diverse answers may be better served by sampling. Beam width increases inference work and candidate management.

Trade-offs

A broader search may help constrained generation but increases compute and can harm open-ended diversity.

Common mistake

Calling beam search a universally superior replacement for greedy decoding.

Follow-up

What happens to latency and repetition if the beam width grows from 1 to 8?

Question 10EasyDecoding

What does temperature do to token probabilities during generation?

#
Show answer

Temperature scales the logits before a probability distribution is formed. Below 1, the distribution becomes sharper, so high-probability tokens dominate; above 1, it becomes flatter and lower-ranked choices become more likely. With supported sampling implementations, setting temperature near zero approaches greedy selection, though deterministic behavior across runs is not guaranteed by the entire serving stack.

Temperature controls diversity, not factual correctness. To compare settings, hold the prompt, model, evaluation set and other generation controls fixed, then measure output quality rather than choosing a number because it sounds creative.

Trade-offs

Lower temperature often improves repeatability while higher temperature can increase diversity and error variance.

Common mistake

Assuming temperature is a confidence score or that lowering it makes retrieved facts true.

Follow-up

Why can two requests still differ even when you set temperature to zero?

Question 11MediumDecoding

What is the practical difference between top-k and top-p sampling?

#
Show answer

Top-k restricts candidate tokens to the k highest probabilities before sampling. Top-p (nucleus) sampling uses the smallest high-probability set whose cumulative probability reaches at least p, so the number of eligible tokens can grow or shrink depending on the model’s uncertainty. Probabilities are renormalized over the retained set before sampling.

For a sharp distribution, top-p may keep very few tokens; for a flat one, it may keep many. The interactions between temperature, top-k and top-p depend on the inference framework. Tune the combined settings against representative tasks, not isolated anecdotes.

Trade-offs

Fixed top-k is predictable in candidate count; top-p adapts to the distribution but may vary more across contexts.

Common mistake

Describing top-p as a fixed number of tokens or treating its threshold as a minimum probability per token.

Follow-up

With one token at probability 0.92, how many candidates might top-p=0.9 retain?

Question 12EasyPrompting

How does prompt design change model behavior, and what cannot a prompt enforce?

#
Show answer

A well-specified prompt identifies the task, supplied context, output format, boundaries and what to do when evidence is missing. Examples can illustrate style or edge cases. Compare prompt variants on a held-out evaluation set instead of selecting the most impressive single response.

Prompts guide a model, but they are not authorization controls or reliable validators. A prompt asking for valid JSON is weaker than schema-checked structured output; a prompt asking not to reveal secrets is weaker than keeping secrets out of the model and enforcing server-side access checks.

Trade-offs

More elaborate prompts may improve compliance while increasing context and maintenance costs.

Common mistake

Using prompt wording as the only defense against unauthorized tool calls.

Follow-up

What would you enforce in application code rather than in a prompt for a SQL assistant?

Question 13EasyTokenization

What happens when the model encounters a word that is not in its vocabulary?

#
Show answer

Modern subword or byte-fallback tokenizers often represent unfamiliar names, typos and new terms as multiple known token units rather than mapping the entire word to one unknown symbol. A newly coined company name may be broken into fragments that can still be embedded and processed.

This solves the mechanics of representing text, not understanding it. A model can tokenize an unfamiliar product ID perfectly while having no reliable knowledge of what the ID means. Retrieval or tools may still be required for the actual answer.

Trade-offs

Byte fallback improves coverage, but rare words may take more tokens and use more context.

Common mistake

Assuming successful tokenization means a new or rare term is present in training knowledge.

Follow-up

How would you test tokenizer behavior and retrieval quality for rare employee names?

Question 14EasyModel families

How do generative and discriminative approaches differ in NLP?

#
Show answer

A generative language model estimates how tokens or sequences are distributed and can produce new text; an autoregressive LM factorizes a sequence into conditional next-token probabilities. A discriminative classifier focuses on predicting labels or decisions from supplied inputs, such as sentiment or spam classification.

The boundary is about task and objective, not just brand names: a pretrained text model can be adapted for classification, and some generative architectures can score labels. Select the simplest solution that meets the required accuracy, latency and audit constraints.

Trade-offs

Discriminative models can be smaller and easier to calibrate for fixed labels; generative models are flexible but need constrained outputs.

Common mistake

Claiming generative models must always model a single joint distribution, or that a model can do only one class of task.

Follow-up

Would you use a generative chatbot or a discriminative classifier for a fixed two-label routing task?

Question 15MediumModel selection

How should you compare GPT model generations without relying on stale specifications?

#
Show answer

Compare the exact model IDs and documented version capabilities, not just names such as GPT-3 or GPT-4. Important dimensions are supported modalities, maximum input/output budgets, tool and structured-output support, latency, cost and task-specific evaluation results. Earlier GPT-4 releases introduced stronger multimodal capability than the original GPT-3 API family, but those product families include different variants and limits.

Run a fixed evaluation set with your actual prompts and data; include refusal, correctness, throughput and price-per-successful-task. Avoid repeating old context-length numbers as if they apply to every model release.

Trade-offs

Benchmark strength may not match your language, domain, document structure or operational constraints.

Common mistake

Attributing one historic context length or price to an entire evolving model family.

Follow-up

What evidence would justify a more expensive model for an internal assistant?

Question 16MediumMultimodal models

What can you say about Gemini multimodal capabilities without inventing internal training details?

#
Show answer

Gemini is a model family offered with variants supporting text and, depending on model and endpoint, modalities such as images, audio and video. Multimodal applications must still handle preprocessing, token or media limits, latency and evaluation for each modality. Check the official capability matrix for the exact version before choosing a model.

Do not claim unreported internal architecture, training recipes or superiority over another provider. Public model documentation describes exposed capabilities, not every proprietary training mechanism. A meaningful engineering comparison tests the same image/audio tasks, quality rubric, costs and failure cases.

Trade-offs

One multimodal endpoint can simplify a pipeline, but specialized components may deliver better control over individual stages.

Common mistake

Treating vendor marketing or an unverified training claim as a reproducible architecture fact.

Follow-up

How would you compare two multimodal APIs on diagrams with tiny labels or noisy meeting audio?

Question 17EasyModel families

What categories of foundation models are useful to distinguish in system design?

#
Show answer

Foundation models are pretrained on broad data and adapted to many tasks. Useful distinctions include text/code models, vision encoders, speech/audio models, image generation systems and multimodal models that accept or emit multiple forms of information. Architectures may be encoder-only, decoder-only, encoder-decoder, diffusion-based or hybrids.

Choose by the input-output contract, not a single label: a document pipeline may need OCR/layout analysis plus language reasoning, while a voice assistant must meet streaming and interruption constraints. A large language model is one category within the broader foundation-model family.

Trade-offs

A single model can simplify orchestration; separate specialized models may make quality and safety easier to measure.

Common mistake

Calling every pretrained vision or diffusion model an LLM.

Follow-up

Which components would you combine to extract a table from an image and answer a question about it?

Question 18MediumPrompting

When do step-oriented prompts help, and how do you validate reasoning outputs?

#
Show answer

For multi-stage problems, decomposing the task into explicit subproblems, requesting concise justifications or adding executable intermediate checks can improve results. For arithmetic, let code validate calculations; for retrieval, check citations against the supplied source IDs. The actual model’s hidden reasoning is not a reliable external audit trail simply because a response sounds methodical.

Measure final-answer correctness and verifiable intermediate artifacts on a held-out set. Do not require long internal reasoning transcripts as a substitute for evidence. Use task decomposition, structured results and tools where they materially improve correctness.

Trade-offs

More elaborate decomposition can improve hard tasks while adding tokens, latency and compounding errors.

Common mistake

Treating a fluent chain of reasoning as proof that each conclusion is correct.

Follow-up

How would you verify a multi-step pricing answer without trusting the model’s explanation?

Question 19EasyPrompting

What is zero-shot prompting, and how would you measure whether it works?

#
Show answer

Zero-shot prompting gives a pretrained model a task instruction without labelled task demonstrations in the prompt. For instance, ask a model to classify a support message into Billing, Technical or Other and describe the allowed output labels. The model relies on knowledge learned during training and instruction tuning rather than examples placed in that request.

Test on representative messages with human-labelled expected categories. Compare accuracy, per-class recall and abstentions rather than assuming that an apparently reasonable example proves the prompt works. When failures cluster around ambiguous labels, refine the instructions or add examples.

Trade-offs

Zero-shot is compact and cheap but may be less robust for domain-specific or ambiguous labels.

Common mistake

Describing zero-shot prompting as proof the model has never seen a similar task during pretraining.

Follow-up

How would you measure whether adding three demonstrations improves the classifier?

Question 20EasyPrompting

How does few-shot prompting work, and when can examples make it worse?

#
Show answer

Few-shot prompting includes a small set of labelled examples in the current request so the model can infer an output pattern or decision boundary. Examples should represent real edge cases and match the desired schema. With an invoice-category task, show both routine invoices and ambiguous exceptions rather than three easy, near-identical ones.

Measure performance against a zero-shot baseline on held-out records. Poor or biased examples can anchor wrong behavior; lengthy demonstrations consume context and raise prefill cost. Keep examples versioned and avoid putting private user information into reusable prompts.

Trade-offs

Examples can reduce ambiguity but increase prompt cost and can overfit the model’s behavior to a few patterns.

Common mistake

Confusing few-shot in-context examples with weight updates from fine-tuning.

Follow-up

How would you choose demonstrations when the minority error class is the most expensive mistake?

Question 21MediumModel families

How do modern LLMs differ from traditional n-gram language models?

#
Show answer

An n-gram model estimates the next-token distribution from a short fixed-length history using corpus counts and smoothing. It is not necessarily trained with supervised labels. A neural language model learns dense representations and parameterized context interactions; transformers use attention to condition on far wider preceding context while training many positions efficiently.

LLMs generally offer better generalization across language tasks but bring substantial compute, operational and governance requirements. N-gram baselines can still be attractive for constrained, explainable or offline tasks. Explain the distinction in terms of modeling assumptions and deployment trade-offs rather than dismissing older approaches.

Trade-offs

Classical models are cheap and inspectable, while large neural models generalize better but require capacity planning and evaluation.

Common mistake

Claiming n-gram models are inherently supervised or unable to model text probabilities.

Follow-up

When would a small statistical model beat an API-hosted LLM for a fixed task?

Question 22MediumModel optimization

How do distillation and quantization differ when reducing inference cost?

#
Show answer

Quantization uses lower-precision representations, often for model weights, to reduce memory requirements. Its speed benefit depends on the hardware and supported kernels. Distillation trains a student to learn from a teacher; it changes the model you deploy and introduces training and data costs.

Compare task quality, memory, latency and throughput on the same workload. Neither method guarantees unchanged quality, and they can be combined. Do not judge a smaller model solely by parameter count or assume lower precision always makes serving faster.

Question 23MediumLoRA

What does LoRA change, and why is it not a substitute for fresh retrieved knowledge?

#
Show answer

LoRA freezes the base weights and trains low-rank updates for selected weight matrices. It reduces the number of trainable parameters compared with full fine-tuning; it does not make training free or guarantee better quality. Record the adapter and compatible base-model version together.

Use adaptation for a demonstrated behavior or task gap, with held-out evaluation. Frequently changing facts may be better supplied through governed retrieval. Compare the adapter with prompting and retrieval baselines, and test regressions beyond the training examples.

Question 24EasyModel types

What is the difference between a base model and an instruction-tuned model?

#

A team downloads an open model, asks it a question, and it continues the question instead of answering.

Show answer

A base model is trained to predict the next token on large amounts of text, so it continues whatever you give it. An instruction-tuned (chat) model has further training, typically supervised examples and preference tuning, to follow instructions and answer in a conversation format.

Instruction-tuned models expect a specific chat template with role markers; sending raw text or the wrong template quietly hurts quality. Use the tokenizer's chat template. Base models are useful as starting points for your own fine-tuning or for pure completion tasks; for assistants, start from the instruct version.

Trade-offs

Base models are more flexible for fine-tuning but need more work before they behave like assistants.

Common mistake

Formatting prompts for one model family's chat template and using them with another.

Follow-up

Why might you fine-tune from the base model rather than the instruct model?

Question 25MediumTokenization

Why does the same text cost more tokens in some languages, and why does it matter?

#

A support bot costs noticeably more per conversation for Telugu and Hindi users than for English users.

Show answer

Tokenizers learn their vocabulary from training data. Languages and scripts that were less common in that data are split into more, shorter tokens, so the same meaning takes more tokens.

That affects cost, latency and how much fits in the context window, and it can quietly make the product worse for some users. Measure tokens per message by language, budget accordingly, compare tokenizers when choosing a model, and test quality in each language rather than only in English.

Trade-offs

Translating to English first can reduce tokens but adds latency and can lose meaning.

Common mistake

Estimating cost from English word counts for a multilingual product.

Follow-up

How would you set a per-language context budget?

Question 26MediumModel choice

What does a reasoning model do differently, and when is it worth the cost?

#

A team wants to switch every endpoint, including tagging, to a reasoning model.

Show answer

Reasoning models generate intermediate reasoning before the final answer. That helps on multi-step problems such as planning, debugging and complex analysis, but the extra tokens add latency and cost, and many APIs let you set the reasoning effort.

Use them where your evaluations show a real accuracy gain, and keep fast, cheaper models for simple classification, extraction or formatting. Route by task, and measure quality, latency and cost per task together.

Trade-offs

Higher effort improves hard tasks but can slow simple ones without changing their accuracy.

Common mistake

Switching everything to the most capable model without measuring the gain per task.

Follow-up

How would you route requests between a fast model and a reasoning model?

Question 27MediumContext windows

A model advertises a 128K-token context window, and the needle-in-a-haystack chart is all green. Your contract-review feature still misses facts beyond the first few thousand tokens. Why is the advertised number not the usable number?

#

Questions often need two clauses from different sections combined.

Show answer

Needle-in-a-haystack tests one easy skill: finding a single distinct sentence. Real tasks need several facts, from different places, combined. The RULER benchmark (Hsieh et al.) was built for this gap: it adds variations with different types and numbers of needles and new task categories such as multi-hop tracing and aggregation. Evaluating 17 long-context models, all claiming 32K tokens or more, it found that only half could maintain satisfactory performance at 32K.

So treat the advertised window as an upper limit on input size, not as a promise of quality. Measure the effective context for your task: build an eval with your own documents where the needed clauses sit at different depths and must be combined, and plot accuracy against input length and position. Position matters too: other work found models use information at the beginning or end of the context better than in the middle.

Design from the measured number. Keep inputs within the length where accuracy holds, retrieve the relevant sections instead of sending the whole contract, or split the task (extract per section, then combine). Re-measure when you change model, because effective context varies widely between models with the same advertised window.

Trade-offs

Sending everything is simple and avoids retrieval misses, but beyond the effective length it costs more and answers worse.

Common mistake

Choosing a model on its advertised context length and a single-needle chart.

Follow-up

How would you build a cheap long-context eval from your own documents with known answers?

Question 28MediumTokenization

Finance approved a self-hosted model budget from token estimates made with one vendor's tokenizer library. After launch, real token counts, cost and context overflows are all off. What went wrong and how do you estimate correctly?

#

Estimates were made with tiktoken; production runs an open-weights model with its own tokenizer and chat template.

Show answer

Token counts are a property of a tokenizer, and different model families use different ones with different vocabularies. Llama 3, for example, uses a vocabulary of 128,000 tokens; another model's tokenizer splits the same text differently, especially for code, numbers and non-English text. Counting with one family's tokenizer and serving another's makes every estimate wrong by a ratio that depends on the content.

There are two more gaps. The chat template adds tokens for roles, separators and special markers to every message, and a long system prompt or tool schema is repeated on every request. And output length depends on the model's style and sampling defaults, not just on the prompt.

Estimate with the production model's own tokenizer and chat template, on a sample of real requests including system prompts, tool definitions and retrieved context. Report input and output tokens separately, as distributions (median and p95), and per language if you serve several. Then set context limits from those measurements, with margin, and monitor actual counts from the serving engine after launch. Any model change means re-measuring, because the tokenizer may have changed with it.

Trade-offs

Measuring with the real tokenizer and templates takes setup time, but rough estimates mis-size both budgets and context limits.

Common mistake

Treating tokens as a universal unit across model families, or counting only the user message.

Follow-up

How would you budget for tool schemas and retrieved context that grow over time?

Question 29HardTokenization

A user pastes text containing a chat-template marker such as <|im_end|> followed by fake system instructions, and the model starts obeying them. How can plain text become a control token, and how do you prevent it?

#

The service builds prompts by concatenating strings and tokenizes the result itself before sending token IDs to the model.

Show answer

Chat templates use special tokens to mark where system, user and assistant turns begin and end. If the service tokenizes user text in a mode that converts special-token strings into the real special token IDs, a user can close their own turn and open a fake system turn. The model then sees genuine control tokens and treats the injected text as higher-priority instructions.

Tokenizers differ in their defaults, so check yours. tiktoken is a good example of a safe default: its encode method raises an error by default if the text contains anything corresponding to a special token (disallowed_special defaults to "all" and allowed_special to an empty set). Setting disallowed_special=() encodes such strings as ordinary text instead, which is what encode_ordinary does; setting allowed_special="all" turns them into real special tokens, which is exactly what you must never do with user input.

Prevent it structurally. Build prompts from structured messages through the model's chat template, so roles are set by your code, not by text. Encode every piece of untrusted text (user input, retrieved documents, tool output) as ordinary text with special tokens disabled, and add a test that sends every special-token string of your model's tokenizer through the public API and checks that they come back as plain text. This closes a structural hole; it does not stop ordinary prompt injection, which still needs the usual defences.

Trade-offs

Rejecting text with special-token strings is simplest but can refuse legitimate content such as documentation; encoding it as plain text is friendlier and equally safe.

Common mistake

Concatenating raw strings into a templated prompt and tokenizing the whole thing with special tokens allowed.

Follow-up

Which other places in an agent pipeline (tool results, retrieved pages, file names) can carry special-token strings into the prompt?