supraj.dev

Interview questions & answers

LLM Evals & Observability

Golden sets, judges, CI gates and traces that explain a bad answer.

  • 30 questions
  • 1 easy
  • 15 medium
  • 14 hard
  • 1 related article

30 questions

Question 1EasyDatasets

What goes into a good golden dataset for an LLM feature?

#

A team tests its summariser with ten hand-picked examples that always pass.

Show answer

Use real inputs, sampled from production or realistic user research, not examples written to pass. Cover the main use cases in proportion, then add edge cases, known past failures, adversarial inputs and cases where the right answer is to refuse or say "I don't know".

For each item, store the input, the expected outcome or grading criteria, and tags such as language or difficulty so you can slice results. Version the set, keep it out of any training data, and add every new production bug as a case.

Trade-offs

Larger sets give more confidence but cost more to label and to run; start small and grow from failures.

Common mistake

Using the same examples to write the prompt and to evaluate it.

Follow-up

How would you keep the dataset representative as user behaviour changes?

Question 2MediumOpen-ended tasks

How do you evaluate a summariser when there is no single correct answer?

#

Two summaries of the same incident report are both reasonable, but one leaves out the root cause.

Show answer

Turn "good" into checkable criteria: faithful to the source with no invented facts, includes the required items such as root cause and impact, fits the length and audience. Score each criterion separately rather than giving one overall score.

Some checks can be automatic, such as length and required fields; others need a judge, human or model, with a written rubric. Pairwise comparison of two versions on the same inputs is often more reliable than absolute scores. Calibrate any model judge against human ratings on a sample.

Trade-offs

Criteria-based scoring is more work to set up but tells you what to fix, unlike a single score.

Common mistake

Using word-overlap metrics against one reference summary as the main quality signal.

Follow-up

How would you detect invented facts in a summary automatically?

Question 3MediumProduction

Offline evals pass but users still complain. What is missing?

#

The evaluation score went up last release; support tickets about wrong answers also went up.

Show answer

The dataset probably does not match production any more: new kinds of questions, other languages, longer conversations or different documents. Offline evaluations measure only what they contain.

Sample production traces regularly, group the failures, and add them to the dataset. Collect online signals such as thumbs down, retries, escalations and abandoned sessions, and track them per feature and per release. Use online results to decide what to add offline, so the two stay connected.

Trade-offs

Production sampling needs privacy controls and redaction before anything lands in a dataset.

Common mistake

Raising the offline score by tuning against the same fixed set.

Follow-up

Which online signal would you trust most for your product, and why?

Question 4HardCI

How would you gate deployments on evaluations in CI?

#

Prompt and model changes ship daily. You want regressions to block merges without flaky failures.

Show answer

Run a fast smoke set on every change and a full set before release. Gate on no significant regression compared with the current production version, per important slice, rather than on one absolute number.

Because outputs vary, run nondeterministic cases several times or use pass rates, and set thresholds with that noise in mind. Keep hard checks such as schema validity and safety refusals as strict pass/fail. Report results in the pull request with examples of failures, and cap evaluation cost per run.

Trade-offs

Strict gates prevent regressions but can block useful changes when the dataset is noisy.

Common mistake

Failing the build on a one-point drop that is within normal run-to-run variation.

Follow-up

How would you handle a change that improves one slice but hurts another?

Question 5MediumDatasets

How do you build an evaluation set before you have any production traffic?

#

A new internal assistant launches in a month. There are no logs yet.

Show answer

Start with people who know the work: ask experts and future users to write real questions they have, and mine existing sources such as support tickets, search logs, chat channels or FAQs for real phrasing. Write expected outcomes or grading criteria with the experts.

Use a model to generate variations and edge cases, but review them; synthetic data alone drifts toward easy, uniform questions. Run an internal pilot and add its failures. Treat this first set as temporary and replace it with sampled production data as soon as you launch.

Trade-offs

Synthetic examples scale cheaply but can miss how real users ask; keep a human-written core.

Common mistake

Letting the same model generate both the questions and the answers it is later graded against.

Follow-up

How many expert-written examples would you want before the pilot?

Question 6HardAgents

How do you evaluate an agent rather than a single response?

#

An airline-support agent must change bookings correctly through several tool calls.

Show answer

Run the agent against a realistic environment with simulated users and tools, and check the end state: was the booking changed correctly, were policies followed, were no forbidden actions taken. Also look at the trajectory: tool choices, arguments, number of steps and cost.

Agents are inconsistent, so run each task several times. The τ-bench benchmark introduced pass^k, the chance that all k attempts succeed, which shows reliability better than a single success rate. Track cost and latency per task alongside success.

Trade-offs

Simulated environments take effort to build, but without them agent tests either touch real systems or check nothing.

Common mistake

Judging only the agent's final message, which can sound right while the booking is wrong.

Follow-up

How would you simulate a user who changes their mind halfway through?

Question 7MediumStatistics

With 200 examples, can you trust a 3-point improvement?

#

Version B scores 83% and version A scores 80% on a 200-item set.

Show answer

Probably not on its own. The standard error of an accuracy p on n items is about √(p(1−p)/n). At 80% on 200 items that is √(0.16/200) ≈ 2.8 points, and the difference between two independent scores has an even larger error, around 4 points. A 3-point gap is within noise.

To decide, compare the versions on the same items (a paired comparison is more sensitive), report confidence intervals, run more items, or repeat runs for nondeterministic outputs. Evaluation-statistics work makes the same recommendation: report error bars, not single scores.

Trade-offs

Larger evaluation sets give tighter intervals but cost more per run.

Common mistake

Declaring a winner from a single run on a small set.

Follow-up

How many items would you need to detect a 3-point difference reliably?

Question 8HardLLM as judge

Your pairwise LLM judge says the new prompt beats the old one 62% of the time. A teammate reruns the same comparison with the two answers shown in the opposite order, and the win rate drops to 49%. What happened, and how should you run pairwise judging from now on?

#

The judge prompt always showed the new answer as "Response A" in the first run. No other setting changed.

Show answer

That is position bias: the judge prefers an answer because of its slot. The MT-Bench paper measured it by asking judges the same question with the answers swapped. With its default prompt, GPT-4 gave the same verdict both ways in 65.0% of cases and favoured the first position in 30.0%; Claude-v1 was consistent in 23.8% and GPT-3.5 in 46.2%. Those are older models, so measure your own judge, but the lesson holds: a single-order pairwise result mixes quality with slot preference.

How to run it: - Judge every pair twice, once in each order. The paper's suggestion is to count a win only when both orders agree, and to call it a tie when they disagree. - Randomise which answer appears first in any single-order run, and never put the candidate in the same slot every time. - Track the judge's consistency rate (both orders agree) as a health metric for the judge itself. If it falls, the judge or prompt changed. - Report wins, losses and ties separately. A 62% win rate with many inconsistent pairs hidden inside it is weaker evidence than it looks.

Here, the 62% was probably inflated by the new answer always sitting in the favoured slot. Rerun both orders before deciding.

Trade-offs

Judging both orders doubles judge cost and produces more ties, but it removes a bias that can flip the conclusion.

Common mistake

Always placing the candidate as Response A and reading the win rate as pure quality.

Follow-up

How would you decide whether ties caused by inconsistency should count as half a win or be dropped?

Question 9HardLLM as judge

A new model wins 70% of pairwise comparisons against your current model on an automatic judge, but its answers are about twice as long, and your product wants short answers. How do you tell whether it is better or just longer?

#

The judge is a strong model with a generic helpfulness rubric. You have the token count of every answer.

Show answer

LLM judges tend to favour longer answers. The MT-Bench paper calls this verbosity bias and tested it with a "repetitive list" attack, where an answer is padded with repeated content: Claude-v1 and GPT-3.5 were fooled 91.3% of the time and GPT-4 8.7%. Even a good judge leans toward length, and a long answer looks thorough.

Two ways to separate quality from length: - Control for length statistically. Length-Controlled AlpacaEval fits a generalised linear model that predicts the judge's preference from the length difference and other features, then predicts the preference with the length difference set to zero. The paper reports this raised the Spearman correlation with Chatbot Arena from 0.94 to 0.98. You can do a simpler version: model win probability against length difference and read the win rate at zero difference. - Control length in the experiment. Give both models the same length guidance, or compare answers in matching length bands, and check whether the win rate survives.

Also put your product's requirement in the rubric, such as "answers longer than needed are worse", and check a sample by hand.

If the length-controlled win rate is near 50%, the new model is mostly winning on length, which your users do not want.

Trade-offs

Statistical length control keeps all the data but depends on the model you fit; matched-length comparisons are easier to explain but use less data.

Common mistake

Adding "ignore length" to the judge prompt and assuming the bias is gone without measuring it.

Follow-up

Which other surface features besides length might a judge reward, and how would you test for them?

Question 10HardLLM as judge

A small model from a vendor tops your internal leaderboard, which is scored by an LLM judge. Reading its outputs, you see oddly structured text that does not really answer the questions. How can that happen, and how do you protect the leaderboard?

#

Vendors can submit outputs. The judge prompt and rubric have not changed for a year and are known to several partners.

Show answer

The judge is being gamed. Text in the candidate answer reaches the judge model, so it can steer the verdict the same way a prompt injection steers an agent. The "Null Models" paper showed how far this goes: a model that always returns the same constant response, unrelated to the instruction, reached an 86.5% length-controlled win rate on AlpacaEval 2.0, 83.0 on Arena-Hard-Auto and 9.55 on MT-Bench.

Protect the leaderboard by treating outputs as untrusted: - Wrap the candidate answer in clear delimiters and tell the judge it is data to assess, never instructions. This helps but is not enough on its own. - Add adversarial baselines to every run: a constant answer, an empty answer, an off-topic answer and an answer that only praises itself. If any scores well, the judge is broken. - Check relevance separately, with a cheap check that the answer addresses the question, before quality scoring. - Keep the judge prompt private, rotate it, and use more than one judge family for high-stakes rankings. - Read top-ranked outputs by hand before announcing results, and use deterministic checks where answers are verifiable.

The same failure happens inside your own team when people tune prompts against the judge for months: they learn what the judge rewards, not what users want.

Trade-offs

Private, rotating judges and adversarial baselines make rankings harder to game but less reproducible for outsiders.

Common mistake

Assuming a strong judge model cannot be manipulated by the text it is grading.

Follow-up

How would you detect, after the fact, that past leaderboard entries were gaming the judge?

Question 11MediumLLM as judge

Your LLM judge scores answers from 1 to 10, and almost every answer gets a 7 or an 8. The scores cannot separate a good prompt from a bad one. How would you redesign the scoring?

#

The rubric says "rate overall quality from 1 to 10". Scores are averaged and compared to one decimal place.

Show answer

Wide numeric scales collapse onto a few values. The G-Eval paper observed that when a model outputs a score directly, one value usually dominates (for example 3 on a 1 to 5 scale), which gives low variance and low correlation with humans, and that models usually output integers even when asked for decimals.

Redesign options: - Split the score into specific yes/no checks: "states the refund window correctly", "cites a source", "asks for missing information instead of guessing". Binary checks are easier for both judges and humans to agree on, and they tell you what got worse. - If you need a graded score, anchor each level with a written description and an example answer. - Use pairwise comparison when the question is "is version B better than A?"; it is often more sensitive than two absolute scores. - G-Eval's own fix needs token probabilities: it computes the score as the probability-weighted sum of the possible scores, so a judge that is torn between 7 and 8 gives 7.5. That only works if your judge API returns log probabilities for the score tokens.

Whatever you choose, validate against human labels before using it. G-Eval also reported that its judge always scored model-written summaries higher than human-written ones, a reminder that a judge has preferences of its own.

Trade-offs

Binary checks are reliable and explainable but need more rubric work; probability-weighted scores are smoother but depend on log probabilities and still need validation.

Common mistake

Reporting a change from 7.4 to 7.6 on a compressed 1 to 10 scale as an improvement.

Follow-up

How would you choose which binary checks to include for a customer support assistant?

Question 12MediumLLM as judge

Your judge marks wrong answers to calculation-heavy finance questions as correct, even though the same model gets the calculations right when you ask it directly. Why does this happen, and how would you grade these questions?

#

The judge sees the question and the candidate answer with its working, and decides correct or incorrect.

Show answer

When the judge reads a confident worked answer, it tends to follow the candidate's reasoning instead of checking it independently. The MT-Bench paper found this "limited capability in grading math and reasoning questions": on 10 math questions (20 judgments), the default judge prompt failed 70% of the time, a chain-of-thought prompt 30%, and a reference-guided prompt 15%. In the reference-guided method, the judge first answers the question on its own, and that answer is then shown as the reference when it grades.

For finance calculations, go further: - Do not use a judge for the number at all if you can avoid it. Store the correct value, extract the final number from the answer, and compare it with a tolerance and unit check. Code is cheaper and does not get persuaded. - If answers are free-form, use the judge only to extract the final figure, then compare in code. - Use a judge for the parts that need judgment, such as whether the explanation is clear or the assumptions are stated, and keep that separate from correctness. - Keep a small set of known-wrong answers with convincing working in your judge tests, to see whether the judge can be talked into them.

Trade-offs

Deterministic checks need reference values and parsing work; judges cover open answers but are weaker at verifying reasoning.

Common mistake

Assuming a judge that can solve a problem will reliably catch a wrong solution to it.

Follow-up

How would you build reference answers for a few thousand finance questions without hand-solving each one?

Question 13MediumRegression gates

Your nightly eval pass rate fell from 88% to 81% with no change to the product. The cause: the judge was configured with a model alias, and the alias moved to a newer model. How should judge models be managed?

#

Release gates compare tonight's pass rate against a baseline computed months ago.

Show answer

Treat the judge like a measuring instrument with a calibration record. If it changes, every score after the change is on a different scale.

Practices: - Pin the judge to a dated model snapshot, not an alias, and version the judge prompt and rubric. Record the judge model, prompt version and sampling settings in every eval run. - Never compare scores produced by different judge versions. A gate should compare candidate and baseline scored by the same judge in the same run, not tonight's score against a months-old number. - Upgrade the judge on purpose. Rerun it on a frozen, human-labelled calibration set, check agreement per class (especially on failures), and only then switch. - When you switch, rescore the current production version with the new judge to set a fresh baseline. - Keep the raw judge outputs and reasons, so you can explain why a score moved.

Plan for the judge model to be retired too, since providers deprecate models; a calibration set lets you move to a new judge without guessing.

Trade-offs

Pinning and calibration add process and some cost, but without them a score change cannot be traced to the product or the judge.

Common mistake

Gating releases on an absolute score from an unpinned judge.

Follow-up

How large should a judge calibration set be, and how often would you refresh it?

Question 14MediumBenchmarks

You are choosing between two frontier models. Their MMLU scores differ by half a point, both near the top of the scale. A manager wants to pick the higher one. Does that difference mean anything?

#

Both scores come from vendor model cards. Your use case is internal policy question answering.

Show answer

Probably not. Near the top of a saturated benchmark, small differences are within label noise.

Two papers show why: - MMLU-Redux re-annotated 5,700 MMLU questions and estimates that 6.49% of MMLU questions contain errors; in the Virology subset, 57% of the analysed questions had errors. When the remaining headroom is close to the error rate, a higher score can mean agreeing with wrong labels. - MMLU-Pro was built because MMLU stopped separating strong models. It moves from four answer options to ten and adds harder reasoning questions; the authors report accuracy drops of 16% to 33% compared with MMLU, and lower sensitivity to prompt wording (from 4-5% on MMLU to 2%).

What to recommend: - Use public benchmarks only as a coarse filter for a shortlist, and prefer harder or newer ones that still spread models out. - Make the decision on your own task: a few hundred real policy questions with graded answers, plus cost and latency. - Ask for confidence intervals and check how the scores were produced (prompt format, few-shot or not, chain of thought or not), since vendor numbers are often not comparable. - Remember contamination: an old public benchmark may be in the training data.

Trade-offs

Your own eval costs time to build but measures what you care about; public numbers are free but saturated, noisy and possibly contaminated.

Common mistake

Ranking models by decimal differences on a saturated benchmark.

Follow-up

What would make you trust a public benchmark result enough to skip your own eval?

Question 15HardContamination

A closed vendor model scores far higher on a public coding benchmark than on your own internal coding tasks of similar difficulty. You cannot see its training data. How would you test whether the benchmark score is inflated by contamination?

#

The benchmark was published two years ago and its solutions are widely posted online. The vendor publishes a training cutoff date.

Show answer

Use time. If the model has seen a problem in training, it scores well on problems from before its cutoff and worse on comparable problems released afterwards.

LiveCodeBench is built on this idea. It continuously collects problems from LeetCode, AtCoder and CodeForces contests with their release dates, so you can evaluate a model only on problems released after its cutoff. The paper reports that DeepSeek's performance "starkly drops" on LeetCode problems released after August 2023, and that GPT-4o drops on problems released since November 2023, its stated cutoff. The drops were mainly on LeetCode, with smoother trends on AtCoder.

Your test: - Split comparable problems into before and after the vendor's cutoff, and compare scores. A sharp drop is a strong contamination signal. - Perturb benchmark items: rename variables, change numbers or constraints while keeping difficulty. A memorising model drops on the perturbed versions. - Build a small private set of fresh problems and never publish it.

This is different from checking your own fine-tuning data for leakage, where you can search the training set directly. With a closed model, behavioural tests like these are all you have, and none proves contamination alone; look for several signals together.

Trade-offs

Time-split sets are clean but small and need constant refreshing; perturbation tests are cheap but can change difficulty by accident.

Common mistake

Treating a high score on an old public benchmark as evidence of skill on your tasks.

Follow-up

How would you check that post-cutoff problems are as hard as pre-cutoff ones before comparing?

Question 16MediumContamination

Your team is about to publish the internal benchmark it built for its domain. How do you keep it useful for measuring models once it is public?

#

The benchmark has 2,000 questions with reference answers. It will be posted on a public code host.

Show answer

Once a benchmark is public, plan for it to leak into training data and keep a part that never leaves.

- Keep a private test split. Publish a development split and keep a held-out split from the same distribution that only you run. If a model scores much higher on the public split than on the private one, suspect leakage. - Mark the files. BIG-bench puts a canary string in every task file, with the stated aim "to prevent benchmark tasks from leaking into web-scraped training data". It only helps if data builders filter on it, but it is cheap and also lets people search for your data. - Do not publish reference answers next to questions in plain text if you can avoid it; provide a scoring service or keep answers in the private split. - Add dated items over time. LiveCodeBench records each problem's release date so models can be scored only on problems after their training cutoff. A refreshed benchmark stays meaningful longer. - Version the benchmark and report which version and split each score uses.

Expect the public split to stop being informative after a while, and budget for building new items.

Trade-offs

A private split keeps results trustworthy but makes them hard for others to reproduce; a fully public set is easy to use but leaks.

Common mistake

Publishing every question with its answer and continuing to treat new scores on it as clean.

Follow-up

How would you let outside teams score against the private split without giving it to them?

Question 17HardStatistics

Before you build a new eval, a product manager asks how many questions it needs to detect a 3-point improvement in accuracy. How do you answer?

#

You have a pilot of 100 questions scored for the current model and one candidate.

Show answer

This is a power analysis. Choose the significance level, the power you want (the chance of detecting a real effect of that size) and the minimum detectable effect, then estimate the variances from a pilot.

The "Adding Error Bars to Evals" paper gives the formula for comparing two models on the same questions: n = (z for alpha/2 + z for beta)² × (ω² + σA²/KA + σB²/KB) / δ², where δ is the minimum detectable effect, ω² is the variance of the paired difference across questions, and the σ²/K terms are within-question variance from sampling each answer K times. In its worked example (parameters the authors call fictional but reasonable), detecting δ = 0.03 at alpha = 0.05 and 80% power needs about 969 questions, and the paper concludes new evals should contain at least 1,000 questions.

Levers that reduce the number: - Paired comparison: score both models on the same questions. Question scores are usually positively correlated across models, which the paper calls a "free" variance reduction. - Resample answers several times per question to reduce within-question variance; once that term is small, more resampling does little. - Avoid clustered questions (many questions about one document); they reduce the effective sample size.

Then give the manager the inverse too: with the budget you have, the smallest effect the eval can detect. If that is 6 points, say so before anyone reads meaning into a 3-point change.

Trade-offs

Larger evals detect smaller effects but cost more to build, label and run; paired designs and resampling buy power more cheaply.

Common mistake

Building a 200-question set and then reading 2-point changes as real.

Follow-up

How would you estimate the variance terms when you have no pilot data yet?

Question 18HardStatistics

You report pass@10 for a code generation model by drawing exactly 10 samples per problem and counting a problem as solved if any sample passes the tests. A colleague says the number is noisy and should be computed differently. What is the standard estimator, and when is pass@k the wrong metric?

#

Each sample costs one model call plus a test run. You also want pass@1 from the same run.

Show answer

Drawing exactly k samples gives an unbiased but high-variance estimate. The standard approach, from the HumanEval work, is to draw n samples per problem with n at least k, count the c that pass, and estimate pass@k for that problem as 1 − C(n−c, k) / C(n, k): the chance that a random subset of k of your n samples contains at least one passing one. Average over problems. The reference code in openai/human-eval computes the same value in a numerically stable way, as 1 − prod(1 − k / arange(n − c + 1, n + 1)), and evaluates k of 1, 10 and 100 by default.

With one set of n samples you get pass@1, pass@10 and so on together, with lower variance than separate small runs. Do not use the shortcut 1 − (1 − p)^k with p estimated as c/n: plugging a noisy estimate into a non-linear formula gives a biased result (it underestimates).

When it is the wrong metric: - pass@k assumes something picks the passing sample, such as hidden tests or a verifier. For a user who sees one answer, pass@1 matters. - For agents that must succeed every time, pass^k (all k attempts succeed) measures reliability instead. - pass@k depends on sampling temperature, so report it, and compare models at the same settings.

Trade-offs

Larger n gives more precise estimates of every pass@k at higher sampling and test cost.

Common mistake

Comparing pass@10 numbers computed from 10 samples each, or computed at different temperatures.

Follow-up

How would you choose n for a budget of 20,000 sampled solutions across 200 problems?

Question 19HardStatistics

The team tried 40 prompt variants on the same 300-question golden set and shipped the best one, which scored 4 points above the baseline. In production, quality did not change. What went wrong?

#

All 40 variants were scored once. The golden set has also been used for every prompt decision for the past six months.

Show answer

Two related problems.

Winner's curse. Each score has noise: on 300 questions, the standard error of one accuracy is a couple of points. Pick the maximum of 40 noisy scores and you pick the variant that got lucky as well as any that is truly better, so its score overstates its real gain. With that many tries, a 4-point lead can be mostly noise.

Overfitting the eval set. After six months of decisions made on the same 300 questions, prompts have been tuned to those questions. The set has stopped being an independent test and become training data for the prompt.

Fixes: - Separate selection from confirmation. Search on a development set, then evaluate only the chosen variant (and the baseline) once on a held-out test set nobody tunes against. Report the held-out number. - Decide the comparison before looking: one candidate versus baseline, paired on the same items, with a confidence interval. - If you must report many comparisons, correct for them (for example Holm or Bonferroni) or treat the results as hypotheses to confirm. - Refresh the held-out set from recent production traffic, and keep a set that only a release gate ever sees. - Expect smaller real gains than the search showed.

Trade-offs

Holding data back for confirmation leaves less for searching, but without it you cannot tell real gains from lucky picks.

Common mistake

Reporting the score of the selected variant on the selection set as the expected improvement.

Follow-up

How would you size the held-out set so it can confirm a 2-point gain?

Question 20MediumEval design

You need two things from evals: to track quality of your assistant week by week for a year, and to decide quickly whether a candidate model beats the current one. Would you use pointwise scoring, pairwise comparison, or both?

#

You have an LLM judge and a small budget for human review each month.

Show answer

Use both, for different jobs.

Pairwise comparison (which of these two answers is better?) is usually more sensitive for launch decisions: judges and humans find relative choices easier than absolute scores, and comparing the same inputs removes much of the noise. Chatbot Arena is built on this approach, collecting crowdsourced pairwise votes and turning them into rankings with statistical models; it reports that the votes agree well with expert raters. The costs: results are only relative (you learn A beats B, not whether either is good enough), comparing many versions needs many pairs or a model such as Bradley-Terry to combine them, and pairwise judges have position and length biases you must control.

Pointwise scoring (does this answer pass each criterion?) gives absolute numbers you can track over time and set thresholds on. Its risks are scale drift and compressed scores, so use specific binary checks rather than broad 1 to 10 ratings, and keep the judge pinned.

A practical design: - Weekly tracking: pointwise binary checks on a stable, versioned set, plus a sample of live traffic. - Launch decisions: pairwise candidate versus production on the same inputs, judged in both orders, with human review on disagreements. - Spend the human budget on calibrating the judge in both modes.

Trade-offs

Pairwise is more sensitive but relative and bias-prone; pointwise is trackable over time but coarser and drifts if the judge or rubric changes.

Common mistake

Using pairwise win rates to claim a model is "good enough" when it only beat a weak baseline.

Follow-up

How would you rank six candidate prompts without running all fifteen pairs?

Question 21MediumHuman evaluation

Three annotators label chatbot answers as helpful or not. Their pairwise Cohen's kappa is about 0.3. Your plan was to use these labels to validate an LLM judge. Can you?

#

The rubric is one sentence: "Was the answer helpful?". Annotators worked separately with no calibration session.

Show answer

Not yet. Low agreement means the labels are noisy or the question is ambiguous, and a judge cannot be validated against a target that people do not agree on. Human agreement is the ceiling for judge agreement.

Fix the labels first: - Read the disagreements. Usually they cluster: one annotator punishes length, another rewards politeness, a third cares only about correctness. - Replace "helpful" with specific criteria, each with examples: answers the question asked, correct according to policy, asks for missing information, appropriate length. - Run a calibration session on 30 to 50 shared items, discuss disagreements, update the rubric, then label again and remeasure agreement per criterion. - Use an adjudication step for remaining disagreements, and keep a "genuinely ambiguous" tag rather than forcing a label. - For three or more raters, or when not everyone labels every item, Krippendorff's alpha is a better summary than averaging pairwise kappas.

Then compare the judge with humans on the same footing. The MT-Bench paper framed it this way: GPT-4 agreed with human experts 85% of the time on non-tie votes, compared with 81% agreement between humans, so the judge was as consistent with people as they were with each other.

Trade-offs

Detailed rubrics and calibration sessions raise agreement but cost annotator time and can narrow what gets noticed.

Common mistake

Treating majority votes from a vague rubric as ground truth for a judge.

Follow-up

What agreement level would you require before using the labels as a release gate, and why?

Question 22HardHuman evaluation

You have three clinicians and a budget of 300 human judgments to decide whether a new model writes better discharge summaries than the current one. How do you design the study?

#

Clinician time is scarce. Inputs vary from short visits to long multi-day stays.

Show answer

Design it so 300 judgments answer the question and tell you how far to trust the answer.

- Paired and blind: show each clinician both summaries for the same record side by side, with model names hidden and left/right order randomised per item. Ask which is better, allowing "tie" and "both unacceptable", and record a reason code (missing medication, wrong date, unsafe advice). - Stratify inputs by case type and length so long stays are not missed, and sample them from real records. - Split the budget: have all three clinicians judge the same 50 items (150 judgments) to measure agreement, and spread the other 150 judgments as single ratings on 150 new items. That covers 200 items in total. - Control fatigue: short sessions, random item order, and a few repeated items to check consistency. - Write the decision rule before starting, such as "ship if the new model wins more than it loses with a confidence interval above zero, and has no increase in unsafe-advice reasons". - Be honest about power: 200 paired items detect a clear preference, not a small one.

Use the labelled items afterwards to calibrate an LLM judge, so future comparisons can run at larger scale with clinicians checking only samples.

Trade-offs

More overlap gives a better agreement estimate but covers fewer items; single ratings cover more inputs but hide annotator noise.

Common mistake

Letting clinicians see which model wrote which summary, or always showing the new model on the same side.

Follow-up

How would you handle a clinician who disagrees with the other two on most overlap items?

Question 23HardAgents

Your agent eval checks that the agent calls tools in the exact sequence written in each test case. A new model scores much lower, but reviewers say it solves the tasks fine, often with fewer steps. How should the eval grade?

#

Tasks are account changes in a CRM sandbox. Expected tool sequences were recorded from the previous model.

Show answer

Grade the outcome, and check the path only for things that must never happen.

tau-bench grades this way: it compares the database state at the end of the conversation with an annotated goal state. Anthropic's engineering guide on agent evals makes the same recommendation, saying "it's often better to grade what the agent produced, not the path it took", and calls rigid step-sequence checks brittle. A sequence recorded from the old model rewards copying the old model.

A better grader for the CRM tasks: - Outcome checks in code: after the run, query the sandbox. Is the account in the expected state? Are unrelated records unchanged? - Hard constraints on the trajectory: no forbidden tools, no writes without required confirmation, no access to other customers' records, within a step or cost budget. - Model-based grading for what code cannot check, such as whether the final message to the user is accurate and polite, validated against human labels. - Efficiency metrics (steps, tokens, time) reported separately, not mixed into pass or fail.

Where several end states are acceptable, check the invariants rather than one exact state.

Trade-offs

Outcome grading allows new valid strategies but needs a resettable environment and state checks; sequence matching is easy to write but brittle.

Common mistake

Using the previous model's tool calls as the definition of correct behaviour.

Follow-up

How would you catch an agent that reaches the right end state by a risky path, such as deleting and recreating a record?

Question 24HardAgents

Your coding-agent eval score jumps 10 points on a rerun with no change to the agent, and failures in the first run cluster in one batch of tasks. What would you look for, and how do you make the eval trustworthy?

#

Trials run in parallel on shared containers. Some tasks call external package registries. The task set was written quickly from old tickets.

Show answer

Suspect the harness first.

Environment noise. Anthropic's guide on agent evals says each trial should start from a clean, isolated environment, warns that shared state can inflate scores (in one internal eval, Claude examined git history left by earlier trials), and that resource limits and flaky infrastructure cause correlated failures that are not about the agent. A batch of failures together points to that: a registry outage, rate limits, a full disk or a slow node.

Task quality. Some tasks may be impossible, underspecified or have tests that reject valid fixes. SWE-bench Verified exists for this reason: it is a human-validated subset of 500 SWE-bench instances, where annotators checked that problem descriptions are clear, test patches are correct and tasks are solvable.

Make it trustworthy: - Fresh container per trial, pinned dependencies, cached or mocked registries, and resource limits recorded. - Classify every failure as infrastructure (timeout, out of memory, 429, network) or agent, and rerun or exclude infrastructure failures, reporting how many. - Treat failures from one batch or machine as correlated when computing intervals. - Audit tasks: have a reference solution pass each one, and review tasks no model ever solves. - Track the infrastructure failure rate as its own metric.

Trade-offs

Isolation and task audits cost compute and engineering time but make score changes attributable to the agent.

Common mistake

Counting infrastructure failures as agent failures, or letting trials share a working directory.

Follow-up

How would you detect an agent that games tests instead of fixing the code?

Question 25HardLong context

A vendor says its model supports a 128K-token context and shows perfect needle-in-a-haystack results. Your contract-analysis feature gets noticeably worse once documents pass about 30K tokens. How would you evaluate long-context ability for your use case?

#

Questions often need clauses from several sections, and contracts contain many similar clauses.

Show answer

Needle-in-a-haystack tests whether a model can find one planted fact. The RULER paper calls it "indicative of only a superficial form of long-context understanding": models score almost perfectly on it, yet on RULER's harder tasks, such as multi-hop tracing and aggregation, performance drops as context grows. Of 17 models that all claimed contexts of 32K tokens or more, only about half kept satisfactory performance at 32K.

Build an eval that looks like your workload: - Real contracts at several lengths (for example 8K, 16K, 32K, 64K and 128K), with the same questions asked at each length, so you see a curve rather than one number. - Vary where the needed clauses sit (start, middle, end); models often use the middle of long inputs worse. - Include questions that need two or three clauses combined, and aggregation questions ("list every termination right"). - Add similar-looking distractor clauses, as real contracts have. - Grade with references written by someone who knows the contracts.

Then choose an effective context length for the product: the length where quality is still acceptable. Beyond it, retrieve the relevant sections or split the task, and compare that against putting everything in context.

Trade-offs

A realistic long-context eval is expensive to label and slow to run, but vendor needle tests do not predict multi-clause reasoning.

Common mistake

Taking the advertised context window as the length the model can reason over reliably.

Follow-up

How would you decide between retrieval and full context for a 90K-token contract?

Question 26MediumSafety evals

After a safety prompt update, your harmful-request eval shows refusals rising to 99%. A week later, support tickets show the assistant refusing to explain how to kill a hung Linux process. How should the safety eval have been built?

#

The safety eval contained only harmful prompts. Success was measured as refusal rate.

Show answer

A safety eval needs both sides: harmful prompts the model should refuse, and safe prompts that look similar and should be answered. With only harmful prompts, a model that refuses everything scores perfectly.

XSTest is built around this. It contains 250 safe prompts across ten types and 200 unsafe prompts as contrasts, to measure exaggerated safety, where "clearly safe prompts are refused if they use similar language to unsafe prompts or mention sensitive topics".

For your product: - Build contrast pairs from your domain: "kill a process", "shoot a photo", "exploit a market gap", questions about medication doses that are within policy, next to genuinely harmful requests. - Report two numbers: harmful-request compliance (should be near zero) and safe-request refusal (should also be low). Gate releases on both. - Count partial refusals and heavy hedging, not only outright refusals; validate the refusal classifier or judge against human labels. - Add real over-refusal cases from support tickets to the safe side of the set.

The safety prompt change should have shown the trade-off before release: refusal of harmful prompts up, refusal of safe prompts also up.

Trade-offs

Tighter safety settings lower harmful compliance but raise over-refusal; measuring both makes the trade-off a decision instead of a surprise.

Common mistake

Measuring safety by refusal rate on harmful prompts alone.

Follow-up

How would you set the acceptable over-refusal rate for a general assistant compared with a children's product?

Question 27MediumMetrics

You extract 25 fields from invoices into JSON. Exact match on the whole JSON is 41%, but users say extraction is mostly fine. What should you measure instead?

#

Fields include dates, totals, supplier name, tax lines and a list of line items. Some invoices lack some fields.

Show answer

Whole-document exact match is too strict to guide work: one wrong field out of 25 fails the invoice, and formatting differences count as errors. Measure at field level.

- Per field: precision, recall and accuracy, so you know which fields fail. Report them by document type or supplier too. - Normalise before comparing: dates to ISO format, amounts to numbers with currency, whitespace and case for names. Otherwise "03/04/2026" and "2026-04-03" count as wrong. - Separate the error types for missing values: field absent and the model returned null (correct), field present and the model missed it (recall error), field absent and the model invented a value (hallucination, often the worst). - For lists such as line items, match predicted items to reference items (by description and amount) before scoring, and report item-level precision and recall. - Weight fields by business cost: a wrong total matters more than a wrong reference number. - Keep schema validity as a separate hard check.

Also check whether the model's confidence or a validation rule (line items sum to total) predicts errors; that can route uncertain invoices to review.

Trade-offs

Field-level metrics guide fixes and match user experience, but need normalisation rules and matching logic that can hide errors if wrong.

Common mistake

Treating null handling as a formatting detail when invented values are the costliest errors.

Follow-up

How would you choose the threshold for sending an invoice to human review?

Question 28MediumBenchmarks

Leadership wants to choose the model for your customer-facing assistant by picking the top entry on a public human-preference leaderboard. What do you tell them?

#

The assistant answers billing and policy questions in three languages.

Show answer

Use the leaderboard to build a shortlist, not to decide.

What it measures: Chatbot Arena collects crowdsourced pairwise votes from its users on their own prompts (over 240K votes at the time of its paper) and reports good agreement with expert raters. That measures general chat preference of that user base. It does not measure accuracy on your policies, your languages, your latency or your cost.

Why ranks can mislead: "The Leaderboard Illusion" reports that undisclosed private testing benefits a few providers, who can test variants and keep only the best score; it identifies 27 private variants tested by Meta before the Llama 4 release. It estimates that Google and OpenAI received 19.2% and 20.4% of all arena data, and that even limited additional arena data gave relative gains of up to 112% on the arena distribution. Ranks can partly reflect optimising for the arena.

Recommend: - Shortlist three or four models from public results. - Run them on your own eval: real billing and policy questions in all three languages, graded against your policies, plus refusal and safety checks. - Compare cost per resolved conversation and latency at your traffic. - Pilot the winner on a slice of live traffic before switching.

Trade-offs

Your own eval takes a few weeks to build but answers the actual question; public ranks are instant but answer a different one.

Common mistake

Treating a general-preference rank as evidence of accuracy on your domain.

Follow-up

What would you include in the pilot to detect regressions the offline eval missed?

Question 29MediumRegression gates

Your full eval suite takes three hours and a large judge bill to run, so developers skip it and only run it before releases, when regressions are expensive to fix. How would you redesign it?

#

The suite has 4,000 items across 12 features, scored by a strong LLM judge plus some code checks.

Show answer

Split the suite into tiers so most feedback is cheap and fast.

- Every pull request: deterministic checks (schema, required fields, refusal on known-bad prompts) plus a stratified sample from each feature. Size the sample deliberately; the "Adding Error Bars to Evals" paper recommends power analysis to check whether an eval, or a random subsample, can test the hypothesis you care about. A 200-item sample that can only detect a 10-point drop should be labelled that way. - Changes to one feature: run that feature's full slice, not all 12. - Nightly and before release: the full suite, paired against production.

Cut cost per run: - Cache model outputs and judge verdicts keyed by model version, prompt version and input, so unchanged parts are not rerun. - Use a cheaper judge or code check first and send only uncertain or failing cases to the strong judge, after checking that this cascade agrees with the strong judge on a labelled set. - Drop items that never fail and never discriminate between versions, but keep known past failures.

Publish results in the pull request with failing examples, so the cheap tier is useful enough that people do not skip it.

Trade-offs

Smaller PR tiers catch big regressions early but miss small ones, which the nightly full run must still catch.

Common mistake

Running a random small subset on every PR without knowing what size of regression it can detect.

Follow-up

How would you choose which items go into the per-PR sample?

Question 30HardRegression gates

Your release gate blocks a change only if it is significantly worse than production. A cheaper model swap passed the gate on 150 items even though its score was 3 points lower. Why is this gate weak, and what would you use instead?

#

The goal of the swap is to save cost without losing quality. The team accepts at most a 2-point drop.

Show answer

With 150 items the confidence interval on the difference is wide, so even a real 3-point drop is rarely "significant". A gate that only blocks proven regressions lets almost anything through when the eval is small. Absence of evidence is being read as evidence of absence.

For a change whose goal is "no worse", use a non-inferiority test: - Agree on the margin first: the largest drop you accept, here 2 points. - Compare the candidate and production on the same items (paired). - Pass only if the whole confidence interval for the change is above minus 2 points. If the interval runs from -7 to +1, the swap fails, because a drop larger than 2 points is still plausible.

This puts the burden of proof on the change. It also forces the right sample size: to prove a drop is under 2 points you need enough items, which a power analysis tells you; 150 will not be enough.

Keep the other parts of a good gate: hard checks such as schema validity and safety as strict pass or fail, and per-slice checks with a minimum number of items, so a cheap model that is fine on average but fails one language is caught.

Trade-offs

Non-inferiority gates block more changes and need larger evals, but they stop small evals from passing real regressions.

Common mistake

Shipping a cost-saving change because the drop was "not statistically significant".

Follow-up

How would you pick the non-inferiority margin with the product team?