supraj.dev

Interview questions & answers

Production evaluation, observability & cost

Measure useful outcomes, trace failures and control the cost of successful work.

  • 59 questions
  • 1 easy
  • 33 medium
  • 25 hard
  • 4 related articles

59 questions

Question 1EasyProduction reasoning

The demo looks good. What would make you confident enough to release?

#

A support assistant answers a few hand-picked questions convincingly.

Show answer

Create representative, held-out cases including missing evidence, ambiguous requests and unsafe actions. Define task-specific correctness and failure criteria. Compare with a baseline, inspect failures and calibrate automated checks against human review. Repeat variable runs and keep regression cases from real incidents.

Trade-offs

Broader coverage costs time, while a small curated suite can miss production behavior.

Common mistake

Declaring quality from a few polished demos.

Follow-up

Which failure would block release even if the average score improved?

Question 2MediumProduction reasoning

The answer is bad. Which trace would help you locate the failure?

#

The assistant uses retrieval and multiple tools, but the only log is the final response.

Show answer

Correlate model calls, retrieval and tool executions under one request trace. Capture timings, versions, source IDs, token usage and tool outcomes. Redact or omit sensitive content and apply retention controls. Compare successful and failed traces to identify the first point where evidence or intent was lost.

Trade-offs

Detailed traces improve debugging but increase storage and privacy exposure.

Common mistake

Logging raw prompts and credentials by default.

Follow-up

Which metadata could diagnose the problem without retaining customer text?

A useful sequence to explain
  1. Request trace
  2. Retrieval evidence
  3. Model decision
  4. Tool outcome
  5. Final response
Question 3MediumProduction reasoning

A cheaper model raises retries. How would you compare the real cost?

#

Per-call spend falls, but users repeat requests and the workflow calls more tools.

Show answer

Measure completion quality alongside tokens, retries, tool costs and latency for comparable traffic. Compare cost per successful task against the baseline. Test bounded routing to a cheaper model for suitable cases and explicit fallback for failures. Include operational work and external tool fees in the comparison.

Trade-offs

Routing can lower spend but adds complexity and another decision to evaluate.

Common mistake

Choosing only by input-token price.

Follow-up

What counts as success when the assistant safely abstains?

Question 4HardProduction reasoning

Your automated judge likes a change that reviewers dislike. What next?

#

A model-based quality score improves, but engineers find more unsupported claims.

Show answer

Inspect disagreements on a labeled set. Separate correctness, grounding and style; check order and verbosity bias with paired comparisons. Calibrate thresholds against expert review and use deterministic checks for verifiable constraints. Keep the judge version and rubric fixed during a comparison, then reassess after changes.

Trade-offs

Automated judging scales review but can share blind spots with the system being evaluated.

Common mistake

Treating the judge score as ground truth.

Follow-up

How would you test whether longer answers get an unfair advantage?

Question 5HardProduction reasoning

A model upgrade passes offline tests but harms one user segment. How do you respond?

#

Overall success stays flat while a particular language and document type show worse answers.

Show answer

Compare quality, latency and cost by relevant segments, not just overall averages. Use a limited rollout with versioned prompts, models and indexes. Inspect affected traces and roll back when agreed thresholds fail. Add representative failures to the held-out suite before retrying the release.

Trade-offs

Small rollouts reduce exposure but rare failures may need longer observation or targeted tests.

Common mistake

Using a global average to dismiss a concentrated regression.

Follow-up

How would you avoid tuning directly on every held-out example?

Question 6HardQuality & safety

A deployed LLM gives incorrect or unfair answers. What is your investigation and release plan?

#
Show answer

Start by defining the failure precisely: unsupported facts, harmful stereotyping, uneven error rates or authorization mistakes need different mitigations. Preserve privacy-safe examples, segment the incidents by language/user task and compare them against a representative labelled evaluation set. Trace whether the issue starts in source data, retrieval, prompt design, model behavior or a postprocessor.

Choose the smallest tested intervention: fix indexing or instructions when evidence is missing, add deterministic checks for verifiable constraints, improve datasets or adapters only when the measured issue warrants retraining. Gate release on critical-case regressions and monitor live feedback with rollback. Do not claim all bias can be removed by a single fine-tune.

Trade-offs

Stronger refusals may reduce one harm but increase false rejections; tune to the actual risk and affected population.

Common mistake

Immediately fine-tuning when the real defect is missing source data or an unsafe tool boundary.

Follow-up

Which metrics would reveal an accuracy regression concentrated in one language while the average remains stable?

Question 7MediumClassification evaluation

A classifier has high micro F1 but low macro F1. What does that tell you?

#
Show answer

Micro F1 pools true positives, false positives and false negatives across labels before calculating the score. Macro F1 calculates each label’s F1 and averages them equally. The gap can indicate that frequent classes perform well while rare classes struggle.

Inspect per-class precision, recall, support and the confusion matrix. For a single-label multiclass task with all classes included, micro F1 equals accuracy. Choose release criteria around error consequences and minority-class needs; no single aggregate score explains every failure.

Question 8HardModel routing

A model router uses cheaper models but the total bill rises. What would you measure?

#
Show answer

Measure the full session, including router calls, extra input/output tokens, fallbacks, retries and tool work. Compare successful task cost and end-to-end latency with the previous baseline, using the same quality requirements. Check whether switching models reduces prefix-cache reuse or causes repeated context setup.

Test routing decisions on representative easy and difficult cases. Model affinity within a session may preserve useful reuse, but can keep a task on a poor model. Make escalation explicit and verify it with quality evaluations before assuming lower token prices mean lower total cost.

Question 9MediumCost controls

How would you control agent spend without relying on a monthly dashboard?

#
Show answer

Attribute every run to a tenant, project or user using a non-sensitive identifier. Track model tokens, cache usage and tool costs across the whole run. Then enforce limits in the gateway or tool executor: maximum steps, output budget, request deadline and per-owner spend allowance.

Account for concurrent requests and delayed usage reports; reserve capacity or use conservative limits before dispatch. Define a clear stop, downgrade or approval path when a limit is reached. Tracing gives visibility, while authorization and budget enforcement must happen before more work is allowed.

Question 10MediumEngineering judgment

How would you explain an agent failure you owned without inventing a perfect project story?

#
Show answer

Describe the real task, constraints, observable failure and its effect. Show how you used a trace or reproduction to separate a bad model decision from a tool, state or permission problem. Explain the change, the alternative you rejected and the trade-off you accepted.

Finish with evidence from a regression case and representative evaluation, plus a remaining limitation. Use anonymized technical detail without customer secrets. If you have only a prototype, say so; a clear account of measured failure and recovery is more useful than claiming production experience you do not have.

Question 11MediumTechnical discovery

A customer asks for an AI assistant without clear requirements. What do you clarify first?

#
Show answer

Identify the user, the decision or task to improve, the current workflow and a measurable success criterion. Ask which data is available, who may access it, which actions require review, and the latency, cost and operational constraints. Separate confirmed requirements from assumptions.

Propose a narrow first workflow with a baseline, representative evaluation cases, an owner and an explicit acceptance test. Explain what evidence would stop or change the proposal. Do not commit to a multi-agent architecture before showing why simpler automation or retrieval cannot meet the need.

Question 12MediumMLOps

What must an ML release record beyond the application’s Git commit?

#
Show answer

Record the model artifact, training configuration, data snapshot or lineage, preprocessing and feature definitions, dependencies, evaluation results and serving configuration. A source commit alone cannot reproduce a model trained from changing data.

Validate incoming data and the trained model, compare relevant segments with a baseline, and test serving compatibility before promotion. Keep the previous release’s complete artifact set for rollback. CI/CD and GitOps can deliver these components, but neither removes data validation, model evaluation or monitoring for changing real-world inputs.

Question 13MediumCost

Token spend tripled in a week while traffic stayed flat. How do you find out why?

#

Finance asks why the model bill jumped. Request counts look normal.

Show answer

Break spend down by feature, model and step using your telemetry, for example token counts recorded on each model call with OpenTelemetry's GenAI conventions. Compare input and output tokens per request before and after.

Common causes: a prompt or retrieved context that grew, an agent loop or retry storm, a cache that stopped hitting, or a routing change sending traffic to a pricier model. Fix the cause, then add budgets and alerts per feature so the next jump shows up in a day.

Trade-offs

Detailed per-call telemetry costs storage; sample payloads but always keep token counts.

Common mistake

Looking only at the monthly invoice, which cannot tell you which feature changed.

Follow-up

Which alert would have caught this on day one?

Question 14MediumRelease

How do you ship a prompt change safely?

#

A small wording change to the system prompt broke JSON output for 8% of requests last month.

Show answer

Version prompts in the repository, and run the change against a fixed evaluation set before merging, including format checks and the edge cases that broke before. Compare against the current version, not against an ideal score.

Roll out gradually: a small share of traffic or a shadow run, with monitoring on error rates, format failures and user feedback, and a quick rollback to the previous version. Record the prompt version on every request so incidents can be traced to a change.

Trade-offs

Evaluation gates slow prompt iteration, so keep a fast smoke set for every change and a full set for releases.

Common mistake

Editing prompts in a dashboard with no version history.

Follow-up

What would your minimum evaluation set include for this prompt?

Question 15HardLifecycle

Your provider is retiring the model your product uses. How do you migrate?

#

You have 90 days. Several prompts were tuned for the old model's quirks.

Show answer

Run your evaluation sets on candidate models and compare quality, latency and cost per task. Expect some prompts to need changes, since newer models can follow instructions differently.

Keep model choice in configuration behind a small internal interface so switching is a config change. Roll out per feature behind a flag, watch the same metrics as for a prompt change, and keep the old model as fallback until it is retired. Afterwards, record the model version in telemetry and keep evaluations ready for the next migration.

Trade-offs

Abstracting models eases switching but can hide provider-specific features you rely on.

Common mistake

Swapping the model name the week before retirement and testing in production.

Follow-up

How would you detect regressions that your evaluation set does not cover?

Question 16HardRate limits

A nightly job sends 40,000 Claude requests as fast as it can. It gets 429 errors in the first minute, although the console shows the organisation far below its input and output token limits. What is going on, and how do you fix the job?

#

The job starts at 02:00 from a cold start; daytime traffic on that model is light. Retries are immediate.

Show answer

Two things can cause 429s below the headline limit.

First, the limits are enforced over short intervals. Anthropic's rate limit docs say a rate of 60 requests per minute "might be enforced as 1 request per second", so a burst inside one second can fail even though the minute total is fine. The API uses a token bucket that refills continuously rather than resetting each minute.

Second, and more likely here, there is a ramp limit. It caps how fast an organisation's usage of a model can grow: roughly your recent peak per-minute usage plus a step, rising as successful usage grows and falling back after quiet periods. A cold job that jumps from almost nothing to full speed hits it. The error type is still rate_limit_error, but error.details.error_code is ramp_limit_exceeded, so log that field to tell the cases apart.

Fixes: - Honour retry-after, and replace immediate retries with backoff plus jitter. - Ramp up: start at a low rate, hold where requests succeed, and raise by at most one step per minute, as the docs advise. Spread requests evenly instead of releasing them all at once. - Better still, move the job to the Message Batches API. It is not subject to ramp limits, has its own rate limits, and is billed at 50 percent of standard prices. A nightly job rarely needs synchronous answers. - Give the job its own workspace with its own limits, so a burst cannot starve daytime traffic.

Trade-offs

A slow ramp makes the job take longer to reach full speed; the Batches API removes the ramp problem but gives up per-request latency and can take up to 24 hours.

Common mistake

Treating every 429 as "we need a higher tier" and asking for a limit increase, when the real cause is burst shape.

Follow-up

How would you build a client-side limiter that reads the rate limit headers and adapts its sending rate?

Question 17MediumReliability

On the first of the month your Claude-backed feature works again after an outage. The previous evening every request had failed with 429, and your retries kept hammering the API for hours. Why did retrying never help, and how should the client classify errors?

#

The SDK retries automatically, and your service wraps it in its own retry loop.

Show answer

Not every 429 is a temporary rate limit. When an organisation reaches its usage tier's monthly spend cap, the Claude API returns 429 rate_limit_error with error.details.error_code set to enforced_spend_limit_reached and no retry-after header. Usage stays paused until 00:00 UTC on the first day of the next month unless the limit is raised. The docs say retrying, "including the SDK's automatic retries", fails until access resumes. A spend limit you set yourself shows up differently: HTTP 400 invalid_request_error with a message saying you reached your specified usage limits.

So the retries only produced load, log noise and paging. Classify errors by meaning, not by status code alone: - Retry with backoff: 429 with retry-after (honour it), 500 api_error, 529 overloaded_error, connection errors. - Do not retry: 400, 401, 403, 404, 413, and 429 with enforced_spend_limit_reached. Raise an alert to a person instead, because it needs a budget decision. - Branch on the SDK's typed exceptions and on error.details.error_code, not on message strings, which can change.

Then fix the process: alert when spend reaches, say, 80 percent of the cap, show the request ID in logs (every response has a request-id header), and add a circuit breaker so a non-retryable failure trips once instead of being retried by every worker.

Trade-offs

Aggressive retries recover faster from real transient errors; strict classification avoids storms but needs maintenance as providers add new error codes.

Common mistake

Treating status 429 as always retryable and stacking an application retry loop on top of the SDK retries.

Follow-up

Which of these failures should page someone at night, and which should only open a ticket?

Question 18HardReliability

During a brief provider overload, your traffic to the model API rose to many times its normal rate and stayed high after the provider recovered. You have retries in the SDK, in your service and in the API gateway. Explain the failure and redesign the retry policy.

#

Each layer retries 3 times. Retries fire without jitter. Unsuccessful requests still count against your rate limit.

Show answer

Retries multiply across layers. If the gateway, the service and the SDK each make up to 4 attempts (1 plus 3 retries), one user request can become 4 x 4 x 4 = 64 calls. Without jitter they arrive in synchronised waves. And rate limits make it worse: OpenAI's rate limit guide notes that unsuccessful requests count toward your per-minute limit, "so continuously resending a request won't work". You keep yourself rate limited after the provider has recovered.

Redesign: - Retry in one layer only, normally the one closest to the model call. The Anthropic SDK already retries connection errors, rate limits and 5xx errors twice by default with exponential backoff and honours retry-after; either rely on it and set the outer layers to zero retries, or set max_retries=0 in the SDK and own the policy yourself. - Add jitter to every backoff, and honour Retry-After when present. OpenAI recommends falling back to exponential backoff with jitter when the header is missing, and limiting both attempts and total retry time. - Use a retry budget: allow retries only while they are a small share of total traffic, for example 10 percent. Past that, fail fast. - Add a circuit breaker per provider and model that opens on sustained 529 or 5xx errors and probes with a few requests before closing. - Set an end-to-end deadline. A retry that cannot finish before the user's deadline is wasted. - Queue background work instead of retrying it inline, so user traffic gets the capacity first.

Trade-offs

Fewer retries mean some recoverable requests fail; more retries hide transient faults but can turn a short overload into a long self-inflicted one.

Common mistake

Adding retries at every layer "to be safe", without counting the total attempts a single request can generate.

Follow-up

How would you test the retry policy before the next provider incident?

Question 19HardRate limits

You are building an internal LLM gateway in front of both OpenAI and Anthropic. It enforces a per-team "tokens per minute" budget by counting max_tokens plus prompt tokens before each call. Why will this budget disagree with what each provider actually enforces?

#

Teams set max_tokens generously. Several teams use long cached system prompts.

Show answer

Because the two providers do not count the same things.

OpenAI: the rate limit guide says your limit "is calculated as the maximum of max_tokens and the estimated number of tokens", and recommends setting max_tokens close to the expected response size. A generous max_tokens really does consume rate limit there.

Anthropic: limits are split into input tokens per minute (ITPM) and output tokens per minute (OTPM). OTPM counts only tokens actually generated; the docs say max_tokens "does not factor into OTPM rate limit calculations". For ITPM, the default limits count only uncached input: input_tokens and cache_creation_input_tokens count, cache_read_input_tokens do not. With an 80 percent cache hit rate, the docs' example processes 10M total input tokens per minute against a 2M ITPM limit.

So a single pre-call estimate will throttle Anthropic teams far too early (counting cached reads and unused max_tokens) and may under-count OpenAI teams.

Better design: - Model each provider's limiter separately: input and output buckets for Anthropic, request and token buckets for OpenAI. - Reserve tokens before the call with a provider-specific estimate, then reconcile with the usage numbers in the response. - Read the providers' headers (x-ratelimit-remaining-tokens for OpenAI; anthropic-ratelimit-input-tokens-remaining and the output equivalent for Anthropic) to keep the gateway's view close to reality. - Express team budgets in money as well as tokens, since cached and uncached tokens cost different amounts.

Trade-offs

Provider-specific accounting is more code to maintain; a single generic counter is simpler but either wastes capacity or lets teams exceed real limits.

Common mistake

Assuming "tokens per minute" means the same thing at every provider.

Follow-up

How would you share one organisation-wide provider limit fairly across teams with very different traffic shapes?

Question 20HardModel routing

Leadership wants automatic failover from your primary model provider to a second provider when the primary is overloaded. The first real failover made latency and cost jump and broke a structured-output feature. What did the design miss?

#

The backup provider normally gets almost no traffic. Prompts were tuned only on the primary model. The primary returns 529 overloaded errors during the incident.

Show answer

Failover moves traffic to a system that was never tested under that load or with those prompts.

What goes wrong: - Quality and format. Prompts, tool schemas and structured-output settings tuned for one model behave differently on another. The fallback model must be in your evaluation suite, with its own prompt variants, and the structured-output checks must run against it. - Cold caches. Prompt caches are scoped to an account; Anthropic's docs say caches are isolated between organisations and, on the Claude API, by workspace. The backup starts with no cached prefixes, so every request pays full input cost and longer time to first token. - Rate limits. A backup sized for a trickle cannot take full traffic. On Anthropic, a sudden jump can also hit ramp limits, which cap how fast usage can grow even below your rate limits. - Data terms. The backup may have different retention and data processing terms, so check that the data you route there is allowed to go there.

Design instead: - Trigger failover with a circuit breaker on sustained overloaded or 5xx errors, not on one failure. - Fail over gradually and for the features that need it most; let batch work wait. - Keep the backup warm with a small, steady share of real traffic, which keeps its evals honest, its limits grown and its caches partly warm. - Agree capacity with the backup provider in advance. - Record which provider and model answered each request, so quality complaints can be traced.

Trade-offs

A warm multi-provider setup costs engineering time and some steady spend; a cold backup is cheap until the day it has to carry production.

Common mistake

Testing failover with a few requests and assuming the backup behaves the same at full load and with untuned prompts.

Follow-up

Which features would you deliberately not fail over, and what would they show users instead?

Question 21MediumReliability

Your document pipeline reports a 99.9% success rate, yet users find empty summaries. The logs show HTTP 200 for every call. What is the pipeline probably missing?

#

The pipeline uses a recent Claude model, checks only the HTTP status, and stores whatever text comes back.

Show answer

It is probably treating refusals as successes. On recent Claude models, safety classifiers can decline a request, and the response is "a normal response, not an error": HTTP 200, stop_reason: "refusal", empty content, and a stop_details object with a category and a human-readable explanation. A pipeline that checks only the status code stores an empty summary.

Fix: - Treat stop_reason as part of success. Only end_turn (or an expected stop_sequence or tool_use) means a complete answer. max_tokens means truncated output; refusal means no answer. - Count refusals by stop_details.category in your metrics so you can see whether one document type triggers them. - Retry refused requests on a fallback model if that fits your policy. Anthropic offers a server-side fallback (in beta) and SDK middleware for client-side fallback; the docs say you can "usually still get an answer" by sending the same request to another Claude model. - Understand billing: refusals before any output are billed only in certain categories, and a mid-stream refusal bills the input and the output already streamed. - Show users a clear message instead of a blank result, and display the explanation rather than parsing it, because its text is not stable.

More generally, your success metric for an LLM call should be "produced a usable, complete result", which needs checks beyond HTTP.

Trade-offs

Automatic fallback recovers more requests but adds latency and cost and can send content to a model you did not choose for it; failing visibly is simpler and more transparent.

Common mistake

Equating HTTP 200 with a successful generation.

Follow-up

What other "successful" responses would you count as failures in your dashboard?

Question 22MediumReliability

An extraction service sometimes writes invalid JSON to the database, but only for long contracts. Retries with the same settings fail the same way. How do you find and fix the cause?

#

max_tokens was set to 1,024 a year ago when documents were shorter. Output is parsed after the call returns.

Show answer

Look at the stop reason. If it is max_tokens, the model was cut off mid-answer, so the JSON is incomplete; retrying with the same limit reproduces it. Anthropic's stop reason guide says to treat such output as incomplete, and that a response cut off inside a tool_use block is unusable and should be retried with a higher max_tokens.

Fixes: - Check the stop reason on every response and never parse or store output that stopped on max_tokens. Route it to a retry with a larger limit, or split the input. - Size max_tokens from the real output distribution, with margin, and revisit it as inputs grow. - Use structured outputs or strict tool use where available, so a complete response is schema-valid; truncation still needs its own check. - Monitor the share of responses per stop reason. OpenTelemetry's GenAI conventions include gen_ai.response.finish_reasons on inference spans, so a rise in max_tokens stops shows up on a dashboard before users report broken records. - Validate before writing, and keep the raw response for failed parses so you can debug.

A bigger max_tokens has little downside for Anthropic rate limits, because output limits count only generated tokens, but long outputs still cost money and time, so cap them per use case.

Trade-offs

A high max_tokens avoids truncation but lets runaway outputs cost more; splitting long inputs keeps outputs bounded but needs merge logic.

Common mistake

Adding a JSON "repair" step that silently closes brackets on truncated output and stores incomplete data.

Follow-up

How would you handle an extraction whose correct output is genuinely longer than the model's maximum output?

Question 23MediumReliability

Long report generations fail after several minutes with a dropped connection and no error from the provider, but only from the corporate network. Short requests are fine. What is happening and what do you change?

#

Calls are non-streaming with a large max_tokens. Traffic goes through a corporate proxy.

Show answer

A non-streaming call keeps an idle HTTP connection open until the whole answer is ready. Some networks and proxies drop idle connections after a while, so the request fails or times out without ever receiving the provider's response. Anthropic's error docs warn about exactly this and advise against a large max_tokens without streaming or batches; they recommend the streaming Messages API or the Message Batches API for long requests, "especially those over 10 minutes". The SDK validates that non-streaming requests are not expected to exceed a 10-minute timeout and sets TCP keep-alive.

Changes: - Stream long generations. Tokens and periodic events keep the connection active, and users see progress. If you do not need incremental output, the SDK can consume the stream and return the final message. - Handle errors inside the stream. With server-sent events, an error can arrive after the 200 response, for example an overloaded_error event during high usage. Your code must check for error events, not only the HTTP status. - Use batches for work that does not need an immediate answer; you poll for results instead of holding a connection. - If you build your own client, set TCP keep-alive, and align proxy idle timeouts with the longest expected request. - Decide what a partial stream means for your feature: discard, retry, or show it as incomplete.

Trade-offs

Streaming adds client complexity and mid-stream error handling; batches remove connection risk but add latency of minutes to hours.

Common mistake

Raising client timeouts again and again while the proxy keeps dropping the idle connection.

Follow-up

How would you resume or retry a stream that failed halfway through a long generation without paying for the whole thing twice?

Question 24MediumCost

A weekly job classifies two million support tickets with synchronous API calls and is now one of your largest AI costs. How would you move it to batch processing, and what changes in your code?

#

Results are needed by Monday morning. All prompts share a long instruction block.

Show answer

Both major providers discount batch work by half. Anthropic's Message Batches API charges "50 percent of the standard API prices"; OpenAI's Batch API gives a "50% cost discount compared to synchronous APIs". Both use a 24-hour window, and Anthropic says most batches finish within an hour.

What changes: - Split into batches within the limits: on Anthropic a batch holds up to 100,000 requests or 256 MB; on OpenAI up to 50,000 requests and a 200 MB input file. - Match results by custom_id. Both providers say results may not come back in input order, so never rely on position. - Handle per-item outcomes: on Anthropic each result is succeeded, errored, canceled or expired, and you are not billed for errored, canceled or expired items. Resubmit failures in a later batch. - Plan for the deadline: start early enough that a batch hitting its 24-hour expiry still leaves time to resubmit before Monday. - Download results promptly: Anthropic keeps them for 29 days; OpenAI output files expire after 30 days by default. - Caching still helps: Anthropic supports prompt caching in batches on a best-effort basis and suggests the 1-hour cache duration for shared context.

If you need answers in minutes but can tolerate slower responses, OpenAI's flex processing (service_tier: "flex") is priced at Batch API rates on supported models, with occasional 429 "resource unavailable" errors that are not charged.

Trade-offs

Batches halve the price and remove most rate limit pressure, but you lose per-request latency and must build polling, retries and result matching.

Common mistake

Writing results back by array index and silently attaching labels to the wrong tickets.

Follow-up

Which of your real-time features could become asynchronous to benefit from batch pricing?

Question 25HardPrivacy

Your company has a zero data retention (ZDR) agreement with its model provider because you process customer health records. An engineer moves a nightly summarisation job to the Batch API to save 50%. Why should the privacy review block this, and what control would have caught it?

#

Legal believes "we have ZDR, so nothing is stored". The gateway lets any service call any endpoint.

Show answer

ZDR applies to eligible features, not to everything an organisation calls. Anthropic's data retention page lists the Message Batches API and the Files API as not ZDR-eligible: batches keep results for 29 days. Its FAQ is blunt: nothing blocks the request, and using such a feature "is a choice to step outside your ZDR arrangement for that specific data". OpenAI is similar: /v1/batches, /v1/files and vector stores keep data until deleted and are not ZDR-eligible, while ZDR forces store to false on Responses and Chat Completions.

So the cost change quietly moved health records into stored data, outside the arrangement legal relied on. Also note that even under ZDR, Anthropic may keep flagged inputs and outputs for up to 2 years, and some models require 30-day retention, so "nothing is stored" was never quite true.

Controls: - Classify data per workload, and have the gateway allow only endpoints and models approved for that class. Health data gets an allowlist of ZDR-eligible endpoints; batches and files are denied. - Make the provider's feature eligibility table part of design review for any new endpoint or feature. - Alert on first use of a new endpoint per workload. - Document the residual retention (flagged content, legal holds, model-specific rules) so legal's position matches reality.

Trade-offs

Endpoint allowlists slow cost optimisations like batching; open access is faster for engineers and lets one change break a contractual commitment.

Common mistake

Treating ZDR as an account-wide switch rather than a property of specific endpoints and features.

Follow-up

How would you get batch-level savings for this workload without leaving the ZDR arrangement?

Question 26MediumObservability

Engineers want full prompts and model outputs on every trace to debug a customer support assistant. Security worries about personal data in the tracing backend. How do you get debuggability without turning traces into a data leak?

#

Traces are kept 30 days in a SaaS backend that most engineers can read. Conversations include names, addresses and order details.

Show answer

Split traces into metadata that everyone can see and content that few can see.

OpenTelemetry's GenAI conventions already draw this line. Capturing instructions, prompts, completions, retrieval queries and prompt variables "is gated by an explicit user opt-in", and instrumentations must offer a setting to control it. Attributes such as gen_ai.input.messages carry a warning that they are likely to contain personal data. The conventions also define _ref attributes, such as gen_ai.input.messages_ref, that point to content stored elsewhere; they advise against putting credentials like signed URL parameters in that URI and say access should be secured by storage-level access control.

A workable design: - Default traces carry metadata only: model, prompt name and version, token counts, latency, finish reasons, tool names, retrieval document IDs, error types. - Content goes to a separate store with tight access, shorter retention and audit logs, linked by a reference attribute. - Redact or tokenise personal data before storage where the debugging value allows. - Capture content for a sample of traffic, plus all traces that failed, were flagged or got negative feedback, instead of everything. - Provide break-glass access for incidents, logged and time-limited. - Match retention to your privacy policy and deletion requests: if a user asks for erasure, the content store must be included.

Trade-offs

Less content in traces makes some bugs harder to reproduce; full content everywhere makes debugging easy and multiplies where personal data lives.

Common mistake

Turning on full content capture in the default exporter because it was one environment variable.

Follow-up

How would you reproduce a reported bad answer when the trace has only metadata?

Question 27HardCost

Your observability dashboard estimates Claude spend as gen_ai.usage.input_tokens times the input price plus output tokens times the output price. It shows about three times the real invoice. Prompt caching is heavily used. Where is the error?

#

Instrumentation follows the OpenTelemetry GenAI semantic conventions. Most requests reuse a long cached system prompt.

Show answer

The two numbers use different definitions. In the OpenTelemetry GenAI conventions, gen_ai.usage.input_tokens "SHOULD include cached tokens". The Anthropic page of the conventions spells out the mapping: Anthropic's own input_tokens excludes cached tokens, so the attribute is computed as input_tokens + cache_read_input_tokens + cache_write_input_tokens.

Multiplying that total by the base input price charges every cached read at full price. But cache reads are billed at a fraction of the base price (0.1 times on most Claude models, less on some newer ones), and cache writes at a premium (1.25 times for the 5-minute cache, 2 times for the 1-hour cache). With heavy caching, most input tokens are cheap reads, so the dashboard overstates cost.

Fix the cost model: - Price each token class separately, using gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_write.input_tokens, which the conventions also define, and derive uncached input as the remainder. - Know which cache TTL you used, because write prices differ. - Note that gen_ai.usage.output_tokens should include reasoning tokens, which are billed as output. - Keep provider-specific price tables versioned by model, and reconcile the dashboard against the invoice every month.

The broader lesson: token counts in telemetry are not costs. Treat "tokens" as several billing classes.

Trade-offs

A precise per-class cost model needs maintenance as prices and token classes change; a single blended rate is simple but wrong whenever caching or batching changes.

Common mistake

Concluding that caching "does not save money" because the dashboard total did not fall.

Follow-up

How would you show the savings from caching per feature in the same dashboard?

Question 28HardCost

To "maximise caching", a team switched every Claude request to the 1-hour cache duration. Their bill went up. Explain why, and when the 1-hour cache is worth it.

#

A chat feature gets several requests per second with the same system prompt. A separate review tool is used every 15 to 30 minutes by each reviewer on a long document.

Show answer

Cache writes cost more than normal input, and the 1-hour write costs more than the 5-minute write. Anthropic's prompt caching docs give 1.25 times the base input price for a 5-minute write, 2 times for a 1-hour write, and 0.1 times for a read on most models (lower on some newer ones). The default lifetime is 5 minutes, and the cache "is refreshed for no additional cost each time the cached content is used".

For the chat feature, requests arrive every few seconds, so a 5-minute entry never expires: it is refreshed on every hit. Paying 2 times instead of 1.25 times for each write buys nothing. Writes still happen whenever the prefix changes or new conversation turns are cached, so the extra cost shows up on the bill.

The review tool is different. Gaps of 15 to 30 minutes exceed 5 minutes, so with the default TTL each visit rewrites the document at 1.25 times and never reads it. With the 1-hour cache, the first visit pays 2 times and later visits within the hour pay 0.1 times. Two visits cost 2.1 times the base price instead of 2.5 times, and each further visit widens the gap.

Rule of thumb: use the 1-hour cache only when the same prefix is reused with gaps longer than 5 minutes and shorter than an hour, such as batch jobs, slow human loops, or agents waiting on long tools. Measure it: track cache write and read tokens per feature, and check that prompts exceed the model's minimum cacheable length, because shorter prompts are simply not cached and "no error is returned".

Trade-offs

The 1-hour cache saves money for sparse reuse and wastes it for dense reuse; choosing per feature adds configuration but matches price to traffic.

Common mistake

Treating a longer TTL as strictly better without comparing write prices with real reuse gaps.

Follow-up

How would you detect from usage data which features would benefit from the 1-hour cache?

Question 29MediumCost

You moved a workload from Claude Opus 4.6 to Opus 4.7. Both have the same price per million tokens, the prompts are unchanged, yet the bill rose by about a third and some long documents no longer fit. What happened?

#

Finance compared price lists, saw identical per-token prices, and approved the migration as cost-neutral.

Show answer

The tokenizer changed. Anthropic's models overview says 1M tokens is roughly 555k words on the current tokenizer, which was introduced with Claude Opus 4.7, while models before it fit about 750k words in 1M tokens. The same text therefore becomes roughly 750 / 555, about 1.35 times, as many tokens. At an identical price per token, the same work costs about a third more, which matches what you saw.

The same effect explains the documents that no longer fit: a context window measured in tokens holds less text, and output limits allow fewer words.

What to do: - Compare models on cost per task, measured by running your real prompts and reading the usage numbers, never on the price list alone. Token counts differ across model families and across tokenizer versions inside one family. - Re-check every token-based setting after a migration: max_tokens, truncation and chunking thresholds, context budgets, and per-team token quotas. - Update dashboards and alerts that convert tokens to words or pages. - Weigh the higher token count against quality and task success. A model that needs fewer retries or tool calls can still cost less per completed task.

Trade-offs

A newer model may give better results per task at a higher token count; staying on the old model keeps costs predictable until its retirement date.

Common mistake

Calling a migration cost-neutral because the price per million tokens did not change.

Follow-up

How would you build a pre-migration cost estimate from a sample of last week's traffic?

Question 30HardLifecycle

You upgrade from an older Claude model to a current one, and also bump the API version header. Requests that worked for a year now fail with 400 errors. Which kinds of request features break, and how do you make future upgrades safe?

#

The code sets temperature, sometimes prefills the assistant turn to force JSON, uses thinking with a fixed budget, and forces a specific tool with tool_choice.

Show answer

Several features this code relies on are rejected by current Claude models: - Sampling parameters. Anthropic's deprecation table says temperature, top_p and top_k were removed in API version 2026-10-21 and return a 400 error; earlier API versions already return 400 when they are set to non-default values on Claude 4.7 and later models. - Prefill. Claude 4.6 and later models do not support prefilling the assistant message; the error says the conversation must end with a user message. Use structured outputs or output_config.format instead. - Thinking budgets. Claude 4.7 and later reject thinking: {"type": "enabled"}; use adaptive thinking with output_config.effort. - Forced tool use. Some current models, such as Claude Opus 5.5 and Sonnet 5.5, reject tool_choice of type any or tool; use auto with strict tool use or structured outputs.

Making upgrades safe: - Keep model ID and API version in configuration, and change one at a time. - Run contract tests in CI that send each request shape you use to each model you support, so 400s appear before production. - Query the Models API, which reports a capabilities object per model, instead of hard-coding assumptions. - Re-run quality evals after removing parameters: dropping temperature or prefill changes behaviour, so prompts may need work. - Roll out per feature behind a flag, with the old model as fallback until its retirement date.

Trade-offs

Pinning old models and API versions avoids breakage now but builds up migration debt and ends at the retirement date; upgrading often costs regular small migrations.

Common mistake

Upgrading the model, the SDK and the API version together in one change, so you cannot tell which one broke the requests.

Follow-up

How would you replace temperature-based control of output variety when the parameter is no longer accepted?

Question 31MediumLifecycle

Users report that answers "changed overnight". No one deployed. Product suspects the provider updated the model behind the same name. How do you investigate, and what do you record so this is quick next time?

#

The service calls a hosted model by name through a gateway. Prompts live in a database that product managers can edit.

Show answer

Start by checking the provider's versioning rules. Anthropic's models overview says every Claude model ID is a pinned snapshot, including the dateless IDs used from the 4.6 generation on; for older models the alias resolves to a dated ID. Other providers may have aliases that move. Know which kind you are calling.

Then look for the change on your side, where it usually is: - Prompt versions: an edit in the prompt database counts as a deploy, even if no code shipped. - Retrieval: re-indexing, a new embedding model, or changed source documents. - Routing: a router or fallback that sent traffic to another model, for example during a provider incident. - Configuration: changed effort, max_tokens or tool definitions.

Compare traces from before and after on the same kinds of questions.

Record on every request: requested model and the model the response reports (gen_ai.request.model and gen_ai.response.model in the OpenTelemetry GenAI conventions), prompt name and version (gen_ai.prompt.name, gen_ai.prompt.version), retrieval index version, router decision, and request ID.

Also watch lifecycle: Anthropic gives at least 60 days' notice before retiring publicly released models, says deprecated models are likely to be less reliable than active ones, and offers a usage export broken down by API key and model to find stragglers. Partner clouds set their own dates.

Trade-offs

Recording versions on every call adds telemetry volume; without it, every quality complaint becomes a guessing game.

Common mistake

Blaming the provider before checking prompt edits and retrieval changes on your own side.

Follow-up

How would you put prompt edits made by product managers through the same review and rollout as code?

Question 32HardEvals

You are building an LLM judge that marks support answers as acceptable or not, to gate releases. 92% of answers are acceptable. The judge agrees with human labels 91% of the time. Is it ready? How should you validate it?

#

Labels come from two senior support agents. The rubric was written before anyone graded real outputs.

Show answer

Not yet. A judge that always answers "acceptable" would agree 92% of the time, so 91% raw agreement may mean it catches almost no bad answers. With imbalanced labels, measure the judge separately on each class: of the answers humans rejected, how many did the judge reject (its recall on failures), and of those it rejected, how many were truly bad. Report a chance-corrected agreement score such as Cohen's kappa alongside. For a release gate, the judge's ability to catch failures matters most.

Validation steps: - Build a labelled set with enough failures, oversampling known bad answers so each class has real support. - Check that the two human labellers agree with each other first; if they do not, fix the rubric, not the judge. - Expect the rubric to change. The EvalGen paper calls this criteria drift: "users need criteria to grade outputs, but grading outputs helps users define criteria". Grade real outputs, update the rubric, then re-validate. - Hold out part of the labelled set so you do not tune the judge prompt to the same examples you score it on. - Test known biases. The MT-Bench paper examines position, verbosity and self-enhancement biases; it found strong judges reached over 80% agreement with human preferences, but that was on its tasks, not yours. - Pin the judge model and prompt, and re-validate whenever either changes.

Trade-offs

Careful validation costs expert labelling time; an unvalidated judge is cheap and can wave through regressions while showing a high agreement number.

Common mistake

Reporting raw agreement on an imbalanced label set as proof that the judge works.

Follow-up

How would you monitor the judge after launch to notice that it has drifted from human judgment?

Question 33MediumEvals

Your team runs a bake-off between models from two vendors, and the judge is a model from vendor A. Vendor A wins clearly. A colleague says the result cannot be trusted. Is she right, and how would you redesign the comparison?

#

Pairwise comparisons always show vendor A's answer first. Vendor A's answers are also longer on average.

Show answer

She has a point: the setup mixes the real difference with three known judge biases. The MT-Bench paper names position bias, verbosity bias and self-enhancement bias, which is a judge favouring answers from its own model or family. Here, vendor A's answers are always in the first position, are longer, and are judged by a vendor A model. Any of these can produce the win.

Redesign: - Swap positions: judge every pair in both orders, and count a win only when the verdict is consistent; inconsistent pairs count as ties. - Use a judge from a third model family, or several judges from different families, and report where they disagree. - Control length: tell the judge not to reward length, add length-matched examples, and report win rates by answer-length bucket. - Prefer reference-based or checkable criteria where possible: correctness against a known answer, schema validity, policy compliance. These do not depend on taste. - Audit a random sample with human raters blind to the vendor, and report the judge's agreement with them. - Report the result with confidence intervals, compare on the same questions, and include cost and latency per task, since the bake-off is a business decision.

Trade-offs

Multiple judges and position swaps double or triple evaluation cost; a single judge is cheap and can quietly decide a vendor contract.

Common mistake

Using one vendor's model as the only judge in a comparison that includes that vendor.

Follow-up

If the two judges disagree on the winner, how do you decide?

Question 34HardEvals

Your contract-QA eval has 1,000 questions: 25 questions on each of 40 contracts. The new prompt scores 2 points higher, and the reported 95% interval is about plus or minus 2.5 points, computed as if the questions were independent. A reviewer says the interval is too narrow. Why, and how do you analyse it properly?

#

Questions about the same contract tend to succeed or fail together. The team planned to lower temperature to reduce noise.

Show answer

The questions are not independent: 25 questions on one contract share the same document, so if the model misreads that contract, many answers fail together. The effective sample size is closer to the number of contracts than to the number of questions, so the naive interval is too narrow.

The Adding Error Bars to Evals paper gives the method: - Use clustered standard errors when questions come in related groups, such as several questions about one passage. The paper reports that clustered standard errors "can be over 3X larger than naive standard errors"; in its DROP example the ratio was about 3. - Compare the two prompts with paired differences on each question rather than comparing two summary scores. Pairing removes the variation that comes from question difficulty and usually tightens the interval. - Reduce variance by sampling several answers per question and averaging, rather than lowering temperature. The paper advises against lowering the sampling temperature to reduce variance, because it changes the thing you are measuring. - Run a power analysis before the next experiment to see whether the eval can detect a 2-point change at all. If not, add more contracts, not just more questions per contract.

With clustering, a 2-point gain may not be distinguishable from noise. Report it that way.

Trade-offs

Clustered, paired analysis is more honest and sometimes disappointing; naive intervals look precise and lead to shipping changes that do nothing.

Common mistake

Adding more questions about the same documents and expecting the interval to shrink as if they were new, independent cases.

Follow-up

How would you change the eval design so the next comparison has enough power?

Question 35MediumEvals

You want continuous quality scores on live traffic for a RAG assistant, without adding latency for users or judging every request. How do you design online evaluation, and how do its results feed back into offline tests?

#

About one million requests a day. Feedback buttons are rarely used. Traces already follow the OpenTelemetry GenAI conventions.

Show answer

Run evaluation asynchronously, off the request path. The user gets the answer; a worker later picks up a sample of completed traces and scores them.

Design: - Sampling: a small random sample for unbiased trend lines, plus targeted samples: requests with negative feedback, retries, refusals, long outputs, new features and low-traffic segments that a random sample would miss. - Evaluators: cheap deterministic checks on everything (format, citations present, policy keywords), and an LLM judge on the sample for groundedness against the retrieved documents and for answer relevance. The judge must be validated against human labels before you trust its numbers. - Recording: OpenTelemetry's GenAI conventions define a gen_ai.evaluation.result event with gen_ai.evaluation.name, a score value or label, and an explanation. It should be parented to the span being evaluated, or carry gen_ai.response.id when the span ID is not available. That lets you slice scores by model, prompt version and retrieval index on your existing dashboards. - Implicit signals: rephrased questions, copy actions, escalations to a human agent and abandonment are often more plentiful than thumbs ratings. - Privacy: the evaluator reads content, so it needs the same access controls and retention rules as your content store.

Feedback loop: low-scoring traces go to a review queue; confirmed failures, after anonymisation, become new offline test cases, so the CI suite keeps up with what production actually sees.

Trade-offs

A larger sample gives faster, finer signals and costs more judge calls; a tiny sample is cheap and misses rare but important segments.

Common mistake

Treating online judge scores as ground truth without checking them against human review.

Follow-up

How would you alert on a quality drop without paging for normal day-to-day noise?

Question 36HardExperimentation

An A/B test of a new answer style shows an 8% lift in positive feedback. The split was meant to be 50/50, but the treatment has 4% fewer users in the analysis. Should you ship?

#

Assignment happens per request. Feedback and exposure events are logged by the client after the answer finishes streaming. The treatment's answers are longer.

Show answer

Not yet. A split that differs from the design is a sample ratio mismatch. Fabijan and colleagues describe it as "a symptom for a variety of data quality issues", and the result cannot be trusted until you find the cause. A chi-squared test on the counts tells you whether the gap is beyond chance; with real traffic volumes, 4% almost always is.

Likely causes here: - Logging after streaming finishes. The treatment's longer answers take longer, so more users leave or hit timeouts before the exposure event fires. Those users vanish from the treatment, and they are the least satisfied ones, so the remaining treatment users look happier. - Errors that differ by arm: if long answers fail more often and failed requests are not logged, the same bias appears. - Per-request assignment: one user sees both variants in a conversation, and some sessions are counted twice or dropped.

Fixes: - Log exposure at assignment time on the server, before any model call. - Randomise by user or conversation, not by request, so experiences stay consistent. - Add guardrail metrics per arm: latency, errors, refusals, cost per conversation, abandonment. - Rerun after fixing logging, and decide in advance how long to run and which metric decides, to avoid stopping as soon as the numbers look good.

Trade-offs

Fixing the experiment delays the launch; shipping on a biased result can lock in a change that made the product worse for exactly the users who left.

Common mistake

Ignoring a small imbalance in group sizes because the headline lift looks large.

Follow-up

Which metric would you choose as the primary decision metric for an LLM answer style test, and why not thumbs-up rate?

Question 37HardGuardrails

Adding an input safety check and an output policy check doubled p95 latency for your chat product, and the output check conflicts with streaming. How do you keep the guardrails and win back the latency?

#

Both checks are LLM calls run in sequence: input check, then the main model, then the output check, then the answer is sent.

Show answer

Start with the input check: run it in parallel with the main call. OpenAI's guardrails cookbook does this with two concurrent tasks. If the guardrail flags the request, the main task is cancelled and a canned reply is returned; otherwise the main response goes out. You pay for some wasted main-model tokens on flagged requests, but most requests are not flagged, so the input check stops adding its full latency.

The output check and streaming need a policy per risk level: - Low-risk content: stream immediately and run the output check asynchronously, with the ability to retract or flag a message after the fact. - Medium risk: buffer and check in chunks, such as sentences or paragraphs, so users still see progress. - High risk, such as actions, regulated advice or anything sent externally: do not stream; check the full output first.

Other ways to cut cost and latency: - Use a small, fast classifier first and call a larger model only for uncertain cases. - Replace LLM checks with deterministic ones where possible: schemas, allowlists, regexes for secrets. - Tune thresholds on a labelled set with a confusion matrix, as the cookbook recommends, and set them per use case: a jailbreak check can be strict, a topical filter can be loose. - Measure the guardrails themselves: false positives annoy users and false negatives cause harm. The cookbook also notes that LLM-based guardrails share the base model's weaknesses.

Trade-offs

Parallel checks waste some tokens on blocked requests; streaming before the output check is faster but can show users text you later withdraw.

Common mistake

Putting every guardrail in series on every request regardless of risk.

Follow-up

How would you evaluate whether a guardrail change made the product safer without making it less helpful?

Question 38HardCaching

To cut costs, a team proposes a semantic cache in front of the assistant: if a new question is similar enough to an old one, return the old answer. What can go wrong, and where would you allow it?

#

The assistant answers both general product questions and account-specific questions such as order status.

Show answer

A semantic cache embeds each query, searches for similar past queries, and returns a stored answer when a similarity evaluator says the match is close enough. That is how tools such as GPTCache work, and their README is honest that semantic caches produce false positives (serving the wrong cached answer) as well as false negatives.

Risks: - Wrong meaning: "cancel my order" and "don't cancel my order" can be close in embedding space. - Data leakage: an account-specific answer cached for one user is served to another user who asks a similar question. - Staleness: prices, policies or order status change, but the cached answer does not. - Hidden coupling: answers produced by an old prompt or model keep being served after you upgrade.

Where to allow it: - Only for general content with no personal data, such as product FAQs. - Key the cache by tenant, locale, prompt version and model, and set a TTL tied to how often the source content changes; purge on content updates. - Tune the threshold on labelled pairs of similar and different questions, and track the false hit rate in production by sampling cache hits for review. - Never cache answers to personalised, action-taking or safety-sensitive requests.

Try cheaper, safer savings first: exact-match caching of identical requests, provider prompt caching of long prefixes, a smaller model for simple questions, and shorter prompts.

Trade-offs

A semantic cache can cut cost and latency for repetitive traffic, but every hit skips the model, the retrieval and the freshness of your data.

Common mistake

Setting one global similarity threshold and caching every answer, including account-specific ones.

Follow-up

How would you measure the false hit rate of a semantic cache before turning it on for users?

Question 39MediumRate limits

A data team's backfill shares an API organisation with your customer-facing assistant. Whenever the backfill runs, the assistant starts getting 429 errors. How do you isolate the workloads?

#

Both use the same API key. The organisation is already on a high usage tier.

Show answer

Both workloads draw from one organisation-level limit, so the backfill can use all of it. Separate them.

On the Claude API, workspaces let you set lower spend and rate limits per workspace. In the docs' example, an organisation with 40,000 input tokens per minute limits one workspace to 30,000, and the rest stays available to others. Limits are set per limiter type (requests, input tokens, output tokens), unset workspace limits match the organisation's, and "organization-wide limits always apply, even if workspace limits add up to more".

Steps: - Give each workload its own workspace and API key, and cap the backfill's workspace well below the organisation limit. - Move the backfill to the Message Batches API: it has separate rate limits, is not subject to ramp limits and costs half as much. - In the client, read the rate limit headers. The anthropic-ratelimit-tokens-* headers show the most restrictive limit in effect, and anthropic-workspace-id shows which workspace the request counted against, which makes 429s easy to attribute. - Monitor per workspace on the usage page or through the Rate Limits API. - For guaranteed headroom on the assistant, consider a priority capacity arrangement with the provider.

The principle is the same at any provider: background work must not share a quota with user-facing traffic without a cap.

Trade-offs

Static caps leave some capacity idle when the assistant is quiet; a shared pool uses capacity fully but lets the noisiest job win.

Common mistake

Asking for a higher tier while both workloads still share one key and one limit.

Follow-up

How would you let the backfill use spare capacity at night without risking the assistant during the day?

Question 40HardObservability

Thumbs-down feedback on a RAG support assistant doubled over three days. No deploy, no errors, latency normal. Traces follow the OpenTelemetry GenAI conventions but carry no prompt text. How do you find the cause from traces instead of reading random conversations?

#

About 200,000 requests a day. Each request has a retrieval span and one inference span. Feedback is attached to the trace. A content store with restricted access holds sampled conversations.

Show answer

Treat it as a comparison, not a reading exercise.

1. Build cohorts. Bad: traces with negative feedback (or a low judge score) in the last three days. Good: traces for the same feature with positive feedback in the same days, plus a baseline from the week before. Comparing within the same days removes time effects such as weekday traffic. 2. Compare low-cardinality attributes between cohorts: gen_ai.request.model against gen_ai.response.model (a router or fallback answered), gen_ai.prompt.name and gen_ai.prompt.version, gen_ai.response.finish_reasons, plus your own attributes such as feature, locale, client version and index version. A value whose share is far higher in the bad cohort than in the good one is your lead. 3. Walk the spans in order and find the first one whose distribution differs. On the retrieval span the conventions define gen_ai.retrieval.top_k, and the opt-in gen_ai.retrieval.documents holds document IDs and scores: compare how many documents came back, the top score, and which collections they came from. On the inference span compare the gen_ai.usage.input_tokens distribution: with steady traffic, a drop in input tokens means the model is getting less context; a rise means the prompt grew. 4. Only then read content, and only for the suspect slice, from the restricted store. 5. Reproduce offline: replay those inputs against the old and new index or prompt.

A typical result: 40% of bad traces retrieved zero or one document, against 5% of good ones, all from one collection re-indexed three days ago. That is an ingestion fault, not a model problem.

Finally, turn the explaining attribute into a standing chart or alert, so the next drift shows up before users complain. The GenAI conventions are still marked Development, so pin the version you build dashboards on.

Trade-offs

Metadata-only traces are cheap and safe but you can only slice by what you recorded; after each incident, add the attribute you wished you had (index version, document count).

Common mistake

Reading twenty bad conversations without a comparison group, so patterns that are common in good traces too look like causes.

Follow-up

Which three attributes would you add to every trace so the next diagnosis takes minutes?

Question 41HardObservability

After moving to tail sampling in the OpenTelemetry Collector, traces of long agent runs appear split into pieces, and some agent runs that failed are missing. Short chat traces look fine. What is going on, and how do you tune it?

#

Agent runs last 1 to 4 minutes. About 200 new traces a second. The tail sampling processor runs with default settings behind a load-balancing exporter tier. Some spans carry long prompts.

Show answer

The defaults are tuned for short request traces.

- Timing: in the default trace-complete strategy, the processor decides after decision_wait, which defaults to 30s. A 3-minute agent run is judged on its first 30 seconds of spans; a tool failure at minute 2 arrives after the decision. - Late spans: the README describes what happens next. While the decision is still in the buffer, late spans inherit it. Once it has left the buffer, and with no decision cache (sampled_cache_size and non_sampled_cache_size both default to 0), the late spans are treated as a new trace: buffered for decision_wait and decided again. One run can be kept in some parts and dropped in others, which is the fragmentation you see. - Eviction: num_traces (default 50,000) is a circular buffer. When it is full the oldest trace is evicted, possibly before a decision; that shows up in otelcol_processor_tail_sampling_sampling_trace_dropped_too_early.

Sizing arithmetic: traces held at once is about arrival rate x wait. At 200 traces/s and 30 s that is 6,000. Waiting 300 s for agent runs needs 200 x 300 = 60,000, more than the default 50,000. If traces carrying prompts average 200 KB, 60,000 traces are about 12 GB across the sampling tier, which is why prompt text does not belong on spans.

Tuning: - Route agent traces to their own pipeline with a longer decision_wait, and set num_traces above rate x wait with headroom. - Set decision_wait_after_root_received: it can decide earlier once the root span arrives. Spans are exported when they end, and the root span of an agent run normally ends last, so finished runs are decided soon after they end while running ones keep waiting. - Enable both decision caches, much larger than num_traces, so late spans follow the first decision. - Set maximum_trace_size_bytes so one huge trace cannot exhaust memory, and put memory_limiter first in the pipeline, as its README advises. - Watch otelcol_processor_tail_sampling_sampling_trace_removal_age: if its p1 approaches decision_wait, traces are at risk of being dropped.

Trade-offs

Longer waits keep whole agent runs but cost memory and delay when traces become visible; deciding on root arrival only helps if the root span reliably ends last.

Common mistake

Raising decision_wait for all traffic without raising num_traces, which makes early eviction worse.

Follow-up

How would you guarantee that every run with a failed tool call is kept, even when the failure happens after minute 3?

Collector: tail sampling for agent traces (sketch) · yaml
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 12000
    spike_limit_mib: 2400
  tail_sampling:
    decision_wait: 300s
    decision_wait_after_root_received: 5s
    num_traces: 100000
    expected_new_traces_per_sec: 200
    maximum_trace_size_bytes: 5000000
    decision_cache:
      sampled_cache_size: 1000000
      non_sampled_cache_size: 1000000
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow-runs
        type: latency
        latency: {threshold_ms: 120000}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

service:
  pipelines:
    traces/agents:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling]
      exporters: [otlp]
Question 42MediumObservability

Since content capture was turned on for a sample of traffic, some traces never reach the backend and the Collector logs rejected gRPC messages. Other spans with long prompts arrive with their text cut off. Explain both, and what you change.

#

SDKs export OTLP over gRPC to a Collector whose receiver uses default settings. Some prompts include 100-page documents. To be safe, someone set OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT=8192.

Show answer

Rejected exports: the SDK's batch span processor sends up to OTEL_BSP_MAX_EXPORT_BATCH_SIZE spans per export (default 512). The Collector's gRPC server limit is max_recv_msg_size_mib; when it is not set, the gRPC default applies, which is 4 MiB in grpc-go. If a document of about 300 KB is recorded as input on each span, 15 such spans in one batch are 15 x 300,000 = 4,500,000 bytes, more than 4 MiB (4,194,304 bytes). The whole export request is rejected, including the small spans in it, so whole traces go missing.

Cut-off text: the spec's default attribute value length limit is "no limit". Once you set one, SDKs must truncate string values to it. gen_ai.input.messages must be recorded as a structured AnyValue, and the limit then applies to each string inside it separately, so every long message part is cut at 8,192 characters. An older instrumentation that stores messages as one JSON string gets the string cut mid-JSON, which no longer parses. The spec only says an SDK MAY log that it truncated something, so do not count on a warning.

What to change: - Keep long content out of spans. Store it in a restricted content store and record a gen_ai.input.messages_ref URI instead; traces stay small and access control stays in one place. - If you must send larger messages, raise max_recv_msg_size_mib on the receiver deliberately and lower the export batch size, then load-test the Collector's memory. - If you keep a length limit, choose it knowingly and record your own attribute saying the content was truncated, so nobody debugs from partial text without knowing. - Alert on exporter failures and Collector refusals; silent loss of traces is the worst outcome.

Trade-offs

Bigger message limits keep full content but raise Collector memory and blast radius; references to a content store keep spans small at the cost of a second lookup.

Common mistake

Assuming a missing trace means it was sampled out, when the export batch it was in was rejected.

Follow-up

How would you alert on spans being dropped between the SDK and the backend?

Question 43HardCost

To bill customers for AI usage, a team added a customer_id attribute to the OpenTelemetry token counters. Weeks later, finance finds per-customer numbers wrong for newer customers, while the grand total still matches. Prometheus memory has also grown. What happened?

#

8,000 business customers, 6 models, 3 operation names, 2 token types. The OpenTelemetry Python SDK runs with default metric settings and exports to Prometheus.

Show answer

Two separate limits were hit.

1. The SDK cardinality limit. The OpenTelemetry metrics SDK spec sets a default limit of 2,000 attribute sets per metric when no view or reader sets another, and the Python SDK documents the same default. Measurements with further attribute sets are aggregated into one extra point with the attribute otel.metric.overflow=true. With cumulative temporality, attribute sets seen before the overflow keep reporting; new ones land in the overflow point. Every measurement is still counted exactly once, which is why the grand total matches while new customers look empty.

Arithmetic: possible combinations are 8,000 x 6 x 3 x 2 = 288,000 per counter. Even if each customer used one model, one operation and one token type, 8,000 is four times the limit. The customers seen first by each process get real series; later ones are lumped together.

2. Prometheus series. Prometheus warns that every unique label combination is a new time series and says not to use labels for unbounded values such as user IDs. Its instrumentation guide suggests keeping most metrics' cardinality below 10 and looking for alternatives above 100.

Fix: - Billing is not a metric. Write one usage record per request (request ID, customer, model, token counts by type, price version) to a log or event stream that lands in the warehouse, where you can deduplicate and reconcile with invoices. - Keep metrics low-cardinality: provider, model, operation, token type, feature, tier. - Enforce it with a View whose attribute_keys allow-list drops anything else, and alert if a series with the overflow attribute ever appears. - Consider Prometheus sample_limit on the scrape as a guard rail, knowing that exceeding it fails the entire scrape.

Trade-offs

Per-request usage records are exact and auditable but need a data pipeline; metrics are cheap and fast but only for low-cardinality questions.

Common mistake

Raising the cardinality limit until the numbers look right, which moves the cost into Prometheus memory instead of fixing the design.

Follow-up

How would you show a customer their near-real-time usage without per-customer metric labels?

OpenTelemetry Python: allow-list token counter attributes with a View · python
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.view import View

token_view = View(
    instrument_name="gen_ai.client.inference.usage.*",  # wildcards are supported
    attribute_keys={
        "gen_ai.operation.name",
        "gen_ai.provider.name",
        "gen_ai.request.model",
        "gen_ai.token.modality",
        "app.feature",  # your own low-cardinality attribute
    },
)

provider = MeterProvider(metric_readers=[reader], views=[token_view])  # reader: your exporter's reader
Question 44MediumObservability

Your dashboard says the p99 latency of model calls is exactly 10 seconds, every day, while users report waits of 40 seconds or more. The p50 jumps between round-looking numbers. Calls are recorded in milliseconds with an OpenTelemetry histogram created with default settings. Explain the numbers and fix the instrument.

#

Most calls take 2 to 8 seconds; reasoning-heavy calls take 30 to 70 seconds. Prometheus computes quantiles with histogram_quantile.

Show answer

The SDK's default explicit bucket boundaries are 0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000. In milliseconds the largest finite bucket ends at 10,000 ms. Every call slower than 10 s lands in the +Inf bucket.

Prometheus histogram_quantile has a documented special case: if the quantile falls in the highest bucket, it returns the upper bound of the second-highest bucket. Once more than 1% of calls exceed 10 s, p99 reports exactly 10,000 ms, no matter whether the slow calls took 11 s or 70 s.

The p50 is coarse for a similar reason: values between 2,500 and 5,000 ms share one bucket, and Prometheus interpolates linearly inside it, so the median moves in large steps.

Fix: - Record seconds, as the GenAI conventions do, and pass the boundaries they recommend for gen_ai.client.inference.duration: 0.01 doubling up to 81.92 s. Use the API's advisory parameter so the instrument carries them. - Calls longer than 81.92 s still land in +Inf, so for long reasoning or agent work extend the list, or use an exponential (base-2) histogram. - Watch views: the spec says that if a matching View selects explicit bucket histogram aggregation, with or without boundaries, the advisory boundaries are ignored. A view that just says 'histogram' silently brings back the defaults. - Also: the conventions' per-operation token histograms (gen_ai.client.inference.operation.input_tokens, buckets 1, 4, 16 and so on) are for percentiles and outliers only; the spec says not to use them for totals or cost. Use the usage counters for that.

Trade-offs

More buckets give better percentiles but cost one series per bucket per label set; exponential histograms adapt to the data but need backend support.

Common mistake

Trusting a p99 that never moves; a flat percentile at a bucket boundary usually means the data is outside the buckets.

Follow-up

How would you prove to a skeptical colleague that the 10 s p99 is an artefact, using only the histogram's own series?

OpenTelemetry Python: duration histogram with the GenAI-recommended buckets · python
from opentelemetry import metrics

meter = metrics.get_meter("support-assistant")

inference_duration = meter.create_histogram(
    "gen_ai.client.inference.duration",
    unit="s",
    explicit_bucket_boundaries_advisory=[0.01, 0.02, 0.04, 0.08, 0.16, 0.32, 0.64, 1.28, 2.56, 5.12, 10.24, 20.48, 40.96, 81.92],
)

# after each call:
# inference_duration.record(elapsed_seconds, attributes={"gen_ai.operation.name": "chat", ...})
Question 45HardAlerting

Product wants a page when answer quality drops. Your quality score comes from an LLM judge that grades a 1% sample of traffic, with a pass rate near 90%. Someone proposes: page when the hourly pass rate falls below 85%. Will that work?

#

20,000 requests an hour in the day, 1,500 at night. The judge has been validated against human labels. Judge results arrive about 10 minutes after each request.

Show answer

It will page at night for nothing and still be slow in the day.

Arithmetic (normal approximation to the binomial): - Day: 1% of 20,000 = 200 judged answers an hour. Standard error at a 90% pass rate = sqrt(0.9 x 0.1 / 200) = 0.021, about 2.1 points. 85% is 2.4 standard errors below 90%, so a healthy hour falls below it by chance roughly 1% of the time. - Night: 1% of 1,500 = 15 judged answers. Standard error = sqrt(0.09 / 15) = 0.077, about 7.7 points. 85% is only 0.65 standard errors away, so roughly a quarter of healthy night hours would page. - Detection time is at least the hour window plus the judge delay.

The SRE workbook makes the same point for low-traffic services: at 10 requests an hour, one failure is a 1,000x burn rate on a 99.9% SLO. Its options are synthetic traffic, combining services, or changing what counts as a failure.

A better design: - Page on signals measured on every request: schema-valid output, empty or refused responses, truncation (max_tokens), missing citations, error rates. These are complete, fast and cheap. - Use judge scores for tickets, with a minimum sample size in the rule. For example, over 6 daytime hours there are 1,200 judged answers, a standard error of about 0.87 points, so a 3-point drop is about 3.5 standard errors. - Increase the judge sampling rate at night or for small segments so each window has enough samples. - Alert separately if judge results stop arriving, or the quality panel will look calm while nothing is measured.

Trade-offs

Longer windows and minimum counts cut false pages but delay detection; more judging shortens windows at the cost of judge spend.

Common mistake

Treating a sampled judge score like a request-level error rate and putting a static hourly threshold on it.

Follow-up

How would you choose the judge sampling rate so a 3-point drop is detected within two hours, day and night?

Prometheus rule: judge-based ticket with a minimum sample size (sketch, your own metric names) · yaml
groups:
  - name: llm-quality
    rules:
      - alert: JudgePassRateLow
        expr: |
          (
            sum(increase(judge_results_total{result="pass"}[6h]))
            / sum(increase(judge_results_total[6h]))
          ) < 0.87
          and sum(increase(judge_results_total[6h])) > 400
        labels:
          severity: ticket
Question 46MediumReliability

An agent feature returns HTTP 200 on 99.95% of calls, but support keeps finding conversations where the assistant said nothing after a tool call. What is a likely cause on the Claude API, and which detectors would you put on every model call so silent failures reach a dashboard?

#

The agent loop puts each tool result in a user message and adds a short text note ('Here is the result') after it. Final answers are streamed.

Show answer

Likely cause: Anthropic's stop-reason docs describe empty responses with stop_reason: "end_turn": exactly 2 to 3 tokens and no content, typically after tool results. Listed causes include adding text blocks immediately after tool results, and sending Claude's completed response back without adding anything. Fix the message structure first; if empties persist, add a continuation prompt in a new user message rather than retrying the same messages.

Detectors on every call, as low-cardinality counters and span attributes: - Stop reason distribution: record gen_ai.response.finish_reasons on spans and count calls by stop reason. Alert on changes in the share of max_tokens (truncated), refusal (no answer) and empty end_turn. - Empty output: end_turn with no content blocks, or output tokens at or below a small floor. - Incomplete streams: in streaming, stop_reason is null in message_start and only arrives in message_delta. A stream that ends without a message_delta has no stop reason and must count as incomplete, not as success. - Shape checks: valid JSON or schema, required citations present, tool calls well formed. - Self-hosted models: vLLM's vllm:request_success_total is labelled by finish reason, so a rising share of length finishes shows truncation.

Then define a good request for your SLO as "usable, complete output", not "2xx". Each class gets its own counter so you can see which one moved.

Trade-offs

Per-call checks add a little code and a few series per model; they are far cheaper than finding failures through support tickets.

Common mistake

Retrying an empty end_turn response by sending the same messages again, which the docs say will not work because the model has already decided it is done.

Follow-up

Which of these failure classes would you page on, and which would you only ticket?

Question 47HardIncidents

Ten minutes after a deploy that added one tool to your agent, the Claude API starts returning 429 errors for every feature. Request volume did not change. Walk through the diagnosis and the immediate response.

#

The organisation's input-token limit (ITPM) for the model is 2,000,000. Normally about 80% of input tokens are cache reads, out of about 9,000,000 input tokens a minute. Every request sends the same tool list and long system prompt. The new tool's description embeds a per-request trace ID for debugging.

Show answer

Cause: the prompt cache broke, and cached tokens were what kept you under the limit.

- Anthropic's caching docs: the cache prefix runs tools, then system, then messages, and modifying tool definitions invalidates the entire cache. A per-request trace ID in a tool description makes every prefix unique, so no request can ever hit the cache. - Rate limits are cache-aware: only uncached input (input_tokens plus cache_creation_input_tokens) counts toward ITPM; cache reads do not.

Arithmetic: before, 20% of 9,000,000 = 1,800,000 counted tokens a minute, under the 2,000,000 limit. After, all 9,000,000 count: 4.5 times the limit. Every feature sharing that limit gets 429s. Cache writes also count toward ramp limits, and each request now pays a cache write, which costs 1.25 times the base input price for the 5-minute TTL, instead of a cache read at 0.1 times (on most models).

Diagnosis signals: - Cache read tokens per minute fall to near zero while cache write tokens jump (from response usage, or the cache_read and cache_write token counters if you emit them). - 429 rate_limit_error responses, with anthropic-ratelimit-input-tokens-remaining near zero. Check error.details.error_code: ramp_limit_exceeded points to a ramp limit rather than the ITPM limit. - The timing matches the deploy.

Immediate response: roll back the deploy; it is the fastest fix. Pause batch work to free capacity, honour retry-after, and make sure retries back off instead of adding load.

Afterwards: keep tool definitions and the system prompt byte-stable and in a deterministic order, put per-request data at the end of the messages, alert on the cache read share per feature, and run a cache-hit check in the deploy pipeline.

Trade-offs

Stable prefixes need discipline in how prompts and tools are built; the payoff is lower cost, lower latency and much more headroom under the same limit.

Common mistake

Asking for a higher rate limit during the incident, when the deploy multiplied the counted tokens by 4.5.

Follow-up

What check would you add to CI so a change that breaks caching fails before it ships?

Question 48MediumIncidents

Write the on-call runbook for 'primary LLM provider degraded'. What does it contain, and which decisions must be made in advance rather than at 3 a.m.?

#

Five features use one provider. Two have an evaluated fallback model. One is a nightly batch job. Last quarter, incidents ran 40 minutes before anyone decided anything.

Show answer

Sections of the runbook:

1. Detect and confirm. Alerts on error ratio by provider and error type, plus a synthetic probe. Is it the provider or us? Provider trouble shows across features and models: 529 overloaded_error, 500 api_error, timeout_error. A 429 is usually not an outage: spend-cap and ramp-limit 429s have their own runbooks. 2. Declare. Write the criteria down. The SRE book suggests declaring early, when a second team is needed, when customers can see it, or when it is unsolved after an hour of focused analysis. Name the roles: incident commander, operations lead (the only person changing the system), communications lead. 3. Pre-decided actions per feature, as a table agreed with product owners: - Features with an evaluated fallback: switch, with the trigger defined (for example a circuit breaker after sustained overloaded errors). - Features without one: degrade (cached or static answer, 'try again later', queue and notify) or switch off with a feature flag. - Batch: pause, so it does not compete for limits during recovery. 4. Retry discipline. One retry layer with backoff and jitter; the circuit breaker stops hammering a failing provider. 5. Communications. Status page and support templates ready to paste. 6. Recovery. Return traffic gradually. On the Claude API, a ramp limit caps how fast usage can grow, and it comes back down after a period of lower usage, so a sudden full return after an outage can hit ramp_limit_exceeded. Raise traffic in steps. 7. Close. Postmortem criteria and where the timeline lives.

Decisions made in advance: which features may fail over and to what, who can approve switching off a feature, the declaration threshold, and the customer message. With those written down, the 40 minutes become 5.

Trade-offs

Pre-agreed degradations make response fast but must be re-tested as features change; untested fallbacks are worse than a clear 'temporarily unavailable'.

Common mistake

Treating every 429 or 5xx as a provider outage and failing over, when some are your own spend cap or a bug in your request.

Follow-up

How would you test this runbook without waiting for a real outage?

Question 49MediumObservability

You need to know within five minutes if your LLM endpoint is broken at 4 a.m., when real traffic is near zero. Design synthetic probes: what they check, what they cost, and how you stop them distorting your SLO numbers.

#

The endpoint is your API in front of a hosted model. Prometheus and the blackbox exporter already probe your websites.

Show answer

What a probe does: POST a fixed request through the full public path (auth, gateway, routing, model) with a tiny output limit, and check the answer, not only the status code. The blackbox exporter's HTTP prober supports method, headers, body, valid_status_codes, and fail_if_body_json_not_matches_cel, which fails the probe if the JSON body does not satisfy a CEL expression. Add a slower, deeper probe for dependencies: a retrieval question whose answer must cite a known document.

Gotchas from the exporter docs: - The HTTP method defaults to GET and preferred_ip_protocol defaults to ip6; set both. - The probe timeout comes from the Prometheus scrape_timeout, slightly reduced, and is further limited by the module's timeout; the default scrape timeout is 10s. Leave room for a model call. - Keep the API key in a file (credentials_file) rather than in the config.

Cost arithmetic: one probe a minute from 3 regions is 3 x 1,440 = 4,320 calls a day. At about 65 tokens each that is roughly 281,000 tokens a day, negligible. A deep probe with 4,000 tokens of context every minute from 3 regions is 17,280,000 tokens a day, so run deep probes less often.

Keeping SLOs honest: the SRE workbook recommends synthetic traffic for low-traffic periods, but warns that when an issue affects real users and not the synthetic requests, the successful probes hide the real signal. So tag probe requests (a header that becomes a span and metric attribute), exclude them from the user SLO, and alert on probe_success separately. Probes cover only a few request types, so they complement real-traffic alerts.

Trade-offs

Frequent probes find breakage fast but cost tokens and rate-limit headroom; deep probes catch more failure types but are slower and pricier.

Common mistake

Counting probe successes in the user-facing SLO, so a night of real failures looks healthy.

Follow-up

How would you detect a quality regression, not just an outage, with synthetic probes?

blackbox exporter module: LLM endpoint probe (sketch) · yaml
modules:
  llm_chat_probe:
    prober: http
    timeout: 25s
    http:
      method: POST
      preferred_ip_protocol: ip4
      valid_status_codes: [200]
      headers:
        Content-Type: application/json
        X-Synthetic-Probe: "true"
      authorization:
        type: Bearer
        credentials_file: /etc/blackbox/llm-probe-token
      body: '{"prompt": "Reply with the single word PONG.", "max_tokens": 5}'
      fail_if_body_json_not_matches_cel: "body.text.contains('PONG')"
# scrape this module with scrape_timeout: 30s so the probe gets ~25s
Question 50HardRelease

You canary a new prompt on 5% of traffic for one hour. Its JSON-validity failure rate is 2.6% against 2.0% for the control. Is that a regression? How long must the canary run to detect a 1-point increase reliably?

#

40,000 requests an hour. Baseline failure rate 2%. Assignment is by user, so each user sees one version.

Show answer

Counts: the canary saw 5% x 40,000 = 2,000 requests (about 52 failures); the control saw 38,000 (about 760 failures).

Standard error of the difference = sqrt(0.026 x 0.974 / 2,000 + 0.02 x 0.98 / 38,000) = sqrt(0.00001266 + 0.00000052) = 0.0036. The gap is 0.006, so z = 1.65. That is below 1.96, so at the usual 5% two-sided level it is not yet evidence of a regression, though it is a reason to keep the canary running rather than promote it.

How many canary requests are needed to detect 2% to 3%, with 80% power at 5% significance? Because the control is large, treat its rate as known: n = (1.96 x sqrt(0.02 x 0.98) + 0.84 x sqrt(0.03 x 0.97))^2 / 0.01^2 = (0.274 + 0.143)^2 / 0.0001, about 1,745 canary requests, about 52 minutes at 2,000 an hour. Detecting a half-point increase needs roughly four times as many.

Caveats: - Assignment by user makes requests correlated, so the effective sample is smaller; use clustered standard errors. - Decide the duration or use a sequential test. Checking every 5 minutes and stopping at the first significant result inflates false alarms. - Compare canary and control at the same time. The SRE workbook warns that before-and-after comparisons confuse the change with time effects such as weekday and weekend use, and recommends metrics that are attributable to the change. - The canary limits the blast radius: the workbook's example is that a version failing 20% of requests on a 5% canary gives a 1% overall error rate. - Sampled quality scores need far more time: with 1% judge sampling the canary yields 20 judged answers an hour, so judge everything in the canary arm while it runs.

Trade-offs

Longer canaries detect smaller regressions but slow releases; a bigger canary detects faster but exposes more users.

Common mistake

Promoting or rolling back on a raw difference of rates without computing whether the sample can resolve it.

Follow-up

Which metrics would you make automatic rollback triggers, and which would need a human decision?

Question 51MediumEvals

The weekly answer-quality score fell from 86% to 80%. Every team says nothing changed, and each language's score is flat. Marketing launched in a new region that week. Explain the drop and redesign the dashboard.

#

Quality is judged on a random sample. Last week: 80% of traffic in English (90% quality), 20% in Hindi (70%). This week: 50% English, 50% Hindi, with the same per-language scores.

Show answer

It is a change in traffic mix, not a regression.

Last week: 0.8 x 90% + 0.2 x 70% = 72 + 14 = 86%. This week: 0.5 x 90% + 0.5 x 70% = 45 + 35 = 80%.

No segment got worse; the weaker segment got bigger. The reverse also happens: a real regression in one segment can be hidden when traffic shifts toward a stronger segment. This is why the SRE workbook calls before-and-after comparisons risky: time changes who is using the system.

Dashboard redesign: - Show quality per segment (language, feature, customer tier, input length band, new versus returning users), each with its sample size and an error bar. - Add a mix-adjusted score: reweight segment scores to a fixed reference mix, so the trend line moves only when quality moves. Show the raw score beside it, because the raw score is what users experience. - Track segment shares as their own drift metric, and annotate launches. - Alert per segment, with a minimum sample size, rather than only on the overall average.

The business reading also changes: Hindi quality at 70% now affects half of users, so it has become the most valuable thing to improve.

Trade-offs

Segmented dashboards are more accurate but harder to read and need enough samples per segment; one headline number is simple but can mislead in both directions.

Common mistake

Opening an incident and rolling back prompts because the overall average dropped, without checking the segment mix.

Follow-up

How would you decide the sampling rate per segment so small segments get usable error bars?

Question 52MediumIncidents

An LLM summariser wrote wrong dates in about 3% of summaries for nine days before a customer noticed. There was no outage and no page. Does this deserve a postmortem, and what should an AI-incident postmortem contain that a normal one might not?

#

Summaries are stored and shown to customers. A prompt edit and a retrieval change both shipped in the window.

Show answer

Yes. The SRE book's common postmortem triggers include user-visible degradation beyond a threshold, data loss, on-call intervention, a resolution time above a threshold, and a monitoring failure, which usually means the incident was discovered by hand. A customer finding it after nine days is exactly that. The book also says to define the criteria before incidents happen and to keep postmortems blameless.

What an AI postmortem needs in addition: - Impact by re-scoring: run a checker over every summary from the window to count wrong outputs, list affected customers, and decide whether stored outputs must be regenerated or corrected. Persisted outputs make AI incidents last longer than the fix. - Exact versions in the timeline: requested and responding model (gen_ai.request.model, gen_ai.response.model), prompt name and version, retrieval index version, judge version, router decisions, and parameters such as temperature and max tokens. - Separating the contributors: replay a sample of inputs with each change alone (prompt only, retrieval only) to see which one caused the errors, accepting that sampling makes replays noisy. - Detection analysis: why offline evals and online checks missed it. Did the eval set contain dates? Did the judge rubric check them? - Avoiding 'the model hallucinated' as a root cause. It describes the symptom; the causes are the conditions that made it likely and let it through. - Action items with owners: add the failing cases to the regression set, add a deterministic check (dates in the summary must appear in the source), add a monitor on that check, and shorten the time to detection.

Trade-offs

Re-scoring and replay cost time and model calls, but without them the impact and the cause are guesses.

Common mistake

Closing the incident after reverting the prompt, without measuring how many stored summaries were wrong.

Follow-up

Which deterministic check would have caught this on day one, and what would it cost to run on every summary?

Question 53MediumPrivacy

Security wants personal data scrubbed from GenAI telemetry centrally in the OpenTelemetry Collector instead of trusting every service. The plan is the redaction processor with a few regexes for emails and card numbers. What will it catch, what will it miss, and how should it be configured?

#

Some services capture gen_ai.input.messages for debugging. Traces go to a SaaS backend outside your region.

Show answer

What the processor does, per its README: it deletes span, log and metric datapoint attributes that are not on allowed_keys, then masks parts of values that match blocked_values regexes. The allow-list fails closed: an empty allowed_keys removes all attributes unless allow_all_keys: true. blocked_key_patterns masks the values of matching keys. Instead of a fixed mask you can hash; supported functions are md5, sha1, sha3, hmac-sha256 and hmac-sha512, and HMAC is recommended for low-entropy data. The summary setting adds diagnostic attributes, and at debug it lists the names of redacted keys, which can itself leak information. The README calls the processor one line of defence, not the only compliance measure.

What regexes catch: structured patterns such as card numbers, emails, phone numbers and key-shaped tokens.

What they miss: names, addresses, health details and anything else in free-text prompts. A prompt saying 'my neighbour John at 14 Elm Street' passes untouched.

Configuration that works: - Run GenAI traces in allow-list mode: allow the metadata keys (operation, provider, models, token counts, finish reasons, your feature attribute, _ref attributes) and do not allow content attributes, so captured prompts are dropped at the Collector. - Keep blocked_values as a backstop for allowed keys. - Hash identifiers you need to join on with an HMAC key loaded from the environment. - Use summary: info or silent in production. - Store content, when needed, in a restricted store in-region and reference it with _ref attributes. - Better still, do not capture content at the source by default: the conventions make it opt-in for this reason.

Trade-offs

Allow-lists are safe but drop new attributes until someone adds them; block-lists keep everything useful but leak whatever nobody thought of.

Common mistake

Believing regex redaction makes free-text prompts safe to send to a third-party backend.

Follow-up

How would you test the redaction config in CI so a new attribute cannot leak content?

Collector: redaction processor in allow-list mode (sketch) · yaml
processors:
  redaction/genai:
    allow_all_keys: false
    allowed_keys:
      - gen_ai.operation.name
      - gen_ai.provider.name
      - gen_ai.request.model
      - gen_ai.response.model
      - gen_ai.response.finish_reasons
      - gen_ai.usage.input_tokens
      - gen_ai.usage.output_tokens
      - gen_ai.input.messages_ref
      - gen_ai.output.messages_ref
      - app.feature
      - app.user_id
    blocked_values:
      - "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}"  # email
      - "4[0-9]{12}(?:[0-9]{3})?"                          # Visa-like card number
    blocked_key_patterns:
      - ".*user_id.*"
    hash_function: hmac-sha256
    hmac_key: "${env:REDACTION_HMAC_KEY}"
    summary: info
Question 54MediumCost

An agent bug looped overnight and spent a month's budget in six hours. Finance's alert, built on the provider's usage and cost reporting, fired the next afternoon. Design a cost alert that pages within about 15 minutes.

#

Claude API. Finance polls the Usage and Cost Analytics API. Services already emit token counters. The monthly budget is $30,000.

Show answer

Why it was late: Anthropic's Usage and Cost Analytics API says data typically refreshes within 4 hours of usage and can take up to 24 hours, and values can be revised until about 7 days after the end of the month. (The legacy Usage and Cost API typically shows data within 5 minutes.) Provider reports are the source of truth for the invoice, not for paging.

Arithmetic: $30,000 in 6 hours is $5,000 an hour. A normal month is about $1,000 a day, or about $42 an hour, so the loop ran at about 120 times normal. Every 15 minutes of delay cost about $1,250.

Design: - Compute spend from your own counters: tokens by model and type (input, output, cache read, cache write) times a versioned price table, recorded per feature. The GenAI conventions say the usage counters, not the per-operation histograms, are for totals and cost approximation. - Alert 1, relative: spend rate more than 3 times the same time last week (PromQL offset 1w), and above an absolute floor so $2 an hour going to $7 does not page. - Alert 2, absolute: spend rate above a fixed hourly ceiling. This also covers new features that have no data a week ago, because a > 3 * (b offset 1w) returns nothing when the old series does not exist. - Leading indicator for loops: model calls or tokens per agent run, from traces or a histogram. - Hard limits, because alerts only tell people: a per-run budget in the agent (stop after N calls or tokens), and separate workspaces with their own limits. - Reconcile your estimate against the provider's numbers daily, and fix the price table when they drift.

Trade-offs

Telemetry-based cost is fast but an estimate; provider data is exact but hours late. Use both, for different jobs.

Common mistake

Paging on the provider's billing data and assuming it is near real time.

Follow-up

What per-run budget would you put on the agent, and what should happen when it is reached?

Prometheus rules: spend anomaly (sketch, your own cost counter) · yaml
groups:
  - name: llm-spend
    rules:
      - record: feature:llm_spend_usd:rate5m
        expr: sum by (feature) (rate(llm_cost_usd_total[5m]))
      - alert: LLMSpendAboveLastWeek
        expr: |
          feature:llm_spend_usd:rate5m * 3600 > 50
          and
          feature:llm_spend_usd:rate5m > 3 * (feature:llm_spend_usd:rate5m offset 1w)
        for: 10m
        labels:
          severity: page
      - alert: LLMSpendAboveCeiling
        expr: feature:llm_spend_usd:rate5m * 3600 > 500
        for: 10m
        labels:
          severity: page
Question 55HardGPU monitoring

Your self-hosted vLLM fleet has an alert 'GPU memory used above 90%'. It fires on every node all the time, so on-call muted it, and last week a traffic spike caused 30 minutes of slow responses without a page. Which metrics should capacity alerts use instead?

#

dcgm-exporter with its default counters and vLLM's Prometheus endpoint are scraped. Each replica owns its GPUs.

Show answer

Why the memory alert is useless: vLLM reserves a fraction of GPU memory for the model executor at startup, set by gpu_memory_utilization, and what is not used by weights goes to the KV cache. So DCGM_FI_DEV_FB_USED_BYTES is high on an idle replica and on a saturated one. It measures the reservation, not load. The busy-time metric DCGM_FI_DEV_GPU_UTIL_RATIO has the same problem in another form: it shows the engine was busy, not how much headroom is left.

Capacity signals come from vLLM: - vllm:kv_cache_usage_perc: fraction of KV cache blocks in use (0 to 1). Sustained values near 1 mean new requests wait for memory. - vllm:num_requests_waiting: requests queued. A queue that stays above zero means demand exceeds the replica. - Preemptions (vllm:num_preemptions, exposed as a counter with a _total suffix): requests evicted for KV space, which waste work and add latency. - Latency symptoms: vllm:time_to_first_token_seconds and vllm:request_queue_time_seconds histograms.

Alert design: - Page on the user symptom: time-to-first-token SLO burn rate. - Warn on leading indicators that explain it: waiting requests above zero for 10 minutes together with KV usage above 0.9, or a sustained preemption rate. - Plan capacity from peak tokens per second per replica, not memory.

What DCGM is for: hardware health and efficiency rather than load. From the current default counters: DCGM_FI_DEV_ROW_REMAP_FAILED, DCGM_FI_DEV_XID_ERROR, DCGM_FI_DEV_GPU_TEMP_CELSIUS, DCGM_FI_DEV_BOARD_POWER_WATTS, and DCGM_FI_PROF_TENSOR_UTIL_RATIO for how well the GPUs are used.

Trade-offs

Engine-level metrics describe capacity accurately but are specific to the serving stack; hardware metrics are uniform across stacks but say nothing about queueing.

Common mistake

Treating GPU memory or busy-time utilisation as a capacity signal for an inference server that preallocates memory.

Follow-up

How would you turn these warnings into an autoscaling signal without flapping?

Prometheus rules: vLLM capacity warning (sketch) · yaml
groups:
  - name: vllm-capacity
    rules:
      - alert: VLLMQueueingWithKvPressure
        expr: |
          max by (model_name) (vllm:num_requests_waiting) > 0
          and
          max by (model_name) (vllm:kv_cache_usage_perc) > 0.9
        for: 10m
        labels:
          severity: warning
      - alert: VLLMPreempting
        expr: sum by (model_name) (rate(vllm:num_preemptions_total[10m])) > 0.1
        for: 10m
        labels:
          severity: warning
Question 56MediumGPU monitoring

You import GPU alert rules from an old wiki page: 'ECC double-bit errors increased', 'thermal throttling detected', and 'Xid error > 0'. With today's dcgm-exporter defaults, two rules never fire and one stays firing long after the event. Why, and how do you fix the rule set?

#

dcgm-exporter runs with its built-in default counters file. Rules were written years ago and never tested.

Show answer

Rules that never fire: in the current etc/default-counters.csv, the ECC counters (for example DCGM_FI_DEV_ECC_DBE_VOL_TOTAL) and the throttle counters (for example DCGM_FI_DEV_GPU_THERMAL_THROTTLE_SECONDS_TOTAL) are commented out, so they are not exported. increase(missing_metric[5m]) > 0 returns nothing, and nothing is never an alert. Current names also carry unit suffixes (_TOTAL, _CELSIUS, _RATIO), so rules copied from older guides may reference names that no longer exist.

The rule that stays on: DCGM_FI_DEV_XID_ERROR is a gauge described as the value of the last Xid error encountered. It is an error code, not a count. > 0 can stay true long after the event, and two identical Xids in a row look like one. The exporter has fields for counting: DCGM_EXP_XID_ERRORS_COUNT (Xids within a window, set by --xid-count-window-size, default 300,000 ms) and DCGM_EXP_XID_ERRORS_TOTAL, both commented out by default.

Fix: - Maintain your own counters CSV, enable the fields your rules need, and pass it with -f / --collectors (default /etc/dcgm-exporter/default-counters.csv). - Alert on counts and changes: Xid count in the window by code (with a triage policy per code), DCGM_FI_DEV_ROW_REMAP_FAILED == 1 as a drain signal, uncorrectable row remaps increasing. - Guard against silence: an absent() rule for one always-present metric per node, so a dead exporter or a renamed field shows up. - Unit-test the rules with promtool test rules against sample series, and re-run the tests when you upgrade the exporter.

Trade-offs

More enabled fields mean more series and collection work per GPU; enable what a rule or dashboard actually uses.

Common mistake

Assuming an alert that has never fired means the fleet is healthy, when the metric it reads does not exist.

Follow-up

Which Xid codes would drain a node immediately, and which would only open a ticket?

Question 57MediumObservability

Your latency panel shows exemplar dots linking p99 spikes to traces, but most dots open 'trace not found'. Prometheus has exemplar storage enabled; the Collector tail-samples, keeping errors plus 5% of other traces. Why, and how do you make the jump from metric to trace reliable?

#

SDKs sample 100% of traces at the head so the tail sampler sees everything. Histograms use default exemplar settings.

Show answer

How exemplars get their trace IDs: the OpenTelemetry SDK's default exemplar filter is trace_based, which makes a measurement eligible when it is recorded in the context of a sampled span. With head sampling at 100%, every measurement is eligible. For explicit bucket histograms the default reservoir keeps at most one exemplar per bucket. The tail sampler then drops about 95% of non-error traces, so most exemplars point at traces that were never stored.

Prometheus side: exemplars are stored only with --enable-feature=exemplar-storage, in a fixed-size in-memory circular buffer sized by storage.exemplars.max_exemplars (default 100,000; the docs estimate about 100 bytes for an exemplar with just a trace ID, so about 10 MB). Old exemplars roll off.

Making it reliable: - Keep the traces exemplars are likely to reference. The dots you click are on slow buckets, so add a tail sampling latency policy whose threshold sits below your p99 region; with one exemplar per bucket, the slow buckets' exemplars then point at kept traces. - Keep error traces (already done) and traces with GenAI failure signals, such as max_tokens finish reasons, through attribute policies. - Accept that a dot on a healthy bucket may be missing, and make the UI say 'sampled out' instead of an error. - Size max_exemplars for your series count if old spikes matter.

Trade-offs

Keeping all slow traces raises storage when latency degrades broadly; it is usually the right cost, since those are the traces people open.

Common mistake

Concluding the tracing backend is losing data, when the sampler is doing exactly what it was configured to do.

Follow-up

How would you measure what fraction of exemplars resolve to a stored trace?

Question 58MediumAlerting

For ten days the quality panel showed a steady 91% judge pass rate. Then someone noticed the line was perfectly flat: the evaluation worker had crashed on day one and the panel kept showing its last value. Quality had in fact dropped. How did a dead worker produce a healthy-looking number, and how do you monitor the monitoring?

#

The batch evaluation worker pushed a judge_pass_rate gauge to a Prometheus Pushgateway after each run.

Show answer

The Pushgateway is a metrics cache, not a store with expiry. Its README says the project decided not to implement a timeout or TTL for pushed metrics, so the last pushed gauge is served to Prometheus forever. Prometheus kept scraping a 91% that was ten days old, and the panel looked healthy.

Two design faults: the score was a pre-computed gauge rather than counts, and nothing checked that the evaluator was alive.

Fixes: - Push or expose counts (judge_results_total by result) and compute the rate in Prometheus. A dead worker then produces no increase, which you can detect, instead of a frozen ratio. - Use the freshness metric the Pushgateway adds: push_time_seconds holds the timestamp of the last successful push per group, and the README recommends alerting when it is much older than expected: time() - push_time_seconds{job="quality-eval"} > 3 * 3600. - For scraped workers, alert on silence with absent_over_time(judge_results_total[2h]), which returns 1 when no samples exist in the window. - Alert on evaluation coverage: judged requests divided by served requests should stay near the sampling rate. A drop means the pipeline is losing work even if the worker is up. - Treat a monitoring failure as an incident: the SRE book lists it among common postmortem triggers, because it usually means problems are found by hand.

Trade-offs

Freshness and coverage alerts add a few rules per pipeline; without them every quality dashboard can fail silently.

Common mistake

Exposing a pre-computed ratio as a gauge, which hides whether anything was measured.

Follow-up

Which other parts of your AI monitoring could fail silently in the same way?

Prometheus rules: alert when quality evaluation stops (sketch) · yaml
groups:
  - name: eval-pipeline-health
    rules:
      - alert: QualityEvalStale
        expr: time() - push_time_seconds{job="quality-eval"} > 3 * 3600
        labels:
          severity: ticket
      - alert: QualityEvalNoResults
        expr: absent_over_time(judge_results_total[2h])
        labels:
          severity: ticket
Question 59MediumObservability

Your client-side 'time to first token' chart shows the same values as total duration for feature A, and suspiciously tiny values for feature B. Both call the same model. What is wrong with how TTFT is measured, and how should it be instrumented?

#

Feature A sends non-streaming requests and records span duration as TTFT. Feature B streams and records TTFT when the SDK call returns the stream object, before reading any events.

Show answer

Neither feature measures time to first output.

- Feature A has no first chunk: a non-streaming response arrives all at once, so its 'TTFT' is just total duration. The GenAI conventions say the gen_ai.client.inference.time_to_first_chunk metric should be reported for streaming calls and should not be reported otherwise. - Feature B stops the clock when the stream object is returned, which can be before any output event has arrived. It may also count the opening message_start event. The conventions define a chunk as a decoded provider event carrying new, non-empty output (text, refusals, reasoning, tool-call names or arguments) and say empty start events, usage-only events, keep-alives and end markers must not count.

Instrument it the way the conventions describe: - Start the clock at the start of the logical operation, before authentication and the first attempt, so retries and backoff are included; users wait through them too. - Stop it when the first real output chunk is fully received, before your application processes it. - Record nothing if no chunk arrives; that request belongs in an error or incomplete count, not in the latency histogram. - Use seconds and the recommended buckets (0.01 doubling to 81.92). On spans the matching attribute is gen_ai.response.time_to_first_chunk. - Remember that time per output chunk is per chunk, not per token, when providers batch several tokens into one event.

Trade-offs

Measuring at the client includes network and retries, which is what users feel; server-side TTFT is cleaner for capacity work. Keep both and label them clearly.

Common mistake

Using span duration as TTFT, or stopping the clock on the first stream event regardless of whether it carries output.

Follow-up

How would you compare client-side and server-side TTFT to find where the latency is added?

Measuring time to first chunk around a streaming call (sketch) · python
import time
from opentelemetry import metrics

meter = metrics.get_meter("support-assistant")
ttfc = meter.create_histogram(
    "gen_ai.client.inference.time_to_first_chunk",
    unit="s",
    explicit_bucket_boundaries_advisory=[0.01, 0.02, 0.04, 0.08, 0.16, 0.32, 0.64, 1.28, 2.56, 5.12, 10.24, 20.48, 40.96, 81.92],
)

OUTPUT_EVENTS = {"content_block_delta"}  # also count tool_use block starts if you stream tools

def stream_answer(client, attrs, **request):
    start = time.monotonic()          # before auth, first attempt and any retries
    first_seen = False
    stream = client.messages.create(stream=True, **request)
    for event in stream:
        if not first_seen and event.type in OUTPUT_EVENTS:
            ttfc.record(time.monotonic() - start, attributes=attrs)
            first_seen = True
        yield event
    # no output chunk: record nothing here; count the request as incomplete elsewhere