supraj.dev

Interview questions & answers

RAG

Retrieval, grounding and citations: follow the evidence through the pipeline.

  • 37 questions
  • 3 easy
  • 23 medium
  • 11 hard
  • 1 related article

37 questions

Question 1MediumGrounding

A RAG response cites the wrong document. What do you inspect?

#

A support answer looks plausible, but its citation points to an unrelated policy. Explain how you would locate the failure before changing the prompt.

Show answer

Capture a redacted trace with the query, retrieved chunk IDs, scores, source versions and final context. Check whether the expected document was indexed and allowed by the access filter. Inspect the retrieved text before blaming generation.

If retrieval is wrong, test query rewriting, chunk boundaries and ranking on a small labeled set. If the right evidence reached the model, compare the answer claims with that evidence. Trace citation IDs through context assembly and rendering; resolve them against an allowlist of retrieved sources. Abstain when the evidence does not support an answer.

Trade-offs

More context or reranking can improve recall, but adds latency and may add distracting evidence. Tune against measured retrieval and answer quality.

Common mistake

Assuming that a valid-looking citation proves the answer is grounded.

Follow-up

How would you distinguish a retrieval miss from a citation-rendering bug in a regression test?

Inspect each boundary independently
  1. Query + access filter
  2. Ranked chunks + source IDs
  3. Context sent to model
  4. Claims matched to evidence
  5. Citation IDs resolved to sources
Question 2EasyProduction reasoning

The answer is missing. Would you improve retrieval or change the model?

#

Your assistant cannot answer a policy question even though the policy exists in the document store.

Show answer

Confirm ingestion succeeded and the policy is current. Run the query directly against retrieval and inspect the evidence. If the required passage is absent, isolate search, filters and chunking. If it is present, test context assembly and answer generation separately. Keep a labeled query with the expected source as a regression case.

Trade-offs

Retrieval makes external knowledge available; it also introduces an index and ranking pipeline to operate.

Common mistake

Buying a larger model before checking whether it received the evidence.

Follow-up

What two metrics would you use to evaluate retrieval and answer quality separately?

A useful sequence to explain
  1. Documents indexed
  2. Query retrieves evidence
  3. Context assembled
  4. Answer checked
Question 3MediumProduction reasoning

A policy exception lives in the next chunk. How would you repair retrieval?

#

The retrieved chunk states the general rule, but drops the exception in the following paragraph.

Show answer

Inspect chunk boundaries and source structure. Keep headings and relevant exceptions together where possible; consider bounded overlap or retrieving neighboring passages. Test cases requiring exceptions and compare coverage, relevance and context size. Preserve source IDs so the combined context remains traceable.

Trade-offs

Larger chunks preserve context but can dilute matching and increase tokens; smaller chunks improve precision but may split evidence.

Common mistake

Using one chunk size for every document and calling the problem solved.

Follow-up

How would a table of limits change your chunking strategy?

Question 4HardProduction reasoning

Two tenants share an index. How do you prevent cross-tenant retrieval?

#

A query from tenant A returns a semantically relevant passage belonging to tenant B.

Show answer

Derive tenant identity from authenticated server context. Enforce authorization before evidence reaches the model and validate returned source ownership. Include caches, document previews and citation endpoints in the same boundary. Test adversarial queries across tenants. Consider separate indexes if that simplifies the required isolation and operating model.

Trade-offs

Shared indexes can simplify operations; separate stores can make boundaries easier to inspect at higher management cost.

Common mistake

Asking the model to ignore documents from other tenants after retrieving them.

Follow-up

What happens to a cached answer after the user loses access to its source?

Question 5HardProduction reasoning

An updated policy is indexed, but answers still quote the old version. Where do you look?

#

The upload succeeded yesterday. Today the assistant still answers with superseded guidance.

Show answer

Compare authoritative document versions with index metadata and retrieved source IDs. Check whether old chunks were removed, update jobs finished and caches use version-aware keys. Reproduce using uncached retrieval before changing prompts. Define freshness requirements and test replacement, deletion and citation behavior end to end.

Trade-offs

Fast incremental updates reduce processing; full rebuilds are simpler to reason about but require time and capacity.

Common mistake

Treating a successful upload as proof that every serving layer is fresh.

Follow-up

How would you roll back a bad indexing release without restoring revoked documents?

Question 6EasyRAG pipeline

Walk through a production RAG request from ingestion to a cited answer.

#
Show answer

RAG first ingests source documents: parse content, retain useful document structure, split passages, attach source/version metadata and build a searchable index. On a request, authenticate the caller, enforce document permissions, retrieve candidate evidence (for example lexical, vector or hybrid search), optionally rerank it and assemble a bounded context. The model then produces an answer grounded in eligible evidence, and the application verifies that cited source IDs resolve to what was actually retrieved.

Retrieval does not guarantee factual correctness. Evaluate retrieval recall and ranking separately from answer faithfulness, citation correctness and appropriate abstention. Freshness, deletion and access revocation are part of the design, not later polish.

Trade-offs

Reranking and larger candidate pools can improve evidence recall, but add latency and context costs.

Common mistake

Describing RAG as only "embedding search + LLM" while omitting source permissions, freshness and evaluation.

Follow-up

If the correct paragraph exists but was not in the model context, which stages would you inspect first?

Question 7HardStructured retrieval

When would a knowledge graph help a RAG assistant, and what does it not guarantee?

#
Show answer

A knowledge graph stores entities and typed relationships, making certain connected queries easier to represent explicitly: customers belong to accounts, services depend on clusters and policies apply to jurisdictions. An assistant can resolve entities, query graph neighborhoods and combine structured results with relevant unstructured passages. This may help multi-hop questions and provenance when the graph is accurate and kept current.

A graph is not automatically correct and does not eliminate hallucinations. Identity resolution, graph update lag, authorization at every traversal edge and provenance of asserted relationships must be tested. Start with ordinary retrieval if it meets the task, then justify a graph using measurable multi-hop or relationship requirements.

Trade-offs

Graphs can improve relationship queries, but increase schema, ingestion and access-control complexity.

Common mistake

Treating knowledge-graph integration as a universal hallucination cure.

Follow-up

How would you prevent a user from traversing a graph edge into another tenant’s confidential data?

Question 8MediumDocument ingestion

A RAG assistant misreads a large table. What should ingestion preserve?

#
Show answer

Keep column headings, row labels, units, captions and footnotes connected to their cells. Use a parser that preserves table structure and provenance rather than flattening every page into anonymous text. When splitting a large table, repeat the relevant headers and retain the source page and table identifiers.

Test questions that need comparisons or exceptions across rows. If the task requires exact aggregation, consider a validated structured query rather than asking the model to calculate from retrieved prose. Better extraction adds work, but embeddings cannot repair lost structure.

Question 9MediumHybrid retrieval

Vector search misses an exact product code. How would hybrid retrieval help?

#
Show answer

Use keyword retrieval for exact identifiers alongside vector retrieval for semantic meaning. Fuse the ranked candidate lists, for example with reciprocal rank fusion, then optionally rerank a bounded set before assembling the answer context. Avoid adding raw scores from unrelated retrieval systems without calibration.

Apply the same access controls to every retrieval path. Evaluate exact-code queries and paraphrases separately, including Recall@k and the final answer’s evidence. Hybrid search adds latency and tuning; keep it only if the measured improvement justifies the complexity.

Question 10MediumRetrieval evaluation

When would Recall@k, MRR and nDCG tell different stories about retrieval?

#
Show answer

Recall@k measures how much of the known relevant evidence appears in the top k results. MRR rewards the position of the first relevant result. nDCG considers ranked, graded relevance and normalizes against the ideal ordering. A retriever can find one useful result early while missing other evidence required for a multi-part answer.

Build query-specific relevance labels and evaluate by query type. Track latency and access-control correctness separately. These metrics test retrieval; they do not prove the generated answer is accurate or grounded. Incomplete relevance labels can also distort the measured result.

Question 11MediumReranking

Why rerank a small candidate set instead of cross-encoding every document?

#
Show answer

A bi-encoder can embed documents ahead of time and retrieve candidates efficiently. A cross-encoder processes the query and each candidate together, providing a more detailed relevance comparison at greater per-pair cost. Retrieve a bounded candidate set, rerank it, and retain the best supported passages.

Measure retrieval recall before reranking: the reranker cannot recover a document that never became a candidate. Benchmark final grounding and latency with representative queries. Keep authorization filters in place and test whether truncation removes decisive evidence from long candidates.

Question 12MediumRetrieval

A question needs facts from two documents, but retrieval returns only one. What do you change?

#

"Compare the 2025 and 2026 leave policies" returns three chunks, all from the 2026 policy.

Show answer

A single query vector tends to land near one document. Split the question: rewrite it into sub-queries (one per policy year), retrieve for each, and merge the results. Metadata filters such as year make each sub-query precise.

When many near-duplicate chunks crowd the top results, retrieve a larger candidate set, rerank, and apply diversity (for example maximal marginal relevance) so different sources survive. Add comparison questions to the evaluation set, with both expected source IDs.

Trade-offs

Query decomposition improves coverage but adds model calls and latency for every question.

Common mistake

Raising top-k until both documents appear, flooding the context with weak chunks.

Follow-up

How would you decide automatically when a question needs decomposition?

Question 13HardEvaluation

How do you evaluate a RAG system before launch without labelling thousands of answers?

#

You have two weeks and one domain expert for a few hours.

Show answer

Build a small, sharp set: 100–200 real questions with the expected source documents. That alone lets you measure retrieval recall at k without grading any prose.

For answers, write a rubric (grounded in the cited sources, answers the question, says "I don't know" when evidence is missing) and use a model as judge, but first have the expert grade a sample and check that the judge agrees with them. Slice results by question type, and include unanswerable questions so you measure refusals too.

Trade-offs

An LLM judge scales cheaply but inherits biases; calibrating against human labels is what makes its scores meaningful.

Common mistake

Grading only final answers, so you cannot tell whether a failure came from retrieval or generation.

Follow-up

What agreement rate between judge and expert would you accept before trusting the judge?

Question 14MediumArchitecture

When is putting everything in a long context window better than retrieval?

#

The whole knowledge base is 150 pages and the model accepts very long prompts.

Show answer

If the corpus is small, changes rarely, everyone may see all of it, and query volume is low, sending it all in the context is simpler and avoids retrieval misses. Prompt caching can make a fixed prefix cheaper.

Retrieval wins when the corpus is large or growing, when users have different permissions, when content changes often, or when you pay for every input token on high traffic. Very long contexts can also bury the relevant passage. Test both on your own questions rather than assuming.

Trade-offs

Long context removes an index to run but raises per-request cost and latency.

Common mistake

Using long context for a multi-tenant corpus and leaking documents across permission boundaries.

Follow-up

How would prompt caching change the cost comparison?

Question 15MediumChunking

Retrieval keeps missing chunks like "Revenue grew 3% over the previous quarter" because the chunk never says which company or quarter it is about. How do you fix chunks that lose their context?

#

Thousands of quarterly filings, split into fixed-size chunks. Hybrid search is already in place.

Show answer

The chunk is correct but anonymous: the company and the period are in the document title or an earlier section, not in the chunk, so neither the embedding nor keyword search can match a query that names them.

Anthropic's contextual retrieval approach fixes this at ingestion. For each chunk, a model is given the whole document and asked for a short explanation of where the chunk fits (Anthropic reports usually 50 to 100 tokens), which is prepended to the chunk before both embedding and BM25 indexing. In their evaluation, contextual embeddings reduced the top-20 retrieval failure rate by 35% (5.7% to 3.7%), adding contextual BM25 brought the reduction to 49%, and adding reranking to 67%. They estimated a one-time cost of about $1.02 per million document tokens using prompt caching, under their stated chunk and document sizes. Treat these as their results on their data; measure on yours.

Cheaper alternatives are worth trying first: prepend the document title, section heading path and dates to every chunk as plain metadata text; or index small chunks for matching but return the larger parent section to the model. The contextual approach is most useful when the missing context is not available as clean metadata.

Anthropic also notes that if the whole knowledge base is under about 200,000 tokens, you can skip retrieval and include it all in the prompt.

Trade-offs

Model-written chunk context improves recall but adds ingestion cost and another model step to version; metadata prefixes are cheap but only carry what the metadata already knows.

Common mistake

Shrinking or growing chunk size repeatedly when the real problem is that chunks carry no identifying context.

Follow-up

How would you re-run contextualisation efficiently when only a few documents change each day?

Question 16MediumQuery understanding

In a support chat, the first question retrieves well, but a follow-up like "and what about contractors?" returns irrelevant chunks. Short, vague questions do badly too. How do you transform queries before retrieval, and what does it cost?

#

The raw latest user message is embedded and sent to the index.

Show answer

The follow-up has no meaning on its own: "contractors" only makes sense with the previous turn about, say, the leave policy. Embedding it alone searches for contractors in general.

First, rewrite conversational follow-ups into standalone queries. A small model takes the recent turns and the new message and produces "What is the leave policy for contractors?", which is what you retrieve with. Log both the original and the rewritten query, because a bad rewrite is now a new failure mode you must be able to see.

Second, for short or vague queries, consider generating a hypothetical answer and searching with that. HyDE (Gao et al.) has an instruction-following model write a hypothetical document for the query, embeds it with an unsupervised encoder such as Contriever, and retrieves real documents near it; the paper reports performance comparable to fine-tuned retrievers without relevance labels. The hypothetical text can be wrong, but it usually uses the vocabulary of real answers, which is what retrieval needs.

Both add a model call before retrieval, so they add latency and cost and can introduce errors. Apply them selectively (rewrite only when there is history, expand only when the query is short), and evaluate them on a set of real multi-turn conversations, not single questions.

Trade-offs

Query rewriting and expansion improve recall on vague and multi-turn queries but add latency, cost and a new place for errors.

Common mistake

Evaluating retrieval only on standalone questions, then shipping a chat product where most questions are follow-ups.

Follow-up

How would you stop a rewrite step from silently changing the meaning of a precise query such as an error code?

Question 17MediumContext assembly

To improve recall you raised top-k from 5 to 30 chunks. Recall went up, but answer accuracy went down, and the model sometimes ignores a relevant chunk that is clearly in the context. Why?

#

Chunks are concatenated in retrieval-score order into a long prompt.

Show answer

More context is not free. Liu et al. found that language model performance is often highest when relevant information is at the beginning or end of the input context, and degrades when it sits in the middle of long inputs, even for models built for long contexts. With 30 chunks, the relevant one is often in the middle, surrounded by near-misses that look plausible. Irrelevant chunks also give the model more wrong material to blend into the answer.

Separate recall from what you send to the model. Retrieve broadly (30 or more candidates) so the right chunk is in the pool, then rerank and pass only the best few to the model. Put the strongest evidence first, remove near-duplicates, and keep the instructions and question clearly separated from the evidence. Ask for citations per claim, which makes it visible when the model used the wrong chunk.

Measure both stages. Recall at the candidate stage tells you whether retrieval found the evidence; answer accuracy and citation correctness at the final stage tell you whether context assembly worked. Tune k for the final context against answer quality, not against recall.

Trade-offs

A larger final context raises the chance that evidence is present but lowers the chance the model uses it well, and costs more tokens.

Common mistake

Treating retrieval recall as the goal and sending every candidate to the model.

Follow-up

How would you test whether your current model still shows position sensitivity at your context lengths?

Question 18EasyEmbeddings

You switched to a multilingual E5 embedding model so users can ask questions in Hindi about English documents. Recall got worse for every language, including English. What is the first thing you check?

#

The ingestion and query code were not changed apart from the model name.

Show answer

Check the input prefixes. The multilingual-e5-large model card says each input should start with query: or passage: , even for non-English text, and answers the question of whether that is needed with: this is how the model is trained, otherwise you will see a performance degradation. Code written for a model without prefixes embeds raw text, so queries and passages land in the wrong regions of the vector space.

Add passage: to every document at ingestion and query: to every search query, then re-embed the corpus, because the existing vectors were made without the prefix. For symmetric tasks such as semantic similarity or paraphrase matching, the card says to use the query: prefix on both sides.

The general lesson: every embedding model has its own input contract (prefixes, task instructions, maximum length, normalisation), and the contract is part of the index version. Store the model name and the input recipe with each index, and run a small recall test from your evaluation set before switching traffic. For the cross-language goal itself, also check keyword search: BM25 will not match a Hindi query to English text, so hybrid ranking may need translation or a per-language keyword index.

Trade-offs

Multilingual models remove the need for translation but may trail a strong single-language model on that language; test both on your queries.

Common mistake

Swapping embedding models as if they were drop-in replacements and skipping a recall check.

Follow-up

How would you version an index so that the query path can never use a different input recipe from the one used at ingestion?

Question 19MediumEmbeddings

Your vector store bill is growing with 3,072-dimension embeddings. Someone proposes cutting every stored vector to its first 512 numbers. Is that valid, and how would you do it safely?

#

Embeddings come from an OpenAI text-embedding-3 model. The index uses cosine similarity.

Show answer

For the text-embedding-3 models it is a supported technique. OpenAI's documentation says you can shorten embeddings, removing numbers from the end of the sequence, without the embedding losing its concept-representing properties, and that a text-embedding-3-large embedding shortened to 256 dimensions still outperforms a full-length text-embedding-ada-002 embedding on the MTEB benchmark. The default lengths are 1,536 for text-embedding-3-small and 3,072 for text-embedding-3-large.

Two details matter. If you shorten vectors yourself, you must normalise them again; the docs suggest using the dimensions API parameter at creation time instead, which returns a properly sized vector. And every vector in an index must have the same length and recipe, so you cannot mix shortened and full vectors: build a new index (or a new vector field) and backfill it, then switch queries to the same dimension.

Do not assume this works for every model. Truncation is only meaningful for models trained for it; for others it destroys information. Evaluate recall at your k on your own queries at several sizes (for example 256, 512, 1,024) and pick the smallest that keeps recall within your tolerance. Combine with quantization if memory is the main cost.

Trade-offs

Shorter vectors cut storage, memory and search time but lose some recall; the curve is model- and corpus-specific.

Common mistake

Truncating vectors without re-normalising them, or truncating vectors from a model that was not trained for it.

Follow-up

How would you combine dimension shortening with quantization and decide which to apply first?

Question 20MediumEvaluation

Users report answers that sound right but contain claims not found in any cited document. Your evals only check whether the final answer matches a reference. What do you add?

#

RAG assistant for internal policy questions, with an eval set of question and reference-answer pairs.

Show answer

Reference matching cannot see this failure: an answer can match the reference because the model knew it, while the retrieved context never supported it, or it can add extra unsupported claims that the grader ignores. You need a groundedness check.

Ragas defines faithfulness this way: an answer is faithful if all its claims can be supported by the retrieved context. It extracts the claims from the response, checks each against the context, and scores the number of supported claims divided by the total, from 0 to 1. Ragas also documents a variant that uses Vectara's HHEM-2.1-Open, a small T5-based classifier trained to detect hallucinations, for the claim-checking step, which is cheaper than an LLM judge.

Run it on every eval case and on a sample of production traffic, and report it next to retrieval metrics. The two together tell you where to fix: low faithfulness with good retrieval means prompt or model problems (the model ignores or embellishes the evidence); low faithfulness with poor retrieval means the model is filling gaps, so fix retrieval or teach it to say it does not know.

Calibrate the judge before trusting it. Have people label a few hundred claims, measure agreement with the automatic check, and re-check when you change the judge model.

Trade-offs

Claim-level groundedness checks catch hallucinations that answer matching misses but cost model calls and need their own calibration.

Common mistake

Measuring only answer correctness, which rewards a model for answering from memory instead of from the evidence.

Follow-up

How would you make the assistant abstain when faithfulness would be low, without making it refuse too often?

Question 21HardAccess control

An employee was removed from the HR group in the document system this morning, but the company assistant still answers their questions using HR-only documents. How did that happen, and how do you design permission checks for RAG?

#

Document ACLs are copied into chunk metadata during a nightly ingestion job. The retriever filters by the user's groups at query time.

Show answer

Two copies of permission data are stale. The chunk metadata holds the ACL as of the last ingestion, and the user's group list may come from a token or cache issued before the change. Filtering at query time is right, but it is only as fresh as the data it filters on. A nightly job means revocations can take a day to apply, which is a data leak for anything sensitive.

Design for revocation, not just for granting. Keep the index filter as a fast first pass, then check the final candidate documents against the source of truth (or a permission service with a short, defined freshness) before they reach the model. Resolve group membership from the identity provider at request time or with a short cache, not from a long-lived token. Subscribe to permission-change events from the document system and update chunk metadata incrementally, with a separate, faster path for revocations than for new content.

Also cover the side doors: cached answers, conversation history that already contains HR text, citation preview endpoints, and summaries built from many documents. Each must be keyed by the permissions that produced it, or invalidated when they change. Write tests that revoke access and check that the next query cannot see the document, and set an explicit SLA for how quickly a revocation takes effect.

Trade-offs

Checking the source of truth per request adds latency and load on the document system; caching permissions is fast but creates exactly this window.

Common mistake

Treating permissions as static document metadata that only needs refreshing with content.

Follow-up

How would you handle a document whose permissions changed while an answer citing it was already cached?

Question 22HardArchitecture

Leadership asks the incident-report assistant "What are the main recurring themes across all of last year's incidents?" and gets an answer based on five random reports. Why does standard RAG fail here, and what would you use?

#

About two thousand incident reports, indexed as chunks for vector and keyword search.

Show answer

Standard RAG retrieves the passages most similar to the question. A global question such as "main themes across everything" has no matching passage; the answer is a summary over the whole corpus. Retrieval returns a handful of reports that happen to mention "recurring" or "themes", and the answer generalises from those.

Microsoft's GraphRAG paper (Edge et al.) targets exactly this, describing it as query-focused summarisation over an entire corpus. It uses a model to build a graph index in two stages: extract an entity knowledge graph from the documents, then pre-compute summaries for groups of closely related entities. A global question is answered from those community summaries. The authors report substantial improvements over a conventional RAG baseline in comprehensiveness and diversity of answers for global sense-making questions on corpora of around a million tokens.

The trade-off is cost and freshness: building the graph and summaries takes many model calls at indexing time, and must be refreshed as documents change. Cheaper options may be enough: a map-reduce pass that summarises each report into structured fields (service, cause, impact) once, then aggregates those fields, often answers theme questions better and more checkably. Use GraphRAG or summarisation for global questions and keep standard retrieval for specific ones, routing between them.

Trade-offs

Graph and summary indexes answer corpus-wide questions but cost many model calls to build and refresh; structured extraction is cheaper and auditable but needs a schema.

Common mistake

Trying to fix corpus-wide questions by raising top-k on a passage retriever.

Follow-up

How would you detect that a user question is global and route it to a different pipeline?

Question 23MediumRanking

For many questions, the top five retrieved chunks are the same boilerplate paragraph from five versions of a document, so the model never sees the one chunk with the actual answer. How do you fix it?

#

Every version of every policy is indexed. Top-k is 5 after reranking.

Show answer

Relevance ranking scores each chunk independently, so five near-identical chunks all score the same and fill the slots. The context gets one fact five times.

Fix it at ingestion first. Deduplicate exact and near-duplicate chunks (hashing normalised text catches exact copies; near-duplicate detection catches small edits), and decide which versions should be searchable at all. Usually only the current version is, with older versions behind an explicit filter for "what did the policy say in 2024" questions.

Then diversify at query time. Maximal marginal relevance (MMR) picks results one at a time, balancing relevance to the query against similarity to results already chosen. Qdrant, for example, implements it as a parameter on a nearest-neighbour query with a diversity value and a candidates_limit for how many candidates to pre-select; its blog examples use 100 candidates and suggest starting at a diversity of 0.5. The candidate pool must be larger than the final limit, or MMR can only reorder what it already has. Libraries differ on which direction the parameter points (some wrappers use a lambda where 1 means pure relevance), so check before tuning.

Measure with questions whose answers sit in non-boilerplate chunks, and check that the right chunk now reaches the final context.

Trade-offs

Diversity brings more distinct evidence but can push out a second, genuinely relevant chunk; deduplication is cheap but needs a policy for versions.

Common mistake

Indexing every historical version without a version policy and hoping ranking will sort it out.

Follow-up

How would you answer a question that explicitly compares two versions of a policy?

Question 24MediumAbstention

To stop hallucinations, the team made the assistant refuse when the top similarity score is below 0.8. Now it refuses good questions and still answers some off-topic ones. Why does a fixed score threshold behave like this?

#

Cosine similarity from a dense embedding model; the threshold was chosen from a few examples.

Show answer

Raw similarity scores are not calibrated probabilities of relevance. Their range depends on the embedding model, the query length and wording, and how dense the corpus is around that topic. A short, generic query can score high against many loosely related chunks, while a precise, long query can score lower against the one exact answer. A single number cannot separate relevant from irrelevant for all queries, so it both refuses good questions and lets off-topic ones through.

Use a signal designed for the decision. A cross-encoder reranker scores query and chunk together and is usually much better at separating relevant from irrelevant; calibrate its threshold on a labelled set of answerable and unanswerable questions and measure both refusal rate and wrong-answer rate. Add a second gate after generation: instruct the model to say when the evidence does not answer the question, and check groundedness of the answer against the cited chunks.

Treat abstention as a feature with its own eval set: real questions the corpus cannot answer, questions it can, and borderline ones. Revisit the threshold whenever you change the embedding model, reranker or chunking, because the score distribution changes with them.

Trade-offs

A strict gate reduces wrong answers but frustrates users with refusals; calibrated reranker scores plus answer checks give a better balance at some latency cost.

Common mistake

Copying a similarity threshold from a blog post or another model and treating it as meaningful.

Follow-up

What would you show the user when the assistant abstains, so the refusal is still useful?

Question 25MediumArchitecture

Users ask the ops assistant "How many P1 incidents did we have last quarter, by team?" and get confident but wrong numbers. Retrieval is working. What is the design flaw?

#

Incident tickets were exported to text and indexed for RAG alongside runbooks.

Show answer

Retrieval returns the top-k most similar chunks, not all matching records. An aggregate question needs every P1 ticket from the quarter, filtered and grouped exactly. The model sees a sample, counts what it was given, and states a number that looks precise. No amount of retrieval tuning fixes this, because the operation is wrong.

Route structured questions to structured data. Keep incidents in a database (or the ticketing system's API) and give the assistant a query tool: either a small set of parameterised queries ("count incidents by severity, team, date range") or carefully controlled text-to-SQL against a read-only view. The model chooses the tool and parameters, the database computes the answer, and the response shows the query or filters used so people can check it. Use RAG for the narrative parts: what happened in an incident, what the runbook says.

Guard the query path like any other data access. Use read-only credentials scoped to the views the user may see, set row limits and timeouts, and validate generated SQL before execution. Evaluate with a set of aggregate questions with known answers, because this is where silent wrong numbers do the most damage.

Trade-offs

Parameterised queries are safe and testable but cover fewer questions; free text-to-SQL covers more but needs strong validation and permission controls.

Common mistake

Indexing tabular or transactional data as text and asking a language model to count it.

Follow-up

How would you stop text-to-SQL from reading data the user is not allowed to see?

Question 26MediumOperations

Your RAG assistant has been live for three months. Without labelled answers for production traffic, how do you notice that retrieval quality is getting worse?

#

New documents are ingested daily; an ingestion parser was upgraded last month.

Show answer

Combine cheap health signals with a small amount of ground truth.

Health signals from every request: rate of queries with no results or only low-scoring results, distribution of top reranker scores, abstention rate, share of answers with citations, number of distinct sources cited, and latency per stage. Watch them over time and by document collection. A parser upgrade that breaks tables, for example, shows up as falling scores and more abstentions for questions about one collection.

Ground truth: a canary set of real questions with known source documents, run daily against production. Track recall at k and answer correctness on it, and alert on drops. Refresh it as the corpus changes, so it does not test only old documents. Add sampled review: every week, people grade a small random sample of production conversations for retrieval relevance and groundedness, which also finds new failure types for the eval set.

Ingestion metrics close the loop: documents parsed per day, parse failures, empty or tiny chunks, and documents with no text extracted. Many retrieval regressions are really ingestion regressions. Keep traces with chunk IDs and versions so any bad answer can be traced back to what was retrieved and why.

Trade-offs

Daily canaries and human review cost time and model calls but catch regressions that health metrics miss; health metrics are cheap but only hint at quality.

Common mistake

Relying on user thumbs-down feedback alone, which arrives late and covers a small, biased sample.

Follow-up

Which of these signals would you alert on, and which would you only review weekly?

Question 27HardChunking

A teammate proposes "late chunking" instead of the contextual retrieval step you run today, which calls an LLM to write a context line for every chunk. What does late chunking actually do, when is it a better choice, and what does it cost you?

#

Internal wiki of about 40,000 pages; most pages fit in 8K tokens. The contextual-retrieval LLM calls are now the largest line in the ingestion bill.

Show answer

Normal chunking splits the text first and embeds each chunk on its own, so a chunk that says "it increased by 3%" never sees which product "it" is. Late chunking, from the Jina paper (arXiv 2409.04701), reverses the order. The whole document goes through the embedding model's transformer once, so every token embedding is computed with the full document as context. The chunk boundaries are applied only afterwards, just before mean pooling: each chunk's vector is the average of its own token embeddings. The chunks are the same size as before, but each one carries information from its neighbours. The paper reports that it needs no extra training and improves retrieval across a range of tasks.

Why it can replace your current step: contextual retrieval pays for an LLM call per chunk at ingestion; late chunking pays for one embedding forward pass per document. For a wiki where most pages fit in the embedding model's window, that is a large saving.

What it costs you: - It needs a long-context embedding model with mean pooling, and documents longer than its window must still be split into segments, so context is lost across those boundaries. - Chunk vectors are no longer independent. Editing one paragraph changes the token embeddings of the whole page, so you must re-embed the whole document, not one chunk. - It adds implicit context, not readable text. Contextual retrieval also writes a sentence you can add to the BM25 index and show in debugging; late chunking does not help keyword search at all.

Decide with an A/B on your own retrieval eval set: current pipeline, late chunking, and late chunking plus BM25. Compare recall at your top-k and the ingestion cost per 1,000 pages.

Trade-offs

Late chunking is cheap and needs no LLM at ingestion, but it ties you to a long-context embedding model and whole-document re-embedding; contextual retrieval costs more but also improves keyword search.

Common mistake

Turning on late chunking with a model whose context window is shorter than the documents, so most of the expected context is cut off.

Follow-up

How would you handle documents that are ten times longer than the embedding model's window?

Question 28MediumChunking

Small chunks (about 100 tokens) give you precise retrieval, but the model often answers without the conditions stated in the surrounding paragraphs. Large chunks (about 1,500 tokens) give context but retrieval gets vaguer. How do you get both?

#

Insurance policy documents with long sections, where a rule and its exceptions sit a few paragraphs apart.

Show answer

Separate the retrieval unit from the generation unit. This is usually called small-to-big or parent-document retrieval.

Index small chunks, because a short chunk has one main idea and its embedding matches a specific question well. Store with each small chunk a pointer to its parent: the section or the surrounding window it came from. At query time, search over the small chunks, then replace each hit with its parent (or with the hit plus N neighbouring chunks) before building the prompt. If several hits share a parent, send that parent once.

Design details that decide whether it works: - Pick the parent from the document structure (the heading section, the clause), not a fixed token count, so a rule and its exception stay together. - Cap the parent size and count, or a few large sections fill the context window and push out other sources. - Keep citations at the small-chunk level, so the user still sees the exact sentence that matched, inside the larger context. - Rerank before the expansion, on the small chunks, because a cross-encoder judges a short passage more reliably than a long one.

Measure two things separately: retrieval recall on the small chunks, and answer correctness after expansion. If recall is good but answers still miss exceptions, your parents are too small or cut at the wrong boundary.

Trade-offs

Expanding to parents improves answers that depend on nearby conditions, at the cost of more prompt tokens per request and a second lookup store.

Common mistake

Choosing one chunk size as a compromise between retrieval and context, which is too big to match well and too small to carry the exceptions.

Follow-up

How would you decide between parent sections and a fixed window of neighbouring chunks for transcripts, which have no headings?

Question 29HardDocument ingestion

Your RAG assistant answers badly on slide decks, scanned forms and reports full of charts. The OCR and layout pipeline is already the most complex part of ingestion. When would you index page images directly with a ColPali-style retriever, and what changes downstream?

#

About 300,000 pages, most of them slides and scanned PDFs. Many answers are in a chart or a table rendered as an image.

Show answer

A text pipeline only retrieves what OCR and layout parsing manage to extract. Chart values, the position of a number in a form, and the relation between a caption and a figure are often lost or garbled before the embedding model ever sees them. The ColPali paper (arXiv 2407.01449) argues this extraction step is slow and brittle, and instead embeds the page image itself with a vision-language model. Each page becomes many vectors (one per image patch), and a query is matched with late interaction, in the ColBERT style. On the paper's ViDoRe benchmark of visually rich page retrieval, the authors report that it clearly beats modern text-based pipelines while being simpler to run.

Use it when much of the answer lives in layout, charts or images, and when the parsing pipeline is a constant source of bugs. Keep text retrieval for born-digital prose, where extracted text is clean and cheap to search.

What changes downstream: - Storage and search: multi-vector pages are far larger than one vector per chunk, and you need a store that supports late-interaction scoring (or a first-stage retriever plus a multi-vector rerank). - Granularity: you retrieve whole pages, not paragraphs, so citations point to a page, and a dense page uses a lot of the context window. - Generation: the model must read the page image, so the generator has to be a vision-language model, and image tokens cost more than the same text. - Exact match: product codes and IDs inside an image still benefit from a keyword index built from OCR, so a hybrid design is common.

Prove it with your own eval: take a few hundred real questions whose answers are in charts or forms, and compare page-level recall of both pipelines before rebuilding ingestion.

Trade-offs

Page-image retrieval removes a brittle parsing stage and sees visual content, but costs more storage, needs a multimodal generator and gives coarser citations.

Common mistake

Spending months tuning OCR for charts and slides, then measuring only text-heavy questions where the parser already works.

Follow-up

How would you combine page-image retrieval with a BM25 index so exact codes are still found?

Question 30MediumGrounding

Your prompt asks the model to cite chunks as [1], [2] and quote the supporting text. Audits find quotes that are slightly reworded or do not exist in the document. How do you make citations verifiable, and what constraints come with an API-level citations feature?

#

A compliance assistant on the Claude API. Answers are shown to auditors, who click a citation and expect the exact sentence.

Show answer

A quote the model writes is generated text, so it can drift from the source even when the citation number is right. The fix is to have the quoted text extracted from the document by the system, not written by the model.

If you control the pipeline, verify every quote: check that it appears in the cited chunk (exact or normalised match), and drop or flag the claim when it does not.

On the Claude API you can use the citations feature instead. You enable it per document block, and the response returns citation objects whose cited_text is extracted from the document, with a location: character indices for plain text, page numbers for PDFs, and block indices for custom content documents. The documentation says citations are guaranteed to contain valid pointers to the provided documents, that cited_text does not count toward output tokens, and that in Anthropic's evaluations the feature cited the most relevant quotes more often than prompt-based approaches.

Constraints to design around: - Granularity: plain text and PDFs are chunked into sentences. For RAG, the docs suggest putting each retrieved chunk in its own document; use custom content documents if you need bullet- or block-level citations with no further chunking. - Scanned PDFs without extractable text cannot be cited, and citing images in PDFs is not supported. - Citations cannot be combined with structured outputs; the API returns an error if you enable both. - Enabling citations slightly increases input tokens.

A valid pointer is not the same as support: the cited sentence can be real and still not justify the claim, so keep a faithfulness check in your evals.

Trade-offs

API-extracted citations remove fabricated quotes, but limit you to the supported document types and rule out strict JSON output in the same call.

Common mistake

Treating a correct citation number as proof that the quoted text is real, without checking it against the source.

Follow-up

Your downstream system needs strict JSON as well as citations. How would you split the work across calls?

Question 31HardEmbeddings

A general-purpose embedding model retrieves poorly on your internal engineering jargon. You have 50,000 (question, correct passage) pairs from support logs. How do you fine-tune it, and what can quietly ruin the training?

#

You plan to use sentence-transformers on one GPU. Many questions are near-duplicates of each other, and some passages answer several questions.

Show answer

The standard recipe is contrastive training with MultipleNegativesRankingLoss in sentence-transformers. It takes (anchor, positive) pairs, optionally with hard negatives as extra columns, and for each anchor it pushes the similarity to its own positive above the similarity to every other passage in the batch. The other passages in the batch are free negatives, so bigger batches give more and harder negatives.

Three things quietly ruin it: - False in-batch negatives. If two near-duplicate questions, or two questions with the same passage, land in one batch, the loss trains the model to push a correct passage away. The library docs recommend BatchSamplers.NO_DUPLICATES so in-batch negatives are not duplicates of the anchor or positive. You should also deduplicate near-identical questions yourself. - False hard negatives. Mining the passages the current model ranks highest finds passages that are often also correct. mine_hard_negatives has options for this: range_min to skip the top few matches, max_score, and relative_margin to keep a negative only if it scores clearly below the positive. Its docs point to the NV-Retriever paper's setting of a 5% relative margin. - Batch size limited by memory. CachedMultipleNegativesRankingLoss uses GradCache to reach much larger batches with the memory of a small one; its docs say it trains roughly 2x to 2.4x slower.

Hold out by time or by document, not by random pair, so near-duplicates do not leak into the test set. Evaluate recall at your production top-k against the base model, and re-embed the whole corpus when you switch, because vectors from the two models are not comparable.

Trade-offs

Fine-tuning usually gives large gains on in-domain jargon but can lose general ability, and every model update forces a full re-index.

Common mistake

Mining the top-ranked passages as hard negatives without a margin, so the model learns to reject correct answers.

Follow-up

How would you keep the fine-tuned model from getting worse on general questions that the base model handled well?

Question 32MediumCaching

To cut cost, the team added a semantic cache: if a new question embeds close to a previous one, return the previous answer. Within a week, one customer saw an answer about another customer's contract, and "how do I cancel my plan" returned the answer for "how do I keep my plan". What went wrong, and how would you design the cache?

#

Multi-tenant support assistant; answers depend on the user's plan and on documents that change weekly.

Show answer

Two design errors.

The key was incomplete. A RAG answer depends on more than the question: the tenant, the user's permissions, their plan or region, the document versions retrieved, the prompt version and the model. A cache keyed only on question similarity serves one tenant's answer to another, which is a data leak, and keeps serving old answers after documents change.

Similarity is not equivalence. Embeddings place "cancel my plan" and "keep my plan" close together because they share topic and words. Negations, numbers, dates and named entities are exactly what a similarity threshold misses.

A safer design: - Partition the cache by tenant and permission scope, and include prompt, model and index version in the key, so a release or re-index invalidates it. - Prefer exact or normalised-text matching for answers. Use semantic matching only for low-risk, tenant-independent content such as public FAQs, with a strict threshold tuned on labelled pairs that include negations. - Cache lower-risk intermediate results instead: query embeddings, retrieval results for a document version, or provider prompt caching for the shared prefix, which saves cost without returning a stale final answer. - Set a TTL and invalidate on document updates. - Log every cache hit with the original question, and sample hits for review.

Measure the saving honestly: hit rate times cost per answer, minus the cost of wrong answers and the extra infrastructure.

Trade-offs

A strict, partitioned cache is safe but has a low hit rate; a loose semantic cache saves more money but serves wrong and leaked answers.

Common mistake

Keying the cache only on question similarity, which ignores tenant, permissions and document version.

Follow-up

How would you build the labelled set of question pairs used to tune the threshold for the public FAQ cache?

Question 33MediumPerformance

Your RAG assistant's p95 time to first token is 6 seconds and product wants under 2. Where does the time usually go, and in what order would you attack it?

#

Pipeline: LLM query rewrite, embedding call, vector search, keyword search, cross-encoder rerank of 100 candidates, then a large model with 20 chunks in the prompt.

Show answer

First measure, per stage, at p50 and p95: rewrite, embedding, each search, rerank, prompt assembly, and the model's time to first token. Guessing is usually wrong, because tail latency often comes from one stage with high variance, not the average.

Common findings, roughly in order of payoff: - The query rewrite is a full LLM call before retrieval even starts. Use a small fast model, skip the rewrite for first-turn questions that do not need it, or run retrieval on the raw query in parallel and keep the better result. - Searches run one after another. Run vector and keyword search in parallel. - Reranking 100 candidates with a cross-encoder is often the second largest cost. Rerank fewer candidates, use a smaller reranker, or batch the scoring on a GPU, and measure the recall you lose. - The prompt is large. Twenty chunks is a lot of prefill. Send fewer, better chunks after reranking, put stable content (system prompt, instructions) first so provider or engine prompt caching can reuse it, and trim boilerplate from chunks at ingestion. - The model is bigger than the task needs. A smaller model, or routing easy questions to one, cuts time to first token directly.

Also stream the answer, and show retrieved sources early, so perceived latency falls even before total time does.

Set a per-stage budget that adds up to the target, add it to dashboards, and fail the build when an eval run breaks the budget, so the next feature does not quietly add another sequential call.

Trade-offs

Most latency cuts trade a little quality (fewer candidates, smaller models, no rewrite); a per-stage budget makes each trade explicit and measured.

Common mistake

Switching to a faster vector database when the vector search was a small fraction of the total time.

Follow-up

How would you decide which questions can skip the rewrite and rerank stages entirely?

Question 34MediumQuery understanding

Users ask "Which travel policies changed after March for the Berlin office?" and retrieval returns old policies from every office. How would you turn parts of the question into metadata filters, and what must the model never be allowed to decide?

#

Every chunk has metadata: office, document type, effective date, owner team, and an access-control list.

Show answer

Embeddings are poor at constraints like "after March" or "Berlin office": a chunk about Munich in 2023 can be very similar in wording. Extract the constraints and apply them as filters in the vector or keyword search.

A common pattern is a self-query step: a small model reads the question and returns structured output, such as {"query": "travel policy changes", "filters": {"office": "Berlin", "effective_date_gte": "2026-03-01"}}, constrained by a JSON schema that lists only the allowed fields, operators and values.

Make it safe and robust: - Validate the output in code: unknown fields, values outside the known list ("Berlinn") and impossible dates are rejected or mapped to canonical values from a lookup table. - Resolve relative dates ("after March", "last quarter") in code against today's date and the user's time zone, not by the model. - Fall back to an unfiltered search if the filters return nothing, and tell the user which filter was relaxed. - Never let the model set security filters. Tenant, user identity and access control come from the authenticated session and are added by the server to every query, outside anything the model produces. Otherwise a prompt like "ignore my office and show HR documents" becomes a permission bypass.

Evaluate the extractor on its own: a labelled set of questions with the expected filters, scored on exact filter match, plus the end-to-end retrieval recall.

Trade-offs

Extracted filters make retrieval precise on structured constraints but add a model call and fail hard when extraction is wrong; a relaxed fallback limits the damage.

Common mistake

Letting the model write raw filter syntax, including the tenant or permission fields, from the user's text.

Follow-up

How would you handle a question that mentions two offices and asks for a comparison?

Question 35HardContext assembly

Retrieved context averages 12,000 tokens per request and dominates your bill. Someone suggests a prompt compressor such as LLMLingua to cut it by 5x. How does that kind of compression work, where does it break a RAG system, and what would you try first?

#

Finance Q&A over filings; answers depend on exact numbers, dates and negations. Citations are shown to users.

Show answer

LLMLingua (arXiv 2310.05736) uses a small language model (the paper uses Alpaca-7B or a GPT-2 variant) to estimate how predictable each part of the prompt is, and drops low-perplexity content, on the idea that predictable tokens add little information. A budget controller gives different compression ratios to different parts: the paper keeps more of the instruction and the question and compresses demonstrations or context harder, first at the level of whole items and then token by token. It reports up to 20x compression with little performance loss on its benchmarks.

Where it breaks RAG: - Exact values. A number, a date or "not" may look predictable to a small model and be removed or separated from what it qualifies. In finance that changes the answer. - Citations. The model now sees a compressed text that no longer matches the source, so quotes and character offsets cannot be checked against the original. - Extra latency. A 7B scoring model runs on every request before the main model.

Try cheaper, safer cuts first: - Retrieve fewer, better chunks: rerank, then send the top few instead of everything. - Remove boilerplate (headers, footers, legal disclaimers) at ingestion. - Use small-to-big only where needed, not by default. - Put the stable part of the prompt first and use prompt caching, which cuts the cost of repeated prefixes without changing the text.

If you still try compression, compress only low-risk parts (long background passages), never the sentences you cite, and evaluate on questions whose answers depend on numbers and negations.

Trade-offs

Compression can cut tokens sharply, but it removes exactly the details that small models find predictable and breaks quote-level citations; better retrieval saves tokens without that risk.

Common mistake

Judging a compressor on average benchmark scores instead of on questions that hinge on one number or one "not".

Follow-up

How would you measure the information loss of a compressor on your own corpus before rolling it out?

Question 36HardArchitecture

Your fixed retrieve-then-answer pipeline gives confident answers even when retrieval found nothing useful. A teammate wants "agentic RAG": let the model judge the results and search again. How would you design that loop so it helps instead of adding cost and new failures?

#

Internal knowledge assistant. Data must not leave the company, so there is no public web search.

Show answer

The useful idea is to grade retrieval before generating. Corrective RAG (CRAG, arXiv 2401.15884) adds a lightweight retrieval evaluator (a fine-tuned T5-large, 0.77B parameters) that scores how relevant the retrieved documents are. Its confidence triggers one of three actions: Correct (at least one document scores above an upper threshold, so its relevant strips are kept and used), Incorrect (all are below a lower threshold, so they are discarded and web search is used instead), or Ambiguous (both sources are combined). The thresholds were set empirically per dataset.

Adapting it to an internal assistant: - Replace web search with internal fallbacks: a rewritten query, a keyword search, a different collection, or a structured lookup. Or abstain with a useful message. - Use a small, fast grader (a reranker score or a small classifier) rather than the main model, and calibrate its thresholds on your own labelled query-document pairs. - Bound the loop: a maximum number of retries and a total latency budget. An agent that can search forever will on hard questions. - Log every step (query, results, grade, action) so you can see whether retries actually find better documents.

Evaluate against the fixed pipeline on the same set: answer accuracy, abstention rate on unanswerable questions, and p95 latency and cost. A loop that only helps 2% of questions but doubles latency for all of them should run only when the grader says retrieval is weak.

Trade-offs

A graded retry loop catches weak retrieval and supports honest abstention, but adds latency, cost and another model that must be calibrated.

Common mistake

Letting the main model decide without limits whether to search again, which gives unbounded latency and no evidence that retries help.

Follow-up

How would you build the labelled data to calibrate the grader's upper and lower thresholds?

Question 37MediumRetrieval evaluation

You need relevance labels for 5,000 query-passage pairs to compare retrievers, and expert annotators are scarce. Can you use an LLM to label relevance, and how do you make sure the labels can be trusted?

#

Two candidate retrievers, one of which uses embeddings from the same model family as the proposed judge.

Show answer

Yes, with checks. Researchers at Bing (arXiv 2309.10621) reported that LLM relevance labels can be as accurate as human labellers and, measured against labels from real searchers, better than third-party workers, at a fraction of the cost. They also found that accuracy changes with the prompt, even with simple paraphrases, so the prompt is part of the measurement.

A trustworthy setup: - Have experts label a few hundred pairs, sampled across query types and both retrievers. This is your gold set. - Write a rubric prompt with graded levels (for example, answers the question, partly relevant, not relevant) and the query intent, and measure the judge's agreement with the gold set (Cohen's kappa or accuracy per level). - Freeze the judge model, prompt and settings once validated, and record them with the results. A new judge version needs a new validation. - Pool candidates from both retrievers and label them blind, so the judge does not know which system found a passage. - Watch for self-preference. A judge from the same family as one retriever's embedding model may share its notion of similarity; compare the judge's agreement with the gold set separately on each retriever's results. - Keep re-checking: have experts relabel a small random sample each cycle.

Then compute recall, MRR or nDCG from the labels and report confidence intervals, because small differences between retrievers may be within the judge's error.

Trade-offs

LLM labels are fast and cheap enough to label thousands of pairs, but they are a measurement instrument that must be validated and versioned like any other.

Common mistake

Using an unvalidated judge prompt and treating its labels as ground truth when choosing between retrievers.

Follow-up

How many gold labels do you need to estimate the judge's agreement precisely enough to trust a 2-point nDCG difference?