A RAG response cites the wrong document. What do you inspect?
#A support answer looks plausible, but its citation points to an unrelated policy. Explain how you would locate the failure before changing the prompt.
Show answer
Capture a redacted trace with the query, retrieved chunk IDs, scores, source versions and final context. Check whether the expected document was indexed and allowed by the access filter. Inspect the retrieved text before blaming generation.
If retrieval is wrong, test query rewriting, chunk boundaries and ranking on a small labeled set. If the right evidence reached the model, compare the answer claims with that evidence. Trace citation IDs through context assembly and rendering; resolve them against an allowlist of retrieved sources. Abstain when the evidence does not support an answer.
Trade-offs
More context or reranking can improve recall, but adds latency and may add distracting evidence. Tune against measured retrieval and answer quality.
Common mistake
Assuming that a valid-looking citation proves the answer is grounded.
Follow-up
How would you distinguish a retrieval miss from a citation-rendering bug in a regression test?
- Query + access filter
- Ranked chunks + source IDs
- Context sent to model
- Claims matched to evidence
- Citation IDs resolved to sources