Day 15 Theory — RAG Retrieval and First Quantitative RAG Evaluation

From semantic search to a RAG system

Yesterday we prepared documents for retrieval. Today we connect retrieval to generation.

Retrieval-Augmented Generation (RAG) separates two responsibilities:

1. find evidence relevant to the user's question;
2. ask the model to answer using that evidence.

The model does not magically query a vector database by itself. Our application owns the retrieval step and decides what context is sent to the model.

User question
     |
     v
Retriever
     |
     v
Ranked source chunks
     |
     v
Prompt/context builder
     |
     v
LLM
     |
     v
Answer + citations

That separation is useful because retrieval and generation can fail for different reasons.

Why we evaluate retrieval separately

Imagine the correct policy document is not in the retrieved top results.

The model may then:

• answer from general training knowledge,
• use an irrelevant document,
• guess,
• refuse because evidence is missing.

If we only inspect the final answer, we may incorrectly blame prompting or the model.

Now imagine the correct document was retrieved, but the model ignores it and answers incorrectly. That is a different failure.

So our first debugging split is:

Wrong answer
   |
   +-- Did retrieval find the needed evidence?
   |       |
   |       +-- no -> retrieval problem
   |
   +-- yes -> generation/context/use-of-evidence problem

This simple separation saves enormous debugging time.

Build a retrieval evaluation dataset

A retrieval eval case needs at least:

• a realistic user query,
• one or more known relevant source IDs/chunks,
• optional metadata such as category or difficulty.

Example:

{
  "query": "I changed my phone and MFA no longer works",
  "relevant_sources": ["identity-handbook#mfa-reenrollment"]
}

The expected source should come from domain knowledge or reviewed labels—not from whichever result the current retriever already returns.

Otherwise we would be grading the system using its own guesses.

Recall@k

Recall@k asks:

Of all relevant items for this query, how many appeared in the first k retrieved results?

Suppose the known relevant documents are:

A, B

and the retriever returns top 5:

A, X, Y, Z, Q

It retrieved one of the two relevant items.

recall@5 = 1 / 2 = 0.5

If the task normally has one required answer source, recall@k often behaves like a simple hit/miss question:

Was the required evidence present in the top k?

High recall matters when missing relevant evidence is costly.

Precision@k

Precision@k asks:

What fraction of the first k results are relevant?

Suppose top 5 is:

A, B, X, Y, Z

and only A and B are relevant.

precision@5 = 2 / 5 = 0.4

Why care?

Because every irrelevant chunk we put into the model context can:

• consume tokens,
• increase latency/cost,
• distract the model,
• introduce conflicting instructions or facts.

RAG is therefore not just “maximize recall by retrieving everything.”

Mean Reciprocal Rank (MRR)

Sometimes the first correct result matters a lot.

Reciprocal rank for one query is:

1 / rank_of_first_relevant_result

If the first relevant document is ranked:

rank 1 -> score 1.0
rank 2 -> score 0.5
rank 4 -> score 0.25

Mean Reciprocal Rank (MRR) averages that value across many eval queries.

For a support assistant where one authoritative policy should ideally appear first, MRR can be informative.

Do not choose MRR simply because it is popular. Choose metrics that reflect the product's retrieval need.

Metric choice depends on the task

Consider two systems.

Policy assistant

A user asks a narrow policy question and one authoritative document should answer it.

We may care strongly about:

• first relevant result rank,
• citation correctness,
• not retrieving stale policy.

Research assistant

The user wants broad coverage across several sources.

We may care more about:

• recall across multiple relevant documents,
• diversity,
• coverage.

There is no universal “best RAG metric.”

Evaluate slices, not only averages

Suppose overall recall@5 is 0.90.

That sounds strong.

But imagine:

Password reset queries:      0.98
MFA queries:                 0.97
Certificate renewal:         0.96
Security policy questions:   0.55

The aggregate hides a serious weakness.

Create slices such as:

• exact identifier queries,
• natural-language paraphrases,
• ambiguous queries,
• security-sensitive questions,
• stale-version traps,
• no-answer queries.

Averages are useful summaries, not substitutes for diagnosis.

Retrieval score is not answer quality

A perfect retrieval score does not prove the generated answer is correct.

Likewise, a plausible final answer does not prove retrieval worked.

Keep the stages explicit:

Retrieval quality
      |
      v
Context quality
      |
      v
Generation quality
      |
      v
Final product behavior

Each stage has its own failure modes.

Grounded generation

Once evidence is retrieved, the prompt should make the evidence contract clear.

For a support-policy answer we may instruct the model to:

• use only supplied approved sources for policy claims,
• cite source identifiers,
• say when evidence is insufficient,
• identify conflicting sources rather than silently choosing one,
• avoid inventing a policy that is not present.

These instructions improve behavior but do not guarantee it.

That is why we evaluate generated answers too.

Faithfulness versus relevance

Two dimensions are easy to confuse.

Relevance

Does the answer address the user's question?

Faithfulness / groundedness

Are the answer's factual claims supported by the supplied evidence?

Example:

User asks:

“Can contractors reset executive passwords?”

Retrieved document says only:

“Employees may reset their own password after MFA verification.”

The model answers:

“Yes, contractors may reset executive passwords after MFA.”

The answer is highly relevant to the question, but it is not faithful to the evidence.

A RAG evaluator should distinguish those dimensions.

Citation design

A citation is useful only if it maps back to real evidence.

Bad design:

Model freely invents: [Policy-17]

Better design:

• every retrieved chunk has a stable source ID;
• the model is shown those IDs;
• the output references only known IDs;
• the application validates or constructs citation links.

For example:

{
  "source_id": "identity-handbook#mfa-reenrollment",
  "title": "Identity Handbook — MFA Re-enrollment",
  "url": "/kb/identity-handbook#mfa-reenrollment"
}

The model should not create new citation identities from imagination.

No-answer behavior is part of quality

A strong RAG system should sometimes say:

“The available documents do not establish that policy.”

That can be more correct than generating a fluent answer.

Add eval cases where:

• the answer is absent from the corpus,
• sources conflict,
• all available sources are stale,
• the user asks outside the allowed domain.

If every eval case has a clean answer, you are not testing realistic uncertainty.

Retrieval tuning should be evidence-driven

Parameters such as:

• chunk size,
• overlap,
• embedding model,
• top_k,
• metadata filters,
• lexical/semantic weighting,
• reranking,

should be changed against a stable eval set.

The workflow is:

baseline
  |
  v
change one retrieval design
  |
  v
run same eval set
  |
  v
compare metrics + slices
  |
  v
keep / reject change

Without this loop, teams tune RAG based on a few memorable demo questions.

Small worked evaluation

Suppose we have three eval queries:

Q1 relevant source appears at rank 1
Q2 relevant source appears at rank 2
Q3 relevant source not in top 5

Reciprocal ranks are:

Q1: 1/1 = 1.0
Q2: 1/2 = 0.5
Q3: 0

So:

MRR = (1.0 + 0.5 + 0) / 3 = 0.5

That number is not “50% correct answers.” It specifically summarizes how early the first relevant item appears across those retrieval cases.

Always explain what a metric means before placing it on a dashboard.

Service Desk connection

Today the Service Desk moves from semantic search to a measurable RAG pipeline.

Question
   ↓
Retrieve
   ↓          <- measure recall/rank/precision
Evidence
   ↓
Generate
   ↓          <- measure groundedness/relevance
Answer
   ↓
Citations     <- validate real source identity

The principle is:

RAG quality is not a feeling. Measure whether retrieval found the right evidence, then separately measure whether generation used that evidence correctly.