Module 3: RAG Infrastructure & Retrieval · 3h

01 · UNDERSTAND

Day 14 Theory — RAG Ingestion: Parsing, Chunking, Metadata and Storage

Retrieval quality starts before the query

A common mistake is to blame the vector database when RAG results are poor. Often the real problem was created during ingestion.

If documents are parsed incorrectly, split at bad boundaries, stripped of useful headings, or stored without source metadata, retrieval begins with damaged evidence.

Today we build the pipeline that prepares knowledge for search.

The ingestion pipeline

source document
   ↓
parse / extract text
   ↓
clean while preserving meaning
   ↓
split into chunks
   ↓
attach metadata
   ↓
embed
   ↓
store/index

Each stage should be observable and repeatable.

Parsing is document-specific

PDF, HTML, Markdown, Word files and ticket exports have different structure.

A parser should preserve meaningful information such as:

headings,
paragraphs,
lists,
table context,
document title,
page/section identity.

Blindly flattening everything into one string can destroy relationships that retrieval needs.

Why chunking exists

Embedding an entire long manual as one vector creates a broad representation that may hide small relevant details. Splitting into smaller chunks creates more precise retrieval units.

But chunks that are too small can lose context.

This creates a trade-off:

small chunks -> precise but context-poor
large chunks -> context-rich but less focused

There is no universally correct chunk size.

Prefer semantic boundaries when possible

Natural boundaries—headings, paragraphs, code blocks, policy sections—often produce better chunks than cutting every fixed number of characters.

Overlap can preserve continuity across boundaries, but overlap also duplicates tokens and can return near-identical chunks.

Tune it using retrieval evaluation.

Metadata is part of retrieval

Useful metadata may include:

source document ID,
title,
section,
version,
updated timestamp,
access classification,
product/team,
language.

Metadata enables filtering and citations. It can also support deletion and re-indexing when a document changes.

Stable identifiers and re-ingestion

Production knowledge changes.

If every ingestion creates unrelated IDs, stale chunks can remain alongside new chunks.

Design stable document/chunk identity so an update can replace or invalidate previous versions.

Access control starts at ingestion design

Do not mix confidential and public documents in a retrieval system without a plan for authorization-aware filtering.

Retrieving a secret document and then hoping the model “doesn't mention it” is not an access-control strategy.

Store metadata that lets the retrieval layer enforce tenant and permission boundaries before content reaches the model.

Service Desk connection

Today we convert internal support documentation into a traceable retrieval corpus.

The principle is:

RAG quality is shaped during ingestion. Preserve structure, choose retrieval-sized chunks, attach source and access metadata, and make updates reproducible.

02 · APPLY

Lesson Overview

This is the applied companion for Day 14. Read DAY_14_THEORY.md first for the beginner-first teaching of RAG Ingestion: Parsing, Chunking, Metadata and Storage. Then use the real service-desk-day-14/ project to trace, run, debug, and explain the concept.

Service Desk Alignment

Day 14 adds RAG Ingestion: Parsing, Chunking, Metadata and Storage to the running Service Desk. Start with storage/metadata_store.py, pipeline.py, storage/vector_store.py, storage/object_store.py, parser.py, manifest.py, then follow imports and tests to identify the actual runtime path.

Why This Topic Matters

The theory chapter explains why RAG Ingestion: Parsing, Chunking, Metadata and Storage is needed. Here the goal is evidence: identify where the capability is implemented, what observable behavior changes, and how the repository proves both success and failure behavior.

System Architecture

This is a repository surface map, not a claim that modules call each other in the displayed order. The modules are ranked by relevance to today's theory.

graph LR
    T[Day 14: RAG Ingestion - Parsing, Chunking, Metadata and Storage]
    T --> M1[storage/metadata_store.py]
    T --> M2[pipeline.py]
    T --> M3[storage/vector_store.py]
    T --> M4[storage/object_store.py]
    T --> M5[parser.py]
    T --> M6[manifest.py]
    T --> M7[deduplicator.py]
    T --> M8[chunkers/base.py]

Follow imports and tests to discover the actual runtime flow.

Repository Implementation Map

Use the real Day 14 repository, not a fabricated sample, to connect theory to implementation.

Theory concepts to locate:

Retrieval quality starts before the query
The ingestion pipeline
Parsing is document-specific
Why chunking exists

Most relevant implementation modules first:

service_desk/storage/metadata_store.py
service_desk/pipeline.py
service_desk/storage/vector_store.py
service_desk/storage/object_store.py
service_desk/parser.py
service_desk/manifest.py
service_desk/deduplicator.py
service_desk/chunkers/base.py
service_desk/chunkers/fixed.py
service_desk/chunkers/structural.py
service_desk/models.py

Follow imports/calls from the relevant module and confirm behavior in tests. Record input → mechanism → observable output/state → failure evidence.

Code Walkthrough & Mechanics

Read storage/metadata_store.py, pipeline.py, storage/vector_store.py, storage/object_store.py, parser.py, manifest.py, deduplicator.py, chunkers/base.py, chunkers/fixed.py, chunkers/structural.py with these theory sections beside you:

Retrieval quality starts before the query — locate its implementation and evidence.
The ingestion pipeline — locate its implementation and evidence.
Parsing is document-specific — locate its implementation and evidence.
Why chunking exists — locate its implementation and evidence.
Prefer semantic boundaries when possible — locate its implementation and evidence.

For each concept identify the real function/class/protocol boundary, its input/state, its observable result, and the assertion in tests/test_ingestion.py that proves the behavior. If a concept has no implementation or evidence, record that as a gap rather than inventing one.

Common Mistakes & Debugging Guidance

Use the theory—not generic timeout or .env advice—to decide what can fail today.

Failure lens: revisit Why chunking exists, Prefer semantic boundaries when possible.

Reproduce the smallest case that violates one of those expectations.
Trace the real Day 14 modules until you find the first incorrect state/output/decision.
Use tests/test_ingestion.py as executable evidence.
Add a regression test if the failure is not already represented.
Fix the smallest responsible boundary and rerun the relevant test before the full suite.

Your debugging explanation must name the topic-specific invariant that failed, not merely say “an exception occurred.”

Practical Lab Instructions

Summarize these theory ideas before opening the implementation:
- Retrieval quality starts before the query
- The ingestion pipeline
- Parsing is document-specific
- Why chunking exists
Inspect the most relevant real Day 14 modules first:
- service_desk/storage/metadata_store.py
- service_desk/pipeline.py
- service_desk/storage/vector_store.py
- service_desk/storage/object_store.py
- service_desk/parser.py
- service_desk/manifest.py
- service_desk/deduplicator.py
- service_desk/chunkers/base.py
- service_desk/chunkers/fixed.py
- service_desk/chunkers/structural.py
- service_desk/models.py
Inspect the automated evidence:
- tests/test_ingestion.py

Establish the baseline:

cd service-desk-day-14
PYTHONPATH=. pytest tests/test_ingestion.py -q

Trace one theory concept through the actual nested modules and tests.
Run one success case and record input → mechanism → observable result.
Exercise one topic-specific failure/boundary case and name the invariant that protects the system.
Re-run the relevant tests and explain theory → implementation → evidence.

Done when: another student can reproduce your trace without relying on an invented sample.

Key Takeaways

Be able to explain Retrieval quality starts before the query and point to its implementation/evidence in Day 14.
Be able to explain The ingestion pipeline and point to its implementation/evidence in Day 14.
Be able to explain Parsing is document-specific and point to its implementation/evidence in Day 14.

Knowledge Check & Scenario Questions

Concept: Using Retrieval quality starts before the query, explain the engineering problem Day 14 is solving without naming a framework as the answer.
Mechanism: How does The ingestion pipeline appear in the real project? Start from service_desk/storage/metadata_store.py and name the observable state/output/event that changes.
Failure: For Parsing is document-specific, describe one incorrect implementation or boundary condition and the evidence you would expect in tests/test_ingestion.py.
Design review: Which assumption in today's design would you verify before reusing this implementation in a different production system?

Official References

LangChain Text Splitters & Chunking: https://python.langchain.com/docs/modules/data_connection/document_transformers/
Unstructured.io Parsing Engine: https://docs.unstructured.io/

03 · EXPLAIN

Interview checkpoint

Explain one design decision from this lesson, the alternative you rejected, and the failure mode or evidence that justified your choice.