Module 1: Engineering Foundations · 3.5h

01 · UNDERSTAND

Day 05 Theory — Evaluation Harness: Stop Building by Vibes

Why evaluation appears this early

Teams often build an AI feature, try five examples manually, decide it “looks good,” and continue adding complexity. That approach breaks down quickly because LLM output is variable and subjective.

We introduce evaluation before building sophisticated agents because every later capability needs a way to answer:

Did this change make the system better, worse, or merely different?

Traditional tests versus AI evaluations

Traditional unit tests work beautifully when the expected result is deterministic.

assert normalize_priority("high") == "HIGH"

An LLM task may have several acceptable answers. Exact string equality may be the wrong measure.

That does not mean AI systems are “untestable.” It means we must choose evaluation methods that match the property we care about.

Start by defining the task

Before choosing a metric, define what success means.

For a Service Desk classifier, useful dimensions might include:

correct category,
correct urgency,
no unsupported claims,
valid structured output,
required safety behavior.

For a RAG answer later in the course, we may care about retrieval quality, citation support, faithfulness, and answer relevance.

There is no single universal “AI quality score.”

Build a representative dataset

An evaluation dataset is a set of examples that captures the work the system must handle.

A weak dataset contains only easy happy-path examples.

A stronger dataset includes:

common cases,
ambiguous wording,
malformed or incomplete requests,
rare but important cases,
adversarial inputs where relevant,
examples from real production distributions when privacy permits.

Each case should include enough expected information to judge the behavior.

Deterministic checks are still valuable

Use deterministic assertions whenever the requirement is deterministic.

Examples:

output parses against the schema,
category belongs to an allowed enum,
a forbidden tool was not called,
a citation references an allowed document,
a dangerous side effect requires approval.

Do not replace a precise deterministic assertion with an LLM judge simply because the application contains an LLM.

Semantic evaluation

Some properties require judgment.

For example:

“Did the response answer the user's question using only the supplied policy?”

Possible methods include:

human review,
task-specific heuristics,
reference-answer similarity where appropriate,
an evaluator model using a carefully designed rubric.

Every method has failure modes. Human review is expensive and inconsistent. Heuristics can miss nuance. LLM judges can be biased, unstable, or fooled.

Use multiple signals when the decision matters.

Baselines matter

A score without context is difficult to interpret.

Suppose a new prompt scores 87% on an eval. Is that good?

We need to compare it with something:

the previous prompt,
a deterministic baseline,
another model,
the current production version.

Evaluation becomes most useful when it supports comparison across controlled changes.

Avoid contaminating the eval set

If we repeatedly tune prompts against the same small evaluation set, we can overfit to it. The prompt begins to perform well on known examples without improving general behavior.

Use separate development and holdout cases when possible, and periodically refresh examples based on real failure modes.

Measure more than average accuracy

A system can look good on average while failing badly on a critical slice.

For example:

Overall classification accuracy: high
Security-ticket accuracy: poor

Slice evaluations by meaningful attributes such as category, language, risk, input length, customer segment, or failure type.

Reproducibility in a probabilistic system

Store enough metadata to understand what produced a result:

model/provider,
model version where available,
prompt version,
temperature/sampling settings,
input case identifier,
tool/retrieval configuration,
application commit.

Without this information, a regression report is difficult to investigate.

Service Desk connection

Today we turn the Service Desk from “something that seems to work” into a system with measurable behavior.

The evaluation harness becomes a permanent engineering asset. Later, when we add RAG, agents, tools, memory, security controls, and new models, we will reuse the same discipline.

The central principle is:

Define the behavior you care about, represent it with test cases and appropriate metrics, and compare changes against a baseline before trusting intuition.

02 · APPLY

Lesson Overview

This is the applied companion for Day 05. Read DAY_05_THEORY.md first for the beginner-first teaching of Evaluation Harness: Stop Building by Vibes. Then use the real service-desk-day-05/ project to trace, run, debug, and explain the concept.

Service Desk Alignment

Day 05 adds Evaluation Harness: Stop Building by Vibes to the running Service Desk. Start with router.py, classifier.py, models.py, then follow imports and tests to identify the actual runtime path.

Why This Topic Matters

The theory chapter explains why Evaluation Harness: Stop Building by Vibes is needed. Here the goal is evidence: identify where the capability is implemented, what observable behavior changes, and how the repository proves both success and failure behavior.

System Architecture

This is a repository surface map, not a claim that modules call each other in the displayed order. The modules are ranked by relevance to today's theory.

graph LR
    T[Day 05: Evaluation Harness - Stop Building by Vibes]
    T --> M1[router.py]
    T --> M2[classifier.py]
    T --> M3[models.py]

Follow imports and tests to discover the actual runtime flow.

Repository Implementation Map

Use the real Day 05 repository, not a fabricated sample, to connect theory to implementation.

Theory concepts to locate:

Why evaluation appears this early
Traditional tests versus AI evaluations
Start by defining the task
Build a representative dataset

Most relevant implementation modules first:

service_desk/router.py
service_desk/classifier.py
service_desk/models.py

Follow imports/calls from the relevant module and confirm behavior in tests. Record input → mechanism → observable output/state → failure evidence.

Code Walkthrough & Mechanics

Read router.py, classifier.py, models.py with these theory sections beside you:

Why evaluation appears this early — locate its implementation and evidence.
Traditional tests versus AI evaluations — locate its implementation and evidence.
Start by defining the task — locate its implementation and evidence.
Build a representative dataset — locate its implementation and evidence.
Deterministic checks are still valuable — locate its implementation and evidence.

For each concept identify the real function/class/protocol boundary, its input/state, its observable result, and the assertion in tests/test_eval_harness.py that proves the behavior. If a concept has no implementation or evidence, record that as a gap rather than inventing one.

Common Mistakes & Debugging Guidance

Use the theory—not generic timeout or .env advice—to decide what can fail today.

Failure lens: revisit Build a representative dataset, Deterministic checks are still valuable.

Reproduce the smallest case that violates one of those expectations.
Trace the real Day 05 modules until you find the first incorrect state/output/decision.
Use tests/test_eval_harness.py as executable evidence.
Add a regression test if the failure is not already represented.
Fix the smallest responsible boundary and rerun the relevant test before the full suite.

Your debugging explanation must name the topic-specific invariant that failed, not merely say “an exception occurred.”

Practical Lab Instructions

Summarize these theory ideas before opening the implementation:
- Why evaluation appears this early
- Traditional tests versus AI evaluations
- Start by defining the task
- Build a representative dataset
Inspect the most relevant real Day 05 modules first:
- service_desk/router.py
- service_desk/classifier.py
- service_desk/models.py
Inspect the automated evidence:
- tests/test_eval_harness.py

Establish the baseline:

cd service-desk-day-05
PYTHONPATH=. pytest tests/test_eval_harness.py -q

Trace one theory concept through the actual nested modules and tests.
Run one success case and record input → mechanism → observable result.
Exercise one topic-specific failure/boundary case and name the invariant that protects the system.
Re-run the relevant tests and explain theory → implementation → evidence.

Done when: another student can reproduce your trace without relying on an invented sample.

Key Takeaways

Be able to explain Why evaluation appears this early and point to its implementation/evidence in Day 05.
Be able to explain Traditional tests versus AI evaluations and point to its implementation/evidence in Day 05.
Be able to explain Start by defining the task and point to its implementation/evidence in Day 05.

Knowledge Check & Scenario Questions

Concept: Using Why evaluation appears this early, explain the engineering problem Day 05 is solving without naming a framework as the answer.
Mechanism: How does Traditional tests versus AI evaluations appear in the real project? Start from service_desk/router.py and name the observable state/output/event that changes.
Failure: For Start by defining the task, describe one incorrect implementation or boundary condition and the evidence you would expect in tests/test_eval_harness.py.
Design review: Which assumption in today's design would you verify before reusing this implementation in a different production system?

Official References

Ragas Quantitative RAG Evaluation: https://docs.ragas.io/
Promptfoo LLM Benchmarking: https://www.promptfoo.dev/

03 · EXPLAIN

Interview checkpoint

Explain one design decision from this lesson, the alternative you rejected, and the failure mode or evidence that justified your choice.