Day 35 Theory — Regression Engineering: Prompt/Model Versions, Drift and CI Gates

Lesson Overview

This is the applied companion for Day 35. Read DAY_35_THEORY.md first for the beginner-first teaching of Regression Engineering: Prompt/Model Versions, Drift and CI Gates. Then use the real service-desk-day-35/ project to trace, run, debug, and explain the concept.

Service Desk Alignment

Day 35 adds Regression Engineering: Prompt/Model Versions, Drift and CI Gates to the running Service Desk. Start with eval/golden_set.py, eval/gate.py, eval/runner.py, eval/reporter.py, eval/evaluator.py, models.py, then follow imports and tests to identify the actual runtime path.

Why This Topic Matters

The theory chapter explains why Regression Engineering: Prompt/Model Versions, Drift and CI Gates is needed. Here the goal is evidence: identify where the capability is implemented, what observable behavior changes, and how the repository proves both success and failure behavior.

System Architecture

This is a repository surface map, not a claim that modules call each other in the displayed order. The modules are ranked by relevance to today's theory.

graph LR
    T[Day 35: Regression Engineering - Prompt/Model Versions, Drift and CI Gates]
    T --> M1[eval/golden_set.py]
    T --> M2[eval/gate.py]
    T --> M3[eval/runner.py]
    T --> M4[eval/reporter.py]
    T --> M5[eval/evaluator.py]
    T --> M6[models.py]
    T --> M7[agent/core.py]
    T --> M8[agent/tools.py]

Follow imports and tests to discover the actual runtime flow.

Repository Implementation Map

Use the real Day 35 repository, not a fabricated sample, to connect theory to implementation.

Theory concepts to locate:

• AI systems regress differently
• Version the behavior inputs
• Evaluation in CI
• Golden sets and holdouts

Most relevant implementation modules first:

• service_desk/eval/golden_set.py
• service_desk/eval/gate.py
• service_desk/eval/runner.py
• service_desk/eval/reporter.py
• service_desk/eval/evaluator.py
• service_desk/models.py
• service_desk/agent/core.py
• service_desk/agent/tools.py
• service_desk/config.py

Follow imports/calls from the relevant module and confirm behavior in tests. Record input → mechanism → observable output/state → failure evidence.

Code Walkthrough & Mechanics

Read eval/golden_set.py, eval/gate.py, eval/runner.py, eval/reporter.py, eval/evaluator.py, models.py, agent/core.py, agent/tools.py, config.py with these theory sections beside you:

• AI systems regress differently — locate its implementation and evidence.
• Version the behavior inputs — locate its implementation and evidence.
• Evaluation in CI — locate its implementation and evidence.
• Golden sets and holdouts — locate its implementation and evidence.
• Drift — locate its implementation and evidence.

For each concept identify the real function/class/protocol boundary, its input/state, its observable result, and the assertion in tests/test_golden_set.py that proves the behavior. If a concept has no implementation or evidence, record that as a gap rather than inventing one.

Common Mistakes & Debugging Guidance

Use the theory—not generic timeout or .env advice—to decide what can fail today.

Failure lens: revisit Golden sets and holdouts, Drift.

• Reproduce the smallest case that violates one of those expectations.
• Trace the real Day 35 modules until you find the first incorrect state/output/decision.
• Use tests/test_golden_set.py as executable evidence.
• Add a regression test if the failure is not already represented.
• Fix the smallest responsible boundary and rerun the relevant test before the full suite.

Your debugging explanation must name the topic-specific invariant that failed, not merely say “an exception occurred.”

Practical Lab Instructions

1. Summarize these theory ideas before opening the implementation:
   • AI systems regress differently
   • Version the behavior inputs
   • Evaluation in CI
   • Golden sets and holdouts
2. Inspect the most relevant real Day 35 modules first:
   • service_desk/eval/golden_set.py
   • service_desk/eval/gate.py
   • service_desk/eval/runner.py
   • service_desk/eval/reporter.py
   • service_desk/eval/evaluator.py
   • service_desk/models.py
   • service_desk/agent/core.py
   • service_desk/agent/tools.py
   • service_desk/config.py
3. Inspect the automated evidence:
   • tests/test_golden_set.py
4. Establish the baseline:

   cd service-desk-day-35
   PYTHONPATH=. pytest tests/test_golden_set.py -q
   

5. Trace one theory concept through the actual nested modules and tests.
6. Run one success case and record input → mechanism → observable result.
7. Exercise one topic-specific failure/boundary case and name the invariant that protects the system.
8. Re-run the relevant tests and explain theory → implementation → evidence.

Done when: another student can reproduce your trace without relying on an invented sample.

Key Takeaways

1. Be able to explain AI systems regress differently and point to its implementation/evidence in Day 35.
2. Be able to explain Version the behavior inputs and point to its implementation/evidence in Day 35.
3. Be able to explain Evaluation in CI and point to its implementation/evidence in Day 35.

Knowledge Check & Scenario Questions

1. Concept: Using AI systems regress differently, explain the engineering problem Day 35 is solving without naming a framework as the answer.
2. Mechanism: How does Version the behavior inputs appear in the real project? Start from service_desk/eval/golden_set.py and name the observable state/output/event that changes.
3. Failure: For Evaluation in CI, describe one incorrect implementation or boundary condition and the evidence you would expect in tests/test_golden_set.py.
4. Design review: Which assumption in today's design would you verify before reusing this implementation in a different production system?

Official References

• Promptfoo CI/CD Evaluation Gates: https://www.promptfoo.dev/docs/ci/
• MLflow Model Registry & Versioning: https://mlflow.org/docs/latest/model-registry.html