Day 11 Theory — Failure Engineering: Timeouts, Duplicates and Loop Control

Lesson Overview

This is the applied companion for Day 11. Read DAY_11_THEORY.md first for the beginner-first teaching of Failure Engineering: Timeouts, Duplicates and Loop Control. Then use the real service-desk-day-11/ project to trace, run, debug, and explain the concept.

Service Desk Alignment

Day 11 adds Failure Engineering: Timeouts, Duplicates and Loop Control to the running Service Desk. Start with failures.py, app.py, tools.py, agent.py, suppressor.py, kill_switch.py, then follow imports and tests to identify the actual runtime path.

Why This Topic Matters

The theory chapter explains why Failure Engineering: Timeouts, Duplicates and Loop Control is needed. Here the goal is evidence: identify where the capability is implemented, what observable behavior changes, and how the repository proves both success and failure behavior.

System Architecture

This is a repository surface map, not a claim that modules call each other in the displayed order. The modules are ranked by relevance to today's theory.

graph LR
    T[Day 11: Failure Engineering - Timeouts, Duplicates and Loop Control]
    T --> M1[failures.py]
    T --> M2[app.py]
    T --> M3[tools.py]
    T --> M4[agent.py]
    T --> M5[suppressor.py]
    T --> M6[kill_switch.py]
    T --> M7[idempotency.py]
    T --> M8[circuit_breaker.py]

Follow imports and tests to discover the actual runtime flow.

Repository Implementation Map

Use the real Day 11 repository, not a fabricated sample, to connect theory to implementation.

Theory concepts to locate:

• Why the happy-path agent is not enough
• Failure domains
• Retry the smallest safe unit
• At-least-once effects

Most relevant implementation modules first:

• service_desk/failures.py
• service_desk/app.py
• service_desk/tools.py
• service_desk/agent.py
• service_desk/suppressor.py
• service_desk/kill_switch.py
• service_desk/idempotency.py
• service_desk/circuit_breaker.py
• service_desk/models.py

Follow imports/calls from the relevant module and confirm behavior in tests. Record input → mechanism → observable output/state → failure evidence.

Code Walkthrough & Mechanics

Read failures.py, app.py, tools.py, agent.py, suppressor.py, kill_switch.py, idempotency.py, circuit_breaker.py, models.py with these theory sections beside you:

• Why the happy-path agent is not enough — locate its implementation and evidence.
• Failure domains — locate its implementation and evidence.
• Retry the smallest safe unit — locate its implementation and evidence.
• At-least-once effects — locate its implementation and evidence.
• Timeout budgets across a workflow — locate its implementation and evidence.

For each concept identify the real function/class/protocol boundary, its input/state, its observable result, and the assertion in tests/test_failure_engineering.py that proves the behavior. If a concept has no implementation or evidence, record that as a gap rather than inventing one.

Common Mistakes & Debugging Guidance

Use the theory—not generic timeout or .env advice—to decide what can fail today.

Failure lens: revisit At-least-once effects, Timeout budgets across a workflow.

• Reproduce the smallest case that violates one of those expectations.
• Trace the real Day 11 modules until you find the first incorrect state/output/decision.
• Use tests/test_failure_engineering.py as executable evidence.
• Add a regression test if the failure is not already represented.
• Fix the smallest responsible boundary and rerun the relevant test before the full suite.

Your debugging explanation must name the topic-specific invariant that failed, not merely say “an exception occurred.”

Practical Lab Instructions

1. Summarize these theory ideas before opening the implementation:
   • Why the happy-path agent is not enough
   • Failure domains
   • Retry the smallest safe unit
   • At-least-once effects
2. Inspect the most relevant real Day 11 modules first:
   • service_desk/failures.py
   • service_desk/app.py
   • service_desk/tools.py
   • service_desk/agent.py
   • service_desk/suppressor.py
   • service_desk/kill_switch.py
   • service_desk/idempotency.py
   • service_desk/circuit_breaker.py
   • service_desk/models.py
3. Inspect the automated evidence:
   • tests/test_failure_engineering.py
4. Establish the baseline:

   cd service-desk-day-11
   PYTHONPATH=. pytest tests/test_failure_engineering.py -q
   

5. Trace one theory concept through the actual nested modules and tests.
6. Run one success case and record input → mechanism → observable result.
7. Exercise one topic-specific failure/boundary case and name the invariant that protects the system.
8. Re-run the relevant tests and explain theory → implementation → evidence.

Done when: another student can reproduce your trace without relying on an invented sample.

Key Takeaways

1. Be able to explain Why the happy-path agent is not enough and point to its implementation/evidence in Day 11.
2. Be able to explain Failure domains and point to its implementation/evidence in Day 11.
3. Be able to explain Retry the smallest safe unit and point to its implementation/evidence in Day 11.

Knowledge Check & Scenario Questions

1. Concept: Using Why the happy-path agent is not enough, explain the engineering problem Day 11 is solving without naming a framework as the answer.
2. Mechanism: How does Failure domains appear in the real project? Start from service_desk/failures.py and name the observable state/output/event that changes.
3. Failure: For Retry the smallest safe unit, describe one incorrect implementation or boundary condition and the evidence you would expect in tests/test_failure_engineering.py.
4. Design review: Which assumption in today's design would you verify before reusing this implementation in a different production system?

Official References

• Google SRE Book - Cascading Failures & Timeouts: https://sre.google/sre-book/addressing-cascading-failures/
• Python asyncio Timeout Utilities: https://docs.python.org/3/library/asyncio-task.html#timeouts