Day 37 Theory — Load, Capacity and On-Call Thinking

Lesson Overview

This is the applied companion for Day 37. Read DAY_37_THEORY.md first for the beginner-first teaching of Load, Capacity and On-Call Thinking. Then use the real service-desk-day-37/ project to trace, run, debug, and explain the concept.

Service Desk Alignment

Day 37 adds Load, Capacity and On-Call Thinking to the running Service Desk. Start with load_test/scenarios.py, load_test/generator.py, capacity/rate_limiter.py, capacity/queue_manager.py, capacity/connection_pool.py, models.py, then follow imports and tests to identify the actual runtime path.

Why This Topic Matters

The theory chapter explains why Load, Capacity and On-Call Thinking is needed. Here the goal is evidence: identify where the capability is implemented, what observable behavior changes, and how the repository proves both success and failure behavior.

System Architecture

This is a repository surface map, not a claim that modules call each other in the displayed order. The modules are ranked by relevance to today's theory.

graph LR
    T[Day 37: Load, Capacity and On-Call Thinking]
    T --> M1[load_test/scenarios.py]
    T --> M2[load_test/generator.py]
    T --> M3[capacity/rate_limiter.py]
    T --> M4[capacity/queue_manager.py]
    T --> M5[capacity/connection_pool.py]
    T --> M6[models.py]
    T --> M7[agent/orchestrator.py]
    T --> M8[monitoring/alerting.py]

Follow imports and tests to discover the actual runtime flow.

Repository Implementation Map

Use the real Day 37 repository, not a fabricated sample, to connect theory to implementation.

Theory concepts to locate:

• A system that works for one request may fail under load
• Capacity model
• Concurrency limits
• Load testing

Most relevant implementation modules first:

• service_desk/load_test/scenarios.py
• service_desk/load_test/generator.py
• service_desk/capacity/rate_limiter.py
• service_desk/capacity/queue_manager.py
• service_desk/capacity/connection_pool.py
• service_desk/models.py
• service_desk/agent/orchestrator.py
• service_desk/monitoring/alerting.py
• service_desk/operations/kill_switch.py
• service_desk/monitoring/slo_monitor.py

Follow imports/calls from the relevant module and confirm behavior in tests. Record input → mechanism → observable output/state → failure evidence.

Code Walkthrough & Mechanics

Read load_test/scenarios.py, load_test/generator.py, capacity/rate_limiter.py, capacity/queue_manager.py, capacity/connection_pool.py, models.py, agent/orchestrator.py, monitoring/alerting.py, operations/kill_switch.py, monitoring/slo_monitor.py with these theory sections beside you:

• A system that works for one request may fail under load — locate its implementation and evidence.
• Capacity model — locate its implementation and evidence.
• Concurrency limits — locate its implementation and evidence.
• Load testing — locate its implementation and evidence.
• On-call thinking — locate its implementation and evidence.

For each concept identify the real function/class/protocol boundary, its input/state, its observable result, and the assertion in tests/test_load_slo.py that proves the behavior. If a concept has no implementation or evidence, record that as a gap rather than inventing one.

Common Mistakes & Debugging Guidance

Use the theory—not generic timeout or .env advice—to decide what can fail today.

Failure lens: revisit Load testing, On-call thinking.

• Reproduce the smallest case that violates one of those expectations.
• Trace the real Day 37 modules until you find the first incorrect state/output/decision.
• Use tests/test_load_slo.py as executable evidence.
• Add a regression test if the failure is not already represented.
• Fix the smallest responsible boundary and rerun the relevant test before the full suite.

Your debugging explanation must name the topic-specific invariant that failed, not merely say “an exception occurred.”

Practical Lab Instructions

1. Summarize these theory ideas before opening the implementation:
   • A system that works for one request may fail under load
   • Capacity model
   • Concurrency limits
   • Load testing
2. Inspect the most relevant real Day 37 modules first:
   • service_desk/load_test/scenarios.py
   • service_desk/load_test/generator.py
   • service_desk/capacity/rate_limiter.py
   • service_desk/capacity/queue_manager.py
   • service_desk/capacity/connection_pool.py
   • service_desk/models.py
   • service_desk/agent/orchestrator.py
   • service_desk/monitoring/alerting.py
   • service_desk/operations/kill_switch.py
   • service_desk/monitoring/slo_monitor.py
3. Inspect the automated evidence:
   • tests/test_load_slo.py
4. Establish the baseline:

   cd service-desk-day-37
   PYTHONPATH=. pytest tests/test_load_slo.py -q
   

5. Trace one theory concept through the actual nested modules and tests.
6. Run one success case and record input → mechanism → observable result.
7. Exercise one topic-specific failure/boundary case and name the invariant that protects the system.
8. Re-run the relevant tests and explain theory → implementation → evidence.

Done when: another student can reproduce your trace without relying on an invented sample.

Key Takeaways

1. Be able to explain A system that works for one request may fail under load and point to its implementation/evidence in Day 37.
2. Be able to explain Capacity model and point to its implementation/evidence in Day 37.
3. Be able to explain Concurrency limits and point to its implementation/evidence in Day 37.

Knowledge Check & Scenario Questions

1. Concept: Using A system that works for one request may fail under load, explain the engineering problem Day 37 is solving without naming a framework as the answer.
2. Mechanism: How does Capacity model appear in the real project? Start from service_desk/load_test/scenarios.py and name the observable state/output/event that changes.
3. Failure: For Concurrency limits, describe one incorrect implementation or boundary condition and the evidence you would expect in tests/test_load_slo.py.
4. Design review: Which assumption in today's design would you verify before reusing this implementation in a different production system?

Official References

• Google SRE Book - Capacity Planning & Load Testing: https://sre.google/sre-book/capacity-planning/
• Locust Load Testing Engine: https://locust.io/