Skip to main content
>_ supraj.dev

Module 6: Evaluation & Production Ops · 2.25h

01 · UNDERSTAND

Day 36 Theory — Cost and Performance Engineering

Cost is produced by architecture decisions

LLM price is only one part of total system cost.

A production agent may pay for:

  • input/output tokens,
  • embeddings,
  • reranking,
  • vector storage,
  • API calls,
  • compute,
  • tracing/log retention,
  • human review,
  • retries and failed runs.

Optimize the task, not one invoice line.

Latency decomposition

Measure where time is spent:

request
  ├─ queue
  ├─ retrieval
  ├─ model first-token latency
  ├─ generation
  ├─ tools
  └─ persistence

Without decomposition, teams may optimize the wrong component.

Token economics

Input tokens include instructions, history, retrieved context and tool schemas. Output tokens depend on generated length.

Strategies include:

  • compacting context,
  • retrieving fewer but better chunks,
  • prompt caching where supported,
  • avoiding repeated tool descriptions,
  • limiting unnecessary verbosity.

Model routing

Not every task requires the most capable expensive model.

A router can send simple classification to a smaller model and complex reasoning to a stronger one—if evaluation proves quality remains acceptable.

Routing itself adds complexity and must be evaluated.

Batch and cache carefully

Caching can reduce repeated work, but cache keys must include every input that affects correctness and authorization.

Never serve one tenant's cached sensitive result to another tenant.

Cost per successful task

A useful metric is not simply cost per API call.

Measure cost per successful user task or per resolved ticket. A cheap model that fails often may be more expensive overall.

Service Desk connection

Today we make the Service Desk measurable in both latency and money while protecting quality.

The principle is:

Optimize end-to-end task economics: measure where time and money go, reduce wasted context/retries, and change models or caching only when evals confirm the product remains correct.

02 · APPLY

Lesson Overview

This is the applied companion for Day 36. Read DAY_36_THEORY.md first for the beginner-first teaching of Cost and Performance Engineering. Then use the real service-desk-day-36/ project to trace, run, debug, and explain the concept.

Service Desk Alignment

Day 36 adds Cost and Performance Engineering to the running Service Desk. Start with cost/ledger.py, cost/budget.py, cost/pricing.py, routing/router.py, performance/metrics.py, performance/profiler.py, then follow imports and tests to identify the actual runtime path.

Why This Topic Matters

The theory chapter explains why Cost and Performance Engineering is needed. Here the goal is evidence: identify where the capability is implemented, what observable behavior changes, and how the repository proves both success and failure behavior.

System Architecture

This is a repository surface map, not a claim that modules call each other in the displayed order. The modules are ranked by relevance to today's theory.

graph LR
    T[Day 36: Cost and Performance Engineering]
    T --> M1[cost/ledger.py]
    T --> M2[cost/budget.py]
    T --> M3[cost/pricing.py]
    T --> M4[routing/router.py]
    T --> M5[performance/metrics.py]
    T --> M6[performance/profiler.py]
    T --> M7[performance/parallel_executor.py]
    T --> M8[models.py]

Follow imports and tests to discover the actual runtime flow.

Repository Implementation Map

Use the real Day 36 repository, not a fabricated sample, to connect theory to implementation.

Theory concepts to locate:

  • Cost is produced by architecture decisions
  • Latency decomposition
  • Token economics
  • Model routing

Most relevant implementation modules first:

  • service_desk/cost/ledger.py
  • service_desk/cost/budget.py
  • service_desk/cost/pricing.py
  • service_desk/routing/router.py
  • service_desk/performance/metrics.py
  • service_desk/performance/profiler.py
  • service_desk/performance/parallel_executor.py
  • service_desk/models.py
  • service_desk/agent/mock_engine.py
  • service_desk/optimization/caching.py
  • service_desk/eval/benchmark_suite.py
  • service_desk/agent/service_desk_agent.py
  • service_desk/optimization/context_reduction.py

Follow imports/calls from the relevant module and confirm behavior in tests. Record input → mechanism → observable output/state → failure evidence.

Code Walkthrough & Mechanics

Read cost/ledger.py, cost/budget.py, cost/pricing.py, routing/router.py, performance/metrics.py, performance/profiler.py, performance/parallel_executor.py, models.py, agent/mock_engine.py, optimization/caching.py with these theory sections beside you:

  • Cost is produced by architecture decisions — locate its implementation and evidence.
  • Latency decomposition — locate its implementation and evidence.
  • Token economics — locate its implementation and evidence.
  • Model routing — locate its implementation and evidence.
  • Batch and cache carefully — locate its implementation and evidence.

For each concept identify the real function/class/protocol boundary, its input/state, its observable result, and the assertion in tests/test_cost_performance.py that proves the behavior. If a concept has no implementation or evidence, record that as a gap rather than inventing one.

Common Mistakes & Debugging Guidance

Use the theory—not generic timeout or .env advice—to decide what can fail today.

Failure lens: revisit Model routing, Batch and cache carefully.

  • Reproduce the smallest case that violates one of those expectations.
  • Trace the real Day 36 modules until you find the first incorrect state/output/decision.
  • Use tests/test_cost_performance.py as executable evidence.
  • Add a regression test if the failure is not already represented.
  • Fix the smallest responsible boundary and rerun the relevant test before the full suite.

Your debugging explanation must name the topic-specific invariant that failed, not merely say “an exception occurred.”

Practical Lab Instructions

  1. Summarize these theory ideas before opening the implementation:
    • Cost is produced by architecture decisions
    • Latency decomposition
    • Token economics
    • Model routing
  2. Inspect the most relevant real Day 36 modules first:
    • service_desk/cost/ledger.py
    • service_desk/cost/budget.py
    • service_desk/cost/pricing.py
    • service_desk/routing/router.py
    • service_desk/performance/metrics.py
    • service_desk/performance/profiler.py
    • service_desk/performance/parallel_executor.py
    • service_desk/models.py
    • service_desk/agent/mock_engine.py
    • service_desk/optimization/caching.py
    • service_desk/eval/benchmark_suite.py
    • service_desk/agent/service_desk_agent.py
  3. Inspect the automated evidence:
    • tests/test_cost_performance.py
  4. Establish the baseline:
    cd service-desk-day-36
PYTHONPATH=. pytest tests/test_cost_performance.py -q
  5. Trace one theory concept through the actual nested modules and tests.
  6. Run one success case and record input → mechanism → observable result.
  7. Exercise one topic-specific failure/boundary case and name the invariant that protects the system.
  8. Re-run the relevant tests and explain theory → implementation → evidence.

Done when: another student can reproduce your trace without relying on an invented sample.

Key Takeaways

  1. Be able to explain Cost is produced by architecture decisions and point to its implementation/evidence in Day 36.
  2. Be able to explain Latency decomposition and point to its implementation/evidence in Day 36.
  3. Be able to explain Token economics and point to its implementation/evidence in Day 36.

Knowledge Check & Scenario Questions

  1. Concept: Using Cost is produced by architecture decisions, explain the engineering problem Day 36 is solving without naming a framework as the answer.
  2. Mechanism: How does Latency decomposition appear in the real project? Start from service_desk/cost/ledger.py and name the observable state/output/event that changes.
  3. Failure: For Token economics, describe one incorrect implementation or boundary condition and the evidence you would expect in tests/test_cost_performance.py.
  4. Design review: Which assumption in today's design would you verify before reusing this implementation in a different production system?

Official References

03 · EXPLAIN

Interview checkpoint

Explain one design decision from this lesson, the alternative you rejected, and the failure mode or evidence that justified your choice.