Day 33 Theory — Tracing, OpenTelemetry and Reproducing Nondeterministic Bugs

Lesson Overview

This is the applied companion for Day 33. Read DAY_33_THEORY.md first for the beginner-first teaching of Tracing and Reproducing Nondeterministic Bugs. Then use the real service-desk-day-33/ project to trace, run, debug, and explain the concept.

Service Desk Alignment

Day 33 adds Tracing and Reproducing Nondeterministic Bugs to the running Service Desk. Start with tracing/tracer.py, tracing/replay_engine.py, rag/tenant_rag.py, agent/mock_model.py, tools/ticket_tools.py, agent/service_desk_agent.py, then follow imports and tests to identify the actual runtime path.

Why This Topic Matters

The theory chapter explains why Tracing and Reproducing Nondeterministic Bugs is needed. Here the goal is evidence: identify where the capability is implemented, what observable behavior changes, and how the repository proves both success and failure behavior.

System Architecture

graph TD
    A[Trace: agent_step] --> B[Span: llm_call]
    A --> C[Span: tool_execution]
    B --> D[Attribute: tokens_used=142]
    C --> E[Attribute: tool_status=success]

Worked Code Example: OpenTelemetry Tracing

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter

# Initialize OpenTelemetry Tracer
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("service_desk.agent")

def run_traced_agent_step(ticket_id: str, prompt: str):
    with tracer.start_as_current_span("agent_step") as span:
        span.set_attribute("ticket_id", ticket_id)
        span.set_attribute("prompt_length", len(prompt))
        
        with tracer.start_as_current_span("llm_call") as llm_span:
            llm_span.set_attribute("model", "gpt-4o")
            # Simulate LLM call
            response = "Password reset instructions sent."
            llm_span.set_attribute("tokens_used", 142)
            
        return response

run_traced_agent_step("TCK-991", "Reset my password")

Code Walkthrough & Mechanics

Read tracing/tracer.py, tracing/replay_engine.py, rag/tenant_rag.py, agent/mock_model.py, tools/ticket_tools.py, agent/service_desk_agent.py, privacy/telemetry_scrubber.py, models.py with these theory sections beside you:

• Why logs are not enough — locate its implementation and evidence.
• Trace, span and context — locate its implementation and evidence.
• OpenTelemetry — locate its implementation and evidence.
• Useful span attributes — locate its implementation and evidence.
• Reproducing nondeterministic behavior — locate its implementation and evidence.

For each concept identify the real function/class/protocol boundary, its input/state, its observable result, and the assertion in tests/test_tracing.py that proves the behavior. If a concept has no implementation or evidence, record that as a gap rather than inventing one.

Common Mistakes & Debugging Guidance

Use the theory—not generic timeout or .env advice—to decide what can fail today.

Failure lens: revisit Useful span attributes, Reproducing nondeterministic behavior.

• Reproduce the smallest case that violates one of those expectations.
• Trace the real Day 33 modules until you find the first incorrect state/output/decision.
• Use tests/test_tracing.py as executable evidence.
• Add a regression test if the failure is not already represented.
• Fix the smallest responsible boundary and rerun the relevant test before the full suite.

Your debugging explanation must name the topic-specific invariant that failed, not merely say “an exception occurred.”

Practical Lab Instructions

1. Summarize these theory ideas before opening the implementation:
   • Why logs are not enough
   • Trace, span and context
   • OpenTelemetry
   • Useful span attributes
2. Inspect the most relevant real Day 33 modules first:
   • service_desk/tracing/tracer.py
   • service_desk/tracing/replay_engine.py
   • service_desk/rag/tenant_rag.py
   • service_desk/agent/mock_model.py
   • service_desk/tools/ticket_tools.py
   • service_desk/agent/service_desk_agent.py
   • service_desk/privacy/telemetry_scrubber.py
   • service_desk/models.py
3. Inspect the automated evidence:
   • tests/test_tracing.py
4. Establish the baseline:

   cd service-desk-day-33
   PYTHONPATH=. pytest tests/test_tracing.py -q
   

5. Trace one theory concept through the actual nested modules and tests.
6. Run one success case and record input → mechanism → observable result.
7. Exercise one topic-specific failure/boundary case and name the invariant that protects the system.
8. Re-run the relevant tests and explain theory → implementation → evidence.

Done when: another student can reproduce your trace without relying on an invented sample.

Key Takeaways

1. Be able to explain Why logs are not enough and point to its implementation/evidence in Day 33.
2. Be able to explain Trace, span and context and point to its implementation/evidence in Day 33.
3. Be able to explain OpenTelemetry and point to its implementation/evidence in Day 33.

Knowledge Check & Scenario Questions

1. Concept: Using Why logs are not enough, explain the engineering problem Day 33 is solving without naming a framework as the answer.
2. Mechanism: How does Trace, span and context appear in the real project? Start from service_desk/tracing/tracer.py and name the observable state/output/event that changes.
3. Failure: For OpenTelemetry, describe one incorrect implementation or boundary condition and the evidence you would expect in tests/test_tracing.py.
4. Design review: Which assumption in today's design would you verify before reusing this implementation in a different production system?

Official References

• OpenTelemetry Python Official Documentation: https://opentelemetry.io/docs/languages/python/
• W3C Trace Context Specification: https://www.w3.org/TR/trace-context/