Day 25 Theory — Unassisted Gate 2: Timed Broken-Repository Debugging

Why debugging is a core agent-engineering skill

Production systems rarely fail in the neat order they were taught. A user may report “the agent gave the wrong answer,” while the actual defect is somewhere completely different:

• stale state was loaded,
• the wrong document was retrieved,
• a tool schema changed,
• a permission check rejected the request,
• a side effect ran twice,
• a conditional graph edge routed incorrectly,
• an exception was swallowed and replaced with a fallback.

Today is intentionally different from a normal lesson. You are given a broken repository and limited guidance. The goal is to demonstrate that you can investigate evidence, isolate the first broken contract, make a focused repair, and prove the fix.

This is much closer to real engineering work than copying a clean implementation.

Debugging is not guessing

A weak debugging loop looks like:

see error
  ↓
change several files
  ↓
run again
  ↓
see different error
  ↓
change more files

This destroys information about cause and effect.

A disciplined loop looks like:

reproduce
  ↓
observe
  ↓
form one hypothesis
  ↓
run the smallest useful experiment
  ↓
confirm / reject hypothesis
  ↓
fix responsible layer
  ↓
rerun regression evidence

The goal is not to make an error disappear. The goal is to understand why the system violated its contract.

Start with the observable failure

Write down four things before changing code:

1. Input — what exact request/test case triggers the problem?
2. Expected behavior — what should happen according to the contract?
3. Actual behavior — what happens instead?
4. Evidence — test output, exception, trace, logs, state snapshot, response payload.

Example:

Input:
"Reset password for alice"

Expected:
verify identity -> approval -> reset once -> success response

Actual:
reset tool called twice

Evidence:
tool-call trace contains two reset_password calls with same operation ID

That is far more useful than “password reset is broken.”

Debug from boundaries inward

The Service Desk now contains many layers. Trace the request through explicit boundaries:

incoming request
      ↓
validation / auth
      ↓
workflow state
      ↓
model input/output
      ↓
routing decision
      ↓
retrieval / tool call
      ↓
external result
      ↓
state update
      ↓
final response

Find the first point where actual behavior diverges from expected behavior.

Why first?

A bad value created early can produce many downstream symptoms. Fixing the last symptom may leave the real cause untouched.

Example: wrong final answer

Suppose the agent answers with an outdated VPN policy.

Possible investigation:

Final answer wrong
   |
   v
Was correct source in model context?
   |
   +-- NO --> retrieval/index/filter problem
   |
   +-- YES --> did model use source correctly?
                 |
                 +-- NO --> generation/prompt/eval problem

Do not immediately rewrite the prompt before checking retrieval evidence.

Example: tool never executes

Trace:

Did model/request produce tool intent?
   |
   +-- no -> model/tool-description issue
   |
   +-- yes
        |
        v
Did argument validation pass?
   |
   +-- no -> schema/input problem
   |
   +-- yes
        |
        v
Did authorization allow it?
   |
   +-- no -> policy/identity problem
   |
   +-- yes
        |
        v
Did dispatcher resolve tool name?

One user-visible symptom can originate from several distinct layers.

Reproduce before fixing

A reproducible failure is one of your strongest debugging assets.

Prefer:

pytest tests/test_specific_failure.py::test_duplicate_reset -q

or the smallest command/request that demonstrates the defect.

A minimal reproduction gives you:

• faster iteration,
• clearer evidence,
• a future regression test,
• less noise from unrelated components.

If the failure is nondeterministic, record the conditions under which it appears: model version, seed where applicable, concurrency, request order, dependency state and trace IDs.

Read the traceback from the bottom—but understand the chain

Python tracebacks show the call path leading to an exception.

Start from the final exception type/message to understand the immediate failure, then move upward to identify which application boundary supplied the bad value.

For example:

ValidationError
  caused while constructing ToolArguments
  called by tool_dispatch()
  called by agent_loop()

The exception may occur inside Pydantic, but the root cause could be an invalid tool payload produced earlier.

Do not “fix Pydantic” when the contract violation belongs to the caller.

State debugging

Stateful agents introduce bugs that ordinary stateless scripts do not.

Inspect:

• state before a node/turn,
• partial update returned,
• reducer/merge behavior,
• state after merge,
• persisted checkpoint if relevant,
• state loaded after resume.

A useful debugging table is:

Step	Important input state	Update	Important output state
classify	query	classification	category=security
decide	classification	tool_call	pending reset
approval	proposal	approved	approval=true
tool	approval	result	reset completed

This makes silent state corruption visible.

Debugging nondeterministic model behavior

Do not expect every LLM call to reproduce exactly.

Separate deterministic infrastructure from probabilistic behavior.

First verify things you can prove exactly:

• correct prompt/context assembled,
• correct tool schemas supplied,
• expected model/provider selected,
• response parsed successfully,
• policy checks executed,
• tool calls bounded,
• state transitions valid.

Then evaluate model behavior across representative cases rather than debugging one generation forever.

One hypothesis at a time

A good hypothesis is specific and falsifiable.

Weak:

“Something is wrong with LangGraph.”

Better:

“route_after_tool sees the previous tool_results length because the node update is not being merged before routing.”

Now you can inspect one state transition and prove or reject it.

Use the smallest change that tests the hypothesis

Suppose you suspect a metadata filter excludes the correct RAG document.

Do not redesign the entire retriever.

First log/inspect:

query
active filters
candidate IDs before filter
candidate IDs after filter

Then change only the suspected filter and rerun the same eval case.

Small experiments preserve causal information.

Do not hide failures with broad exception handling

This pattern makes debugging harder:

try:
    run_everything()
except Exception:
    return "Something went wrong"

User-facing errors may need sanitization, but internal diagnostics should preserve structured failure evidence.

Catch exceptions where you can add useful context or perform a meaningful recovery.

Do not turn every bug into a generic fallback response.

Know when a timeout is ambiguous

For side-effecting operations, a timeout may mean:

request never reached server

or:

server completed action but response was lost

That distinction is why idempotency and operation IDs matter during debugging.

If you see a duplicate effect after retry, do not conclude “the retry library is broken.” Investigate whether the operation was safe to repeat.

Read tests as contracts, not obstacles

A failing test tells you what behavior the repository expects.

Ask:

• What invariant is this test protecting?
• Is the test itself still aligned with the current specification?
• Does the implementation violate the contract, or is the test stale?

This matters for fast-moving protocols such as MCP and A2A. An old test can preserve an obsolete API assumption.

Never change production behavior merely to make a stale test green without checking the authoritative contract.

Time pressure changes prioritization, not rigor

This is a timed gate, but the right response is not random speed.

Prioritize:

1. reproduce blocker,
2. identify likely boundary,
3. inspect high-signal evidence,
4. make minimal repair,
5. rerun focused test,
6. rerun affected regression suite,
7. document root cause.

Do not spend the first half of the exercise polishing unrelated code.

Communicate your debugging reasoning

In a real incident or interview, your reasoning matters.

A strong explanation sounds like:

“The final answer was wrong because the relevant source disappeared before generation. I compared retrieval candidates before and after metadata filtering and found the tenant filter was using the user's display name rather than tenant ID. I corrected that mapping and added a regression case proving the correct document remains in the authorized result set.”

That demonstrates diagnosis, not luck.

Service Desk connection

The Service Desk now has enough moving parts that debugging strategy is part of the architecture.

Today you are proving you can navigate:

typed input
+ async/API boundaries
+ model behavior
+ retrieval
+ tools
+ graph state
+ human approval
+ persistence

without rewriting everything when one layer breaks.

The principle is:

Reproduce the failure, trace data and control flow from boundaries inward, identify the first broken contract, test one hypothesis at a time, and turn the verified fix into regression evidence.