Module 6: Evaluation & Production Ops · 6.75h
01 · UNDERSTAND
Day 39 Theory — OpenAI Agents SDK as a High-Level Contrast
Why learn a framework only after building the loop ourselves
Earlier in the course we built the agent loop from first principles:
model decides -> tool executes -> observation recorded -> model decides again
We also added termination, state, handoffs, guardrails, tracing and persistence concerns ourselves.
That sequence matters. If we had started with a high-level SDK, the framework might have looked magical. Now we can judge what the SDK actually abstracts for us.
The goal today is not to prove that one framework is “best.” The goal is to map concepts we already understand onto the OpenAI Agents SDK and identify:
- what boilerplate disappears,
- what runtime behavior the SDK owns,
- what responsibilities remain ours.
The Agent abstraction
An Agent is a configured runtime participant. It can include things such as:
- name,
- instructions,
- tools,
- handoffs,
- model/runtime configuration,
- guardrails depending on the SDK surface being used.
Think of it as a declared behavior/configuration object—not a separate autonomous process living on its own.
A minimal example looks like:
from agents import Agent
triage_agent = Agent(
name="Triage",
instructions="Classify the support request and route it safely.",
)
Creating the object does not itself run an agent workflow.
Runner owns execution
The Runner executes the workflow.
The current Python SDK provides three important entry points:
Runner.run(...)— asynchronous execution returning a run result,Runner.run_sync(...)— synchronous helper,Runner.run_streamed(...)— asynchronous streamed execution returning a streaming result.
Example:
from agents import Agent, Runner
agent = Agent(
name="Service Desk",
instructions="Help the user with IT support requests.",
)
result = await Runner.run(agent, "My VPN certificate expired")
print(result.final_output)
Map that back to our own loop:
our custom runtime Agents SDK
------------------ ----------
while/turn loop -> Runner
agent configuration -> Agent
runtime result -> RunResult
streamed runtime events -> RunResultStreaming
The abstraction is useful because it packages runtime behavior we previously wrote manually.
A run is still a loop
A high-level SDK does not eliminate the underlying reasoning cycle.
Conceptually, the runner still has to coordinate behavior such as:
input
|
v
current agent
|
v
model call
|\
| \-- final output --------------------> stop
|
+---- tool request -> execute -> result --+
| |
+---- handoff ------> new active agent -----+
|
v
next turn
Understanding this is what lets you debug SDK behavior instead of treating the framework as a black box.
Tools: model request, runtime execution
The rule from Day 09 still applies.
The model can request a tool. The runtime executes the actual Python function or integration.
That means the application still owns:
- argument validation,
- authorization,
- credentials,
- idempotency,
- timeouts,
- side-effect safety.
Using an SDK tool decorator does not make a destructive operation safe.
Handoffs transfer active control
A handoff is different from simply calling another specialist as a bounded tool.
With a handoff, another configured agent becomes the active participant responsible for subsequent turns.
Example mental model:
Triage Agent
|
| handoff
v
Security Agent
|
v
continues workflow
This can make responsibility clear, but it introduces new control-flow risks.
Circular handoffs
Suppose:
Triage -> Security -> Triage -> Security -> ...
If the runtime permits repeated delegation and the application has no meaningful termination design, the workflow can waste turns and cost.
The same principle from our custom loop still applies: delegation must be bounded.
Sessions are explicit state management
An Agent object does not magically mean that every future request remembers previous conversation state.
The SDK supports conversation-management approaches including explicit sessions and server-managed conversation identifiers depending on how the application is built.
The key design question is:
Where does conversational state live, and who owns its lifecycle?
For a Service Desk product, we may need state associated with a support case or authenticated user. That persistence requirement belongs to application design, not to the existence of an Agent object.
Streaming is structured runtime information
Runner.run_streamed(...) returns a streaming result. The application can consume its event stream through the SDK's streaming interface.
This is useful for:
- incremental text UX,
- tool progress,
- handoff visibility,
- operational telemetry.
Do not equate streaming with exposing private chain-of-thought.
A good UI can say:
Searching approved knowledge...
Checking certificate status...
Waiting for approval...
without revealing hidden model reasoning.
Guardrails are runtime checks, not your authorization layer
SDK guardrails can help validate or constrain inputs/outputs around agent execution.
They are useful, but they do not replace deterministic product security.
For example:
User asks: "Reset the CEO's password"
Even if the model output passes an LLM-oriented guardrail, the reset tool still needs a real authorization decision based on authenticated identity and policy.
Think in layers:
input/runtime guardrails
+
authentication
+
authorization
+
tool validation
+
audit
Tracing makes the hidden runtime structure inspectable
The SDK includes tracing support around agent workflow execution. Current SDK tracing can capture structured execution such as agent runs, turns, generations, tool calls, guardrails and handoffs.
That is valuable because agent failures are often not visible in the final answer alone.
When debugging, ask:
- Which agent was active?
- Which turn produced the problematic decision?
- Which tool was requested?
- What arguments were supplied?
- Was there a handoff?
- Where did the run terminate?
Apply the privacy rules from our observability lessons: traces can contain sensitive prompts, tool arguments and outputs.
Turn limits and failure handling still matter
Framework-managed execution still needs bounded runtime behavior.
Potential failures include:
- model provider timeout,
- tool exception,
- invalid tool arguments,
- repeated handoffs,
- policy denial,
- session-store failure,
- maximum-turn exhaustion.
The application should define what the user experiences for each important failure category.
A framework giving you an exception type is not the same thing as having a product recovery strategy.
Compare abstraction levels instead of framework marketing
Our custom loop gave us maximal visibility into execution mechanics.
A high-level SDK can reduce code and provide maintained integrations.
When evaluating any agent framework, ask:
| Question | Why it matters |
|---|---|
| What owns the execution loop? | Determines where control lives. |
| How are tools validated/executed? | Affects security and side effects. |
| How is state persisted? | Determines recovery and continuity. |
| How are handoffs represented? | Determines delegation behavior. |
| How are runs traced? | Determines debuggability. |
| What are the escape hatches? | Determines whether you can implement non-standard requirements. |
| How are limits/failures surfaced? | Determines production reliability. |
Do not choose a framework merely because a ten-line demo is shorter than your custom implementation.
Service Desk connection
Today we rebuild a familiar slice of the Service Desk using the Agents SDK.
The comparison is deliberate:
Before
------
we explicitly own loop + dispatch + delegation plumbing
Today
-----
Agent + Runner own more orchestration mechanics
Still ours
----------
identity
authorization
business rules
side-effect safety
state policy
evaluation
reliability targets
privacy
product UX
The principle is:
A good SDK can remove orchestration boilerplate, but it does not remove engineering responsibility. Because we understand the loop underneath, we can use the abstraction without becoming dependent on magic.
02 · APPLY
Lesson goal
Today we rebuild a familiar Service Desk workflow with the OpenAI Agents SDK and compare the abstraction with the loop and graph runtimes we already understand.
By the end of the lesson you should be able to:
- configure an
Agent, - run it with
Runner.run(...), - explain the difference between tools and handoffs,
- identify the active agent after a handoff,
- explain how sessions differ from an
Agentobject, - consume streamed run events without exposing private reasoning,
- use tracing for tool/handoff debugging,
- keep turn limits, authorization and side-effect safety in the application design.
What changed in the Service Desk today?
Earlier we implemented the execution loop ourselves and later represented complex workflow state explicitly with LangGraph.
Today we ask a different question:
If a maintained SDK already owns common agent-loop mechanics, which parts of our code can disappear—and which engineering responsibilities remain?
graph LR
U[User] --> T[Triage Agent]
T -->|handoff| A[Authentication Agent]
T -->|tool call| KB[Knowledge Tool]
A -->|tool call| ID[Identity Tool]
A --> R[Final Output]
Step 1 — Configure agents
from agents import Agent
triage_agent = Agent(
name="Triage Specialist",
instructions=(
"Classify IT support requests. Handle general requests yourself and "
"handoff authentication incidents to the authentication specialist."
),
)
auth_agent = Agent(
name="Authentication Specialist",
instructions=(
"Handle SSO, MFA, and password-reset incidents. "
"Never claim a reset succeeded unless the authorized tool confirms it."
),
)
An Agent describes behavior/configuration. Creating it does not start a background worker or persistent autonomous process.
Step 2 — Configure a handoff
A handoff transfers active control to another configured agent.
triage_agent = Agent(
name="Triage Specialist",
instructions="Route authentication incidents to the specialist.",
handoffs=[auth_agent],
)
Conceptually:
Triage is active
|
| handoff selected
v
Authentication specialist becomes active
|
v
subsequent turns continue from that agent
This is different from using another specialist as a bounded tool whose result returns immediately to the original agent.
Step 3 — Execute with Runner.run(...)
from agents import Runner
result = await Runner.run(
triage_agent,
"I am locked out of corporate SSO after changing my MFA phone.",
)
print(result.final_agent.name)
print(result.final_output)
Map this to the lower-level loop we built earlier:
Our custom runtime Agents SDK
------------------ ----------
loop entry Runner.run(...)
agent instructions/config Agent(...)
model/tool/handoff turns Runner-managed run loop
final state/output RunResult
The SDK is reducing orchestration boilerplate. It is not changing the fundamental idea that a run can contain multiple model/tool/handoff turns.
Step 4 — Tool execution still belongs to software
Suppose the authentication specialist can inspect an account.
from agents import function_tool
@function_tool
async def get_account_status(user_id: str) -> dict:
"""Return current account lock/MFA status for an authorized user."""
return await account_api.get_status(user_id)
The model may request this tool, but the Python runtime executes it.
Production code still needs to decide:
- which authenticated user is making the request,
- whether that user may inspect the target account,
- how arguments are validated,
- which credentials the function uses,
- how timeouts/errors are represented,
- what data is safe to return to the model.
A decorator is not an authorization system.
Step 5 — Tools versus handoffs
Use a tool when the active agent needs a bounded capability/result.
Triage -> search knowledge tool -> result returns to Triage
Use a handoff when responsibility for the conversation/workflow should transfer to another configured agent.
Triage -> handoff -> Authentication Specialist becomes active
This distinction is architectural. Do not create many agents merely because a Python function could be given a persona.
Step 6 — Sessions are explicit conversation state
An Agent object does not itself mean “this user is remembered forever.”
For multi-turn behavior, the application needs a conversation/session strategy supported by the SDK/application architecture.
Ask:
What identifies this conversation?
Where is history stored?
How long is it retained?
Which tenant/user owns it?
How is it deleted?
Never share one session/history object across unrelated users simply because it is convenient in a demo.
Step 7 — Stream structured events
For a responsive UI, use streamed execution rather than inventing private reasoning output.
Conceptually:
streamed = Runner.run_streamed(
triage_agent,
"Check why my SSO is locked",
)
async for event in streamed.stream_events():
handle_event(event)
The application can translate safe events into UX such as:
Checking account status...
Routing to authentication specialist...
Waiting for tool result...
Do not expose hidden chain-of-thought.
Step 8 — Understand handoff failure modes
Circular delegation
Triage -> Auth -> Triage -> Auth -> ...
A framework can make handoffs easy; it cannot make a bad delegation graph useful.
Runs need bounded turns and sensible responsibility design.
Wrong specialist
If the routing instruction is ambiguous, the model may hand off incorrectly.
Evaluate routing with representative cases rather than trusting one demo.
Specialist lacks required tool/policy
A handoff can succeed technically while the target agent cannot safely complete the business task.
The target configuration needs the right capabilities and security context.
Step 9 — Turn limits are runtime protection
A production run must be bounded.
If a model repeatedly calls tools or agents hand off in a cycle, the runtime should eventually stop rather than consume unbounded latency/cost.
The exact turn budget depends on the workflow. The lesson is not to memorize one number.
When a limit is reached, the application needs an intentional user-facing outcome:
"I could not complete this request within the allowed workflow steps."
and useful trace evidence for debugging.
Step 10 — Trace the run
The Agents SDK provides tracing support around run behavior.
Use traces to answer operational questions such as:
- Which agent started the run?
- Did a handoff occur?
- Which tool was requested?
- Which arguments were passed after validation/redaction?
- Which agent produced the final output?
- Where did an exception or guardrail stop execution?
Tracing helps because a plausible final response can hide a poor or unsafe trajectory.
Apply the privacy rules from Day 33: prompts, tool arguments and outputs can contain sensitive data.
Step 11 — Guardrails complement policy
SDK guardrails can validate or constrain inputs/outputs around the run.
But consider:
"Reset the CEO's password."
Even a perfectly well-formed model output still needs deterministic authorization before a reset tool executes.
A useful layering is:
SDK/runtime guardrails
+
authentication
+
authorization
+
tool argument validation
+
idempotency/approval where required
+
audit
Debugging workflow
If the run behaves unexpectedly:
- Confirm which
Agentwas initially configured. - Inspect trace/run events to see which agent was active each turn.
- Identify whether the model requested a tool or handoff.
- Inspect the target tool/handoff configuration.
- Check tool arguments and deterministic authorization.
- Look for repeated tool calls or handoff cycles.
- Verify session/history belongs to the correct user.
- Check whether the run terminated because of a turn/guardrail/tool error.
Do not start by rewriting every prompt.
Practical lab
Work in:
service-desk-day-39/
Task A — map abstractions
Create a table mapping the course's custom loop concepts to:
Agent,Runner,- tool,
- handoff,
- run result,
- session,
- trace.
Task B — run a triage request
Use the Day 39 implementation to submit an authentication request.
Record:
- starting agent,
- whether a handoff occurred,
- final active agent,
- final output.
Task C — add a tool failure
Make one specialist tool return a controlled failure.
Verify the failure is observable and does not silently produce a false success claim.
Task D — test a circular/incorrect handoff case
Create or simulate a case where two agents could repeatedly hand off.
Verify the configured run limit stops unbounded execution and that the trace makes the cycle visible.
Task E — run the suite
cd service-desk-day-39
PYTHONPATH=. pytest -q
For every failed test, identify whether the defect belongs to:
agent configuration
runner lifecycle
tool implementation
handoff routing
session state
product policy
Knowledge check
1. What does Runner.run(...) own?
It executes the configured agent workflow, coordinating model turns and SDK-supported tool/handoff behavior until the run completes or terminates.
2. Is an Agent object a persistent conversation?
No. Conversation/session persistence is a separate state-management concern.
3. When is a handoff preferable to a tool?
When active responsibility should transfer to another configured agent rather than simply obtaining one bounded result.
4. Does an SDK guardrail authorize a password reset?
No. Business authorization remains a deterministic application responsibility.
5. What should streaming expose?
Safe structured runtime/output progress—not private chain-of-thought.
Scenario
The Triage Agent hands an SSO request to Auth. Auth discovers it lacks permission to call the required identity tool and hands control back to Triage. Triage sends it back to Auth. This repeats until the run stops.
What should you fix?
Answer: Fix responsibility/capability design and routing so a target agent can either complete the request, fail explicitly, or escalate through a bounded path. A higher turn limit would only make the cycle longer and more expensive.
Key takeaways
- The Agents SDK packages runtime mechanics we already learned from first principles.
Agentis configuration;Runnerexecutes the run.- Tools return bounded capability results; handoffs transfer active responsibility.
- Sessions/state must be designed explicitly.
- Streaming and tracing make runtime behavior observable without publishing private reasoning.
- Framework guardrails do not replace authorization, idempotency or product policy.
- Understanding the underlying loop lets us use a high-level SDK without treating it as magic.
Official references
- Running Agents: https://openai.github.io/openai-agents-python/running_agents/
- Agents: https://openai.github.io/openai-agents-python/agents/
- Handoffs: https://openai.github.io/openai-agents-python/handoffs/
- Sessions: https://openai.github.io/openai-agents-python/sessions/
- Tracing: https://openai.github.io/openai-agents-python/tracing/
03 · EXPLAIN
Interview checkpoint
Explain one design decision from this lesson, the alternative you rejected, and the failure mode or evidence that justified your choice.