supraj.dev

Interview questions & answers

Agents, tools & memory

Bound agent loops, connect MCP tools and recover state without repeating side effects.

  • 38 questions
  • 2 easy
  • 19 medium
  • 17 hard
  • 4 related articles

38 questions

Question 1EasyProduction reasoning

An agent keeps calling the same search tool. How would you bound it?

#

A research task repeats a nearly identical search and never returns a useful answer.

Show answer

Inspect the sequence of tool calls and results. Set step, time and cost budgets in the application. Detect repeated ineffective actions and return a useful partial result or ask for missing information. Make success and termination explicit. A tool result should tell the agent what happened and what is still unknown.

Trade-offs

Small budgets control spend but can end legitimate long tasks; tune using representative traces.

Common mistake

Telling the model to stop eventually without enforcing a runtime limit.

Follow-up

What would you log to distinguish a tool failure from poor planning?

A useful sequence to explain
  1. Task
  2. Bounded decision
  3. Validated tool call
  4. Result
  5. Stop or next step
Question 2MediumProduction reasoning

A new MCP tool is discoverable. What must you check before enabling it?

#

A server advertises a tool that can change production configuration.

Show answer

Review the tool input and output contracts, trust boundary and required permissions. Validate arguments and authorize the specific operation server-side. Expose its impact clearly and require a review step for consequential changes. Test malformed inputs and denied actions. Treat tool metadata and tool output as untrusted input to the application.

Trade-offs

Narrow tools are easier to govern but may require more calls than a broad shell interface.

Common mistake

Assuming a read-only annotation or a tool description enforces permissions.

Follow-up

Where should authorization live if two different clients call the same server?

Question 3MediumProduction reasoning

An assistant remembers the wrong customer. How should memory be scoped?

#

A stored preference from one account appears while another user is signed in.

Show answer

Inspect user, tenant and thread identifiers at reads and writes. Distinguish conversation checkpoints from deliberately stored long-term facts. Enforce access controls, retention and deletion requirements on both. Store provenance and allow corrections. Recheck authorization when loading memory rather than trusting the previous session.

Trade-offs

More memory can improve continuity but increases privacy, stale-data and contradiction risks.

Common mistake

Using one global vector store as unrestricted memory.

Follow-up

How would you test account switching and deletion of remembered data?

Question 4HardProduction reasoning

A tool times out after creating a ticket. Should the agent retry?

#

The ticket API may have completed the write, but its response never arrived.

Show answer

Use an operation ID and an idempotency key where supported. Query operation status or look up the created resource before retrying. Persist the confirmed outcome with the workflow state. If the destination cannot deduplicate or reconcile, surface the uncertainty for review rather than blindly repeating the write.

Trade-offs

Reconciliation adds latency and state but protects against duplicate side effects.

Common mistake

Treating every timeout as proof that nothing happened.

Follow-up

How would you resume after the process crashes between the write and checkpoint?

A useful sequence to explain
  1. Intent + operation ID
  2. Execute once
  3. Confirm destination state
  4. Persist outcome
  5. Retry only after reconciliation
Question 5HardProduction reasoning

Would splitting a slow agent into five agents help?

#

A team wants specialized agents for every stage of an incident investigation.

Show answer

Measure current correctness and time spent in models, retrieval and tools. Parallelize independent read operations first. Consider separate agents only where clear responsibilities and handoff contracts improve measured outcomes. Bound their shared state and permissions. Test routing mistakes, duplicated work and partial failures against a simpler baseline.

Trade-offs

Specialization may improve focus, while handoffs introduce coordination, cost and new failure paths.

Common mistake

Assuming more agents automatically make the workflow more capable or faster.

Follow-up

Which investigation steps must remain sequential because they depend on earlier evidence?

Question 6EasyLangChain / LangGraph

When would you use LangChain’s agent APIs versus a custom LangGraph workflow?

#
Show answer

LangChain provides higher-level agent and model/tool integrations. Its agents are built on LangGraph, so the distinction is not simply “chains versus agents.” Start with the higher-level API when its execution pattern meets your needs.

Use LangGraph directly when you need explicit nodes, state updates and branching across deterministic and model-driven steps, or more control over recovery and human review. Explain the concrete workflow that needs that control. A custom graph brings maintenance costs; do not add one just to use another framework.

Question 7MediumLangGraph

What is the difference between graph state, a checkpoint and long-term memory?

#
Show answer

State is the data the graph reads and updates during execution. A checkpoint is a persisted snapshot used to continue or recover a particular thread. A store holds application-defined information that can outlive one thread, such as approved preferences.

Choose an appropriate persistence backend: an in-memory checkpointer will not survive process restarts. Scope access to both thread state and stored memory, and define retention. A checkpoint is not proof that an external side effect happened exactly once; tool calls still need idempotency and recovery rules.

Question 8HardLangGraph

How would you design a bounded repair loop for model-generated queries?

#
Show answer

Generate a candidate, validate its syntax and allowed operations, and return specific validation errors for a bounded retry. Track attempts and an overall deadline in state; route to a clear failure or human review when the budget runs out. A graph recursion limit is an additional guard, not a substitute for a business-level retry policy.

Use narrowly scoped database permissions and read-only validation where possible. Valid syntax does not prove that the query answers the user’s question. Check semantics against examples and policy; an LLM judge cannot authorize unsafe execution.

Question 9MediumContext engineering

How would you separate durable facts, current context and an agent’s procedure?

#
Show answer

Keep authoritative facts in governed data sources, assemble task-specific context with timestamps and access checks, and express required steps and branches in an explicit workflow. A graph can represent relationships, but every agent does not require three separate graph databases.

For a booking task, recheck current availability before acting; a stored preference is not a live inventory record. Version procedures, record decisions and validate permissions at execution. Treat feedback as evidence to review before updating durable knowledge, rather than automatically promoting an agent’s trace to truth.

Question 10MediumExecution evidence

A coding agent says “done” and exits successfully. What would prove the requested fix happened?

#
Show answer

Inspect the intended workspace and diff, then reproduce the reported bug and run the relevant unchanged tests against the resulting code. Record commands, working directory, revision and results. A successful agent process only shows that its own execution ended successfully.

Even a test runner’s successful exit does not show that the right tests exercised the requirement. Check collection and outcomes; do not accept changed assertions or an empty patch as evidence of a repair. Tie the completion claim to observable artifacts and the requested behavior.

Question 11MediumHuman in the loop

How do you make an agent pause for human approval before a risky action, and resume later?

#

A finance agent prepares refunds. Anything over a threshold needs a manager's approval, which may take hours.

Show answer

Interrupt the run before the tool executes, not after. Save the agent's state with a checkpointer so the process can stop completely, and show the approver the exact action and parameters, such as amount, account and reason, not a summary written by the model.

When the approver decides, resume from the checkpoint with their decision as input: proceed, edit the parameters, or reject with a reason the agent can use. Add a timeout with a safe default (usually reject), record who approved what, and make the tool idempotent so a resumed run cannot pay twice.

Trade-offs

Approvals add delay and human effort; apply them only above risk thresholds you can explain.

Common mistake

Asking the model to "wait for approval" in the prompt instead of enforcing it in code.

Follow-up

What should happen if the underlying data changed while the run was paused?

Question 12HardDesign

When should you build a fixed workflow instead of an autonomous agent?

#

A team wants an "agent" for invoice processing: extract fields, validate, post to the ERP.

Show answer

If the steps are known and the same each time, a workflow is better: code calls the model at fixed points, so it is predictable, testable and cheaper. Invoice processing is mostly that, with a model for extraction and code for validation and posting.

Agents, where the model decides the next step, fit open-ended tasks where you cannot list the path in advance. Start with the simplest design that works and add autonomy only where measurements show a fixed path falls short.

Trade-offs

Agents handle variety better but are harder to test, cost more and fail in less predictable ways.

Common mistake

Giving a model full control of a process that a five-step workflow would handle.

Follow-up

Which part of the invoice flow, if any, would justify agent behaviour?

Question 13HardDurability

A long-running agent crashes after step 7 of 12. How do you resume without repeating side effects?

#

The agent already sent an email and created a ticket before the worker died.

Show answer

Checkpoint the agent's state after each step so it can resume from the last completed step instead of starting over; graph frameworks such as LangGraph provide checkpointers for this.

Checkpoints are not enough for side effects, because a crash can happen between doing an action and recording it. Give each action an idempotency key derived from the run and step, have tools return the existing result when they see a key again, and require confirmation for actions you cannot undo.

Trade-offs

Frequent checkpoints add storage writes but make recovery cheaper and more precise.

Common mistake

Restarting the whole run from the beginning after any failure.

Follow-up

What would you do for a third-party API that does not support idempotency keys?

Question 14HardMulti-agent design

One team wants an orchestrator with parallel sub-agents for its research assistant; another wants the same design for its coding agent. Would you approve both?

#

The research assistant answers broad questions that need many independent searches. The coding agent makes changes across a single repository.

Show answer

Probably the research assistant, probably not the coding agent, and only with measurements.

Anthropic's write-up of its multi-agent research system shows where the design pays: a lead agent (Claude Opus 4) with Claude Sonnet 4 sub-agents outperformed single-agent Claude Opus 4 by 90.2% on their internal research eval. Research suits it because the work splits into independent searches that can run in parallel, each in its own context. The same post is clear about the costs: agents typically use about 4 times more tokens than chat, and multi-agent systems about 15 times more, and on their BrowseComp analysis token usage alone explained 80% of the performance variance. So part of the gain is simply spending more tokens. It also says the approach fits poorly where all agents need the same context, and that most coding tasks have fewer truly parallel parts than research.

Cognition's "Don't Build Multi-Agents" makes the coding case directly: share full context and full agent traces, not just messages, because actions carry implicit decisions and conflicting decisions produce bad results.

So for research, approve a pilot measured on quality per dollar against a single agent given the same token budget. For coding, keep one agent that holds the context, and use sub-agents only for read-only side tasks such as searching the codebase and summarising what they found.

Trade-offs

Parallel sub-agents explore more in less wall-clock time but multiply token cost and coordination risk; one agent is cheaper and consistent but slower on broad tasks.

Common mistake

Comparing a multi-agent system with a single agent given far fewer tokens, and crediting the architecture for what the budget bought.

Follow-up

How would you design the equal-budget comparison fairly?

Question 15MediumMulti-agent design

Two sub-agents each built half of a feature in parallel. Each half works alone, but together they do not fit: different naming, different assumptions about the data. What went wrong in the design?

#

The orchestrator gave each sub-agent a one-paragraph task description and merged their outputs at the end.

Show answer

Every action an agent takes carries decisions that are not in its task description: names, data shapes, error handling, style. Cognition's "Don't Build Multi-Agents" uses exactly this example: a task to build a Flappy Bird clone was split in two, one sub-agent built a background in a different game's style, and the other built a bird that did not match. Their two principles are to share context, including full agent traces rather than single messages, and to recognise that conflicting implicit decisions produce bad results.

Fixes, in order of preference: - Do not parallelise work that must be consistent. Let one agent do the writing in sequence, and use sub-agents for independent reading or research that returns facts, not designs. - If you must split, decide the shared parts first: write the interface, schema and naming into a shared spec that both sub-agents receive and cannot change. - Give each sub-agent the relevant decisions made so far, not just a task summary. - Add an integration check that runs both halves together before the orchestrator reports success.

Trade-offs

Parallel writers finish sooner but disagree; a single writer with parallel readers is slower but coherent.

Common mistake

Splitting by component and giving each sub-agent only its own instructions, then expecting the merge step to reconcile them.

Follow-up

Which kinds of subtasks are safe to run in parallel without shared context?

Question 16MediumMulti-agent design

In an orchestrator design, the lead agent's context fills with long sub-agent reports, and details get lost or garbled as the lead rewrites them into the final answer. How do you restructure the hand-back?

#

Eight sub-agents each return 5,000-word reports to the lead, which then writes a final document.

Show answer

Passing every result through the lead agent is a game of telephone: each rewrite drops or distorts detail, and the lead's context fills up. Anthropic's multi-agent research write-up recommends artifact systems, where specialised agents create outputs that persist independently instead of passing everything back through the lead.

In practice: - Sub-agents write their full results to a store (files, a database, object storage) and return a short summary plus a reference to the artifact. - The lead plans and decides using summaries, and opens an artifact only when it needs a detail. - The final output is assembled from the artifacts directly, by code or by a writing step that reads them, rather than from the lead's memory of them. - Each artifact keeps its sources and the sub-agent's task, so the final document can cite them.

This also makes runs easier to debug and resume: artifacts survive a crash of the lead, and you can see exactly what each sub-agent produced.

Trade-offs

Artifacts add storage and a reference format to maintain, but keep the lead focused and preserve detail.

Common mistake

Raising the lead agent's context budget instead of changing how results flow.

Follow-up

How would you stop a sub-agent from writing an artifact that another agent later treats as trusted instructions?

Question 17MediumTool calling

The model issues create_customer and create_invoice as parallel tool calls in one turn. The invoice call fails because the customer does not exist yet, and sometimes both fail in confusing ways. How do you handle parallel tool calls safely?

#

Your executor runs all tool calls from one assistant turn concurrently.

Show answer

Models may emit several tool calls in one turn, and nothing in that turn says one depends on another. If your executor runs them concurrently, a dependent call can run before the call it needs.

Options: - Turn parallel calls off where order matters. With OpenAI's API, setting parallel_tool_calls to false ensures zero or one tool is called per turn. Other providers have similar settings; check yours. You lose some speed on independent calls. - Design tools so dependent steps are one call: a create_invoice that accepts customer details and creates or finds the customer itself, idempotently. - Make the executor smarter: run read-only calls in parallel, but run calls with side effects one at a time in the order given. - Return clear results per call, matched by call ID, including which failed and why, so the model can retry only what failed. A failed create_invoice should say the customer was not found, not just "error".

Also make write tools idempotent, because a partial failure followed by a retry will otherwise repeat the successful half.

Trade-offs

Sequential execution is predictable but slower; parallel execution is faster for reads but needs dependency handling.

Common mistake

Running every tool call from a turn concurrently, including writes that depend on each other.

Follow-up

How would you let the model mark that two calls are independent?

Question 18MediumTool calling

About 2% of tool calls have arguments that do not match the schema (missing fields, wrong types, extra keys), and the agent then loops on validation errors. What do you change?

#

Tools are defined with JSON Schema; the agent framework passes raw arguments to your functions.

Show answer

First, use the provider's strict mode so arguments are generated to match the schema. OpenAI's function calling has strict mode, which requires additionalProperties: false on every object and every property listed in required (optional fields are expressed by allowing null). Anthropic's API supports strict: true on tool definitions for schema validation of tool inputs. With strict mode, structural errors such as missing fields or wrong types should disappear.

Second, strict mode only guarantees shape. Values can still be wrong: an order ID that does not exist, a date in the past, an amount above a limit. Validate those in the tool and return an actionable error the model can use, such as which field was wrong and what is allowed.

Third, stop loops. Count validation failures per tool per run; after two or three, stop retrying and route to a fallback or a person.

Also review the schema itself: deeply nested or ambiguous schemas cause more mistakes, and a clear description per field and an example often help more than extra constraints.

Trade-offs

Strict schemas remove a class of errors but constrain schema features you can use; looser schemas are flexible but need more defensive validation.

Common mistake

Treating schema-valid arguments as correct arguments and skipping business validation.

Follow-up

How would you report a validation error so the model fixes it on the first retry?

Question 19MediumTool calling

To make an agent always look things up, a developer set tool_choice to force a tool on every request. Now the agent never gives a final answer, and on a newer model the requests fail with a 400 error. Explain both problems.

#

The loop sends the same tool_choice on every turn of a Claude API conversation.

Show answer

On the Claude API, tool_choice has four options: auto (the default when tools are given) lets the model decide, any makes it call one of the tools, tool forces a named tool, and none blocks tools. If every turn uses any or tool, the model must call a tool every time, so it can never return a plain final answer: the loop only ends at your step limit. Forcing also prefills the response with the tool call, so there is no explanation text before it, and the docs note that forcing a tool skips thinking.

The 400 error is a model capability issue. The docs list models that reject forced tool use (for example Claude Opus 5.5 and Sonnet 5.5) and say manual extended thinking does not support any or tool either; there, use auto with strict tool use for schema-valid inputs, or structured outputs for a fixed JSON shape.

The fix is to force at most the first step if you must, then switch to auto and use instructions to encourage lookups. Also note that changing tool_choice between requests invalidates cached message blocks under prompt caching, so frequent switching costs cache hits.

Trade-offs

Forcing a tool guarantees a call but removes judgement and thinking; auto with good instructions keeps the model flexible but can skip a lookup.

Common mistake

Using forced tool choice as a permanent setting in a multi-turn agent loop.

Follow-up

How would you verify that the agent did look things up when auto is used?

Question 20MediumTool results

An agent told a customer "you have no open invoices". The billing API had returned an error, and the tool wrapper turned it into an empty list. How do you stop tool failures from becoming confident wrong answers?

#

Tool wrappers catch exceptions and return [] so the agent "keeps going".

Show answer

The wrapper erased the difference between "there are none" and "I could not find out". The model reasonably believed the empty list.

Design tool results so failure is explicit: - Return a clear error result on failure (most tool APIs have an error flag on tool results), with a short reason and whether retrying makes sense. - Distinguish empty from partial: if a search hit a timeout on one shard or returned the first page only, say so, with counts. - Include what was checked: "No open invoices for account 123 as of 10:42" is safer than an empty list, because the model can repeat it accurately. - Never let a catch-all handler turn exceptions into normal-looking data.

Then make the agent's instructions match: when a tool reports an error, say that the information is unavailable and offer a next step; do not guess.

Test it. Add fault injection to the agent's test suite: make each tool fail, time out and return partial data, and check that the final answer reflects it.

Trade-offs

Explicit errors make the agent admit gaps more often, which can look less helpful, but wrong confident answers cost far more.

Common mistake

Swallowing tool exceptions to keep the agent running.

Follow-up

Which tool failures should the agent retry by itself, and which should it report immediately?

Question 21HardTool design

An agent copies a 10,000-row spreadsheet from one tool into another by reading it into its context and writing it back out. It is slow, expensive and sometimes changes values. What design would you move to?

#

The agent connects to many MCP servers; all tool definitions are loaded up front and all results go through the model.

Show answer

Two costs are piling up: every tool definition is loaded into context up front, and every intermediate result passes through the model, which also lets the model alter data it should only move.

Anthropic's "Code execution with MCP" describes the alternative: present MCP servers to the agent as code APIs and let it write a small program that calls them in a sandbox. The program moves the rows from one tool to the other; only a summary (row counts, a few samples, errors) comes back to the model. The agent loads only the tool definitions it needs. Their example cut token use from 150,000 to 2,000, a 98.7% saving. Intermediate data stays in the execution environment by default, and the client can tokenise personal data before anything reaches the model.

The cost is a secure execution environment for agent-written code: sandboxing, resource limits and monitoring, as the post states. That is real infrastructure and a new attack surface. Treat the generated code as untrusted, restrict network and file access to the tools needed, and log what ran.

For simple, high-volume data moves, a dedicated tool such as copy_rows(source, destination) is even simpler than code execution. The rule is the same: do not route bulk data through the model.

Trade-offs

Code execution saves tokens and keeps data exact but needs a sandbox and code review; direct tool calls are simpler but costly for bulk data.

Common mistake

Using the model as the transport for bulk data between tools.

Follow-up

What limits would you put on the sandbox so agent-written code cannot exfiltrate data?

Question 22HardEvaluation

Your support agent passes 80% of test tasks in a single run, and the team calls it ready. Why might customers still see it fail often, and how should you measure reliability?

#

Each test task was run once. Real customers repeat similar requests many times a day.

Show answer

A single run measures whether the agent can do a task, not whether it does it consistently. Agents are sampled and their trajectories branch, so the same task can pass on one run and fail on the next. At scale, customers experience the failure rate across repeats.

The tau-bench paper (Yao et al.) made this point with a benchmark of simulated user-agent conversations in which the final database state is compared with the goal state. It introduced a pass^k metric for reliability over multiple trials, and reported that even state-of-the-art function-calling agents (such as gpt-4o) succeeded on fewer than 50% of tasks, with pass^8 below 25% in the retail domain.

Measure it yourself: - Run each test task several times (for example 5 to 10) and report the fraction of tasks that pass every time, not just the average. - Score outcomes by checking end state (records created, refunds issued) rather than the wording of the reply. - Look at the tasks that pass sometimes: these show where instructions, tools or policies are ambiguous.

Report both the single-run success rate and the all-trials rate, and gate release on the one that matches the business risk.

Trade-offs

Repeated trials multiply eval cost, but single-run scores overstate reliability.

Common mistake

Reporting a one-shot pass rate as the reliability of an agent that customers will use repeatedly.

Follow-up

How would you simulate the user side of a multi-turn task cheaply and realistically?

Question 23MediumLoop control

After a LangGraph upgrade, a misbehaving agent ran thousands of steps before stopping, where it used to stop quickly. What changed, and what should actually bound an agent run?

#

The team relied on the framework's recursion limit as its only safety stop.

Show answer

LangGraph raises GraphRecursionError when a run exceeds its recursion limit, counted in super-steps. The current docs state that the default is 10,007 super-steps starting in version 1.1.4 (and 10,000 in versions 1.0.6 to 1.1.3). A limit that high is a guard against infinite graphs, not a cost or behaviour budget. If your system relied on an older, much lower default, an upgrade removed your real stop.

Set the stops you actually want, in application code and per run: - A step budget suited to the task (for example 30 tool calls for a support task), set explicitly with recursion_limit or your own counter. - A token and money budget, checked before each model call. - A wall-clock deadline. - Repetition detection: the same tool with the same arguments, or no new information, several times in a row.

When a budget runs out, end with a useful partial result or a hand-off to a person, and log which limit fired. Pin framework versions and read release notes for changed defaults, and add a test that a looping agent stops within the budget you expect.

Trade-offs

Tight budgets stop runaway cost but can cut off legitimate long tasks; per-task budgets need tuning but match real needs.

Common mistake

Treating a framework default as a business limit, so it changes silently on upgrade.

Follow-up

How would you choose the step budget for a new agent before you have production data?

Question 24MediumPlanning loops

A writer-and-critic loop improves drafts for two rounds, then the critic keeps finding new issues and the draft oscillates between two versions. How do you make an evaluator-optimizer loop converge?

#

The critic is asked to "find problems and suggest improvements" each round with no stopping rule.

Show answer

Anthropic's "Building effective agents" describes the evaluator-optimizer pattern (one model drafts, another critiques, in a loop) as a fit for tasks with clear evaluation criteria where iteration adds measurable value. The loop here has neither: "find problems" has no finish line.

Make it converge: - Give the critic a fixed rubric with pass or fail per criterion (facts supported, required sections present, length, tone) and have it return a verdict, not just suggestions. Stop when all criteria pass. - Cap the rounds (two or three is often enough) and keep the best-scoring draft, not the last one. - Detect oscillation: if a draft is very similar to one from two rounds earlier, stop. - Ask the critic to rank issues by severity and fix only blocking ones, so cosmetic preferences do not cause endless changes. - Use checks in code where possible (links valid, citations present, schema correct); they are stable and cheap.

Then measure whether rounds help: compare final quality and cost for one, two and three rounds on an eval set.

Trade-offs

More rounds can raise quality on hard tasks but cost tokens and can drift; a strict rubric converges faster but may miss issues outside it.

Common mistake

Letting the loop run until the critic has no comments, which may never happen.

Follow-up

Should the critic be the same model as the writer? What changes if it is?

Question 25MediumPlanning

A planning agent writes a 10-step plan, step 3 fails, and the agent keeps executing steps 4 to 10 as if nothing happened. How should plans and execution interact?

#

The plan is generated once at the start and stored as text in the prompt.

Show answer

The plan was treated as a script rather than as working state. When a step fails or returns something unexpected, the later steps may no longer make sense.

Design it so execution can change the plan: - Keep the plan as structured state (steps with status: pending, done, failed, skipped, and their outputs), not just text in the prompt. - After each step, check its result against what the plan expected. Replan when a step fails, when a result contradicts an assumption, or when new information arrives. Replanning can be cheap: revise only the remaining steps. - Mark dependencies, so a failed step automatically blocks the steps that need its output. - Set a limit on replans, and escalate if the agent cannot find a path.

Also ask whether you need a full upfront plan at all. For many tasks, deciding the next step from the latest observation (the ReAct style of interleaving reasoning and actions) is more robust; upfront plans help most when steps are expensive, need approval or can run in parallel.

Trade-offs

Upfront plans are reviewable and parallelisable but go stale; step-by-step decisions adapt but are harder to predict and approve.

Common mistake

Generating a plan once and never checking step results against it.

Follow-up

How would you show a human approver a plan that may change during execution?

Question 26HardLong-running agents

You moved an agent loop into a durable workflow engine so it survives crashes. After a worker restart, the workflow fails or takes a different path than before. What rule did the code break?

#

The workflow function calls the LLM API directly and uses the current time to decide whether to send a reminder.

Show answer

Engines like Temporal do not restore a memory snapshot after a crash. Temporal's docs say they start the workflow code from the beginning and replay the event history step by step, so the code must make the same decisions given the same history. If the workflow calls an LLM directly, the replay gets a different response; if it reads the current time, the replay sees a later time. Either way the code can take a different path from the recorded history and fail to match it.

The rule is to keep workflow code deterministic and put everything that touches the outside world in activities. Temporal's docs name API calls, database queries, LLM invocations and file I/O as activity work. Each activity's result is recorded in the history and reused on replay instead of being run again.

So: make each LLM call and each tool call an activity; use the engine's own time and random functions inside workflows; and pass decisions based on outside data in as activity results. Give activities retry policies and idempotency keys for side effects, because an activity can run again if it fails before its result is recorded.

This gives you a long-running agent that resumes exactly where it was, including waits of days for human approval.

Trade-offs

Durable workflows give exact resume and long waits but require discipline about determinism and versioning; a plain loop with checkpoints is simpler but resumes less precisely.

Common mistake

Calling the model or the clock directly inside workflow code because it works until the first replay.

Follow-up

How do you change the agent's workflow code safely while old runs are still in progress?

Question 27HardLong-running agents

A coding agent must build a large app over dozens of sessions, each starting with an empty context. Early attempts tried to do everything at once and later sessions declared the project finished too early. How would you structure the harness?

#

Each session has a fresh context window; files and git history persist between sessions.

Show answer

Anthropic's "Effective harnesses for long-running agents" describes these exact failures: the agent tried to do too much at once, leaving features half-built and undocumented, and later sessions declared the project done too early. Their fix splits the work into two roles that share a harness.

An initializer agent runs once. It sets up an init.sh script to start the app, a progress log (claude-progress.txt), an initial git commit, and a JSON feature list (over 200 features in their example) with every feature marked as failing. Later sessions may only change each feature's pass status; the authors chose JSON because the model is less likely to overwrite it than Markdown.

A coding agent then runs each session: it reads the progress file and git log, runs a basic end-to-end test to catch a broken state, works on one feature, tests it end-to-end, commits with a clear message, updates the progress file and leaves the code clean. Testing like a user, with browser automation, worked much better than relying on unit tests or curl alone.

The general lessons: put durable state in files the agent can read and that code can check; make "done" a list of verifiable items, not the agent's opinion; and keep each session to one unit of work.

Trade-offs

A structured harness adds setup and rules but makes progress measurable; letting the agent manage itself is quicker to start and fails on long projects.

Common mistake

Relying on context compaction alone to carry a long project across sessions.

Follow-up

How would you stop the agent from marking a feature as passing without real evidence?

Question 28HardLong-running tools

An MCP tool kicks off a data export that takes 20 minutes. Holding the tool call open times out, and retries start duplicate exports. How do you model long-running tool work?

#

The MCP server and client both run the 2025-11-25 protocol version.

Show answer

Do not hold a tool call open for 20 minutes. Turn it into start, check and fetch.

The MCP 2025-11-25 specification added tasks for this, marked experimental. A requestor can attach a task field (with an optional ttl) to a tools/call, and the server immediately returns a task ID with status working instead of the result. The client polls tasks/get, respecting the suggested pollInterval, until a terminal status (completed, failed or cancelled) or input_required, and then calls tasks/result, which returns exactly what the tool call would have returned. Tools declare support with execution.taskSupport (required, optional or forbidden). Meanwhile the host can return control to the model so it does other work.

Things to get right: - Idempotency: tie the export to an idempotency key, so a retried start returns the existing task instead of a new export. - Cancellation is best effort: the spec requires a cancelled task to stay cancelled even if execution continues to completion, so make the export itself safe to finish or stop. - TTL: servers may delete a task and its results after the TTL, so fetch results in time and store what you need. - Access control: when there is an authorization context, the spec requires binding tasks to it and rejecting access from other contexts.

Without task support, the same pattern works as two tools: start_export returning a job ID and get_export_status.

Trade-offs

Task-based calls avoid timeouts and free the agent to continue, but add polling, state and cleanup; synchronous calls are simpler for work that finishes in seconds.

Common mistake

Retrying a long-running start call on timeout without an idempotency key.

Follow-up

How should the agent explain to the user that the export is running and will finish later?

Question 29HardSide effects

A travel agent books a flight, then a hotel, then a car. The car booking fails, and the customer now has a flight and hotel they did not want. How should multi-step side effects be designed?

#

Each booking is a separate API with its own database; there is no shared transaction.

Show answer

This is the saga pattern. When one business action spans several services that cannot share a transaction, you run a sequence of local steps and, if a later step fails, run compensating steps to undo the earlier ones. The Azure Architecture Center describes the key ideas: compensable steps can be undone; a pivot step is the point of no return; steps after the pivot must be retryable and idempotent so the saga can finish.

Apply it here: - Order steps by how easy they are to undo. Hold or reserve first (hotel holds and flight holds where available), and make the step that charges or issues tickets the pivot, done last. - Define compensation for each step in code: cancel the hotel hold, void the flight reservation. Do not leave it to the model to improvise. - Use an orchestrator (your workflow code) that records each completed step and its compensation, so a crash midway can still compensate. - Make every step and compensation idempotent with keys, because retries happen. - Plan for compensation failing (the guide notes it might not always succeed): alert a person with the exact state.

Also note sagas have no isolation, so another process can see a half-finished booking; the same guide lists countermeasures such as semantic locks and re-reading values before updating.

Trade-offs

Sagas keep services independent but need compensation logic and monitoring; one service owning the whole booking is simpler but couples everything.

Common mistake

Letting the agent call booking tools one by one with no plan for undoing earlier steps.

Follow-up

What would you do when a compensating step, such as a non-refundable cancellation, costs the customer money?

Question 30HardHuman-in-the-loop

An agent asks a manager to approve a supplier payment. The manager approves two days later, and the agent pays, but the invoice amount had been corrected in the meantime. How do you make approvals safe over time?

#

The approval record stores "approved: true" for the run; the agent rebuilds the payment from current data after approval.

Show answer

The approval was attached to the run, not to the exact action, and the action was rebuilt from changed data after approval.

Bind approval to a specific action: - Freeze the proposed action when asking: payee, account, amount, currency, invoice ID and version. Store a hash of these parameters with the approval request. - Show the approver those exact values, not a model-written summary. - At execution, rebuild the action and compare it with the approved one. If anything differs (amount, payee or the invoice version), do not execute: ask for approval again, showing what changed. - Give approvals an expiry suited to the risk, after which the request must be renewed. - Re-check preconditions at execution time: the invoice is still unpaid, the approver still has authority, and the budget still allows it.

Record who approved what, when, and what was executed, so the audit trail shows that the executed action matched the approved one.

Trade-offs

Strict binding causes more re-approvals when data changes, but approving one thing and doing another defeats the control.

Common mistake

Storing approval as a flag on the run and executing whatever the agent produces afterwards.

Follow-up

How would you handle an approval request that is still pending when the agent's code is redeployed?

Question 31MediumHuman-in-the-loop

Operators must approve every action the agent takes, about 400 a day each. Approval takes two seconds on average and nobody has rejected anything in a month. Is this control working, and what would you change?

#

Actions range from reading a log to restarting a production database.

Show answer

Probably not. When people approve hundreds of mostly harmless actions, approval becomes a reflex, and the one dangerous action gets the same two seconds as the rest. A zero rejection rate across very different actions is a warning sign, not a success.

Redesign by risk: - Classify actions. Reads and reversible, low-impact actions run without approval but are logged. Consequential or irreversible actions (production restarts, deletions, payments, messages to customers) need approval. - Put limits in policy rather than in people: allowlists, rate limits, spend caps and scoped credentials stop many bad actions without a click. - Make the few approvals meaningful: show the exact command, target, blast radius and a rollback plan; require a typed confirmation or second approver for the highest-risk actions. - Measure the control: approval time, rejection rate per action type, and periodic tests with deliberately wrong proposals to check that reviewers catch them. - Review the low-risk tier with sampled audits after the fact.

Fewer, better approvals protect more than many rubber stamps.

Trade-offs

Risk tiers reduce human load and sharpen attention but require careful classification; approving everything feels safe but trains people to click through.

Common mistake

Measuring human-in-the-loop by how many actions were approved rather than by whether reviewers catch bad ones.

Follow-up

How would you run a safe exercise to check that approvers would reject a bad action?

Question 32MediumHandoffs

A triage agent hands conversations to specialist agents. The billing specialist sometimes sees internal tool output from triage it should not, and the refunds specialist misses key facts because the history is too long. How do you design the handoff?

#

You use the OpenAI Agents SDK with handoffs between agents.

Show answer

In the OpenAI Agents SDK, handoffs are represented to the model as tools (for example transfer_to_refund_agent), and by default the receiving agent sees the entire previous conversation history. That default explains both problems: everything leaks forward, including tool output, and the important facts are buried.

Design the handoff as a contract: - Use an input_filter to control what the next agent receives. The SDK provides helpers such as remove_all_tools, which removes tool-related items from the history. - Pass structured handoff data: the customer ID, the request, facts already verified, and the reason for the handoff. The receiving agent should not have to reread the whole chat to find them. - Apply permissions at the receiving agent's tools, not by hoping it does not see data. If billing must not see certain internal data, keep it out of the history and out of its tool scope. - Test handoffs with transcripts: does the specialist have what it needs, and nothing it should not?

Watch handoff loops too, where agents pass a conversation back and forth; cap handoffs per conversation and escalate to a person.

Trade-offs

Filtered, structured handoffs are safer and clearer but can drop context the specialist needed; full history is complete but noisy and leaky.

Common mistake

Relying on the default full-history handoff and assuming each agent will ignore what it should not use.

Follow-up

How would you summarise a long conversation for a handoff without losing facts?

Question 33HardConcurrency

Agent workers pull tasks from a queue. A slow task became visible again after its timeout, a second worker picked it up, and both agents replied to the same customer with different answers. How do you prevent duplicate agent runs?

#

Agent runs take from 10 seconds to 15 minutes; the queue visibility timeout is 5 minutes.

Show answer

Most queues deliver at least once: if a message is not acknowledged within its visibility timeout, it is delivered again. A 15-minute agent run with a 5-minute timeout will be processed twice.

Layer the defences: - Match the timeout to the work: extend the message's visibility while the run is alive (a heartbeat), or set the timeout above the longest expected run. - Take a lease on the task in your own store before starting (task ID, worker ID, expiry), renewed by heartbeat. A second worker that finds a live lease stops. - Use a fencing check at every side effect: the worker includes its lease version, and the write is rejected if a newer lease exists. This stops a worker that paused and woke up late. - Make side effects idempotent: a reply to a customer keyed by task ID and step, so a second send is ignored. - Record the outcome against the task, and have any new worker check it before doing anything.

Then test it: kill workers mid-run, pause them past the lease, and confirm one visible reply per task.

Trade-offs

Leases and fencing add a coordination store and code but stop duplicate actions; relying on timeouts alone is simple and fails under slow runs.

Common mistake

Assuming each queue message is processed exactly once.

Follow-up

What should happen to a run whose lease expired while it was waiting for a slow tool?

Question 34HardMemory

After six months, an assistant's long-term memory holds thousands of facts per user, including contradictions ("prefers email", "prefers phone") and things users said once in passing. Answers are getting worse. How should agent memory be written and maintained?

#

Every conversation ends with the model extracting "facts to remember" and appending them to a vector store.

Show answer

Appending every extracted fact creates a pile, not a memory. Retrieval then returns stale and conflicting items, and the model has to guess which is true.

Give memory a policy: - Write less. Store only facts that matter for future tasks (preferences, account context, ongoing projects), and require a clear statement from the user rather than an inference from one passing remark. - Store structure and provenance: subject, attribute, value, source conversation, timestamp and confidence. Then a new value for the same attribute replaces the old one instead of sitting beside it. - Consolidate periodically: merge duplicates, resolve conflicts by recency and source, and expire time-bound facts ("travelling next week"). - Let users see, correct and delete what is remembered, and honour deletion everywhere the memory was copied. - Treat memory as untrusted input on read. Text that came from documents or tools can carry injected instructions; store facts, not instructions, and keep it scoped to the right user.

Measure it: a test set of multi-session scenarios that checks the assistant uses the latest preference and ignores superseded ones.

Trade-offs

Strict write rules keep memory accurate but remember less; liberal writes capture more but degrade retrieval and raise privacy risk.

Common mistake

Appending every model-extracted fact with no way to update, expire or delete it.

Follow-up

How would you migrate an existing pile of memories into the structured form?

Question 35MediumPerformance

A support agent takes 45 seconds on average: about 12 model calls and 9 tool calls, one after another. Users abandon it. Where would you cut time first?

#

Traces show each model call takes 2 to 3 seconds and most tool calls take under 500 ms.

Show answer

Most of the time is serial model calls, so reduce how many happen and how many wait on each other.

- Remove steps. Look at traces: common requests often follow the same path. Turn that path into a workflow in code with one or two model calls, and keep the open-ended agent for unusual requests. - Batch and parallelise. Independent lookups (customer profile, order history, open tickets) can be fetched together, in parallel, or by one tool that returns all three. - Prefetch obvious context before the first model call, such as the customer record for a logged-in user. - Use a smaller, faster model for simple steps such as classification and routing, keeping the large model for reasoning and the final answer. - Cut tokens per call: long tool results and growing histories raise every call's latency. Trim results and reuse cached prompt prefixes. - Stream progress to the user ("Checking your order") and stream the final answer, which reduces abandonment even before the total drops.

Measure end-to-end latency at the 50th and 95th percentile and success rate together, because fewer steps must not mean worse answers.

Trade-offs

Fixed workflows are fast and predictable but handle fewer cases; agent flexibility costs round trips.

Common mistake

Optimising one tool's latency while the agent still makes a dozen sequential model calls.

Follow-up

How would you decide which requests go to the fast workflow and which to the agent?

Question 36HardObservability

Traces of your multi-agent support system are a flat list of about 300 spans per conversation: model calls and HTTP calls with no structure. On-call cannot tell which sub-agent made a call, which tool result led to a bad answer, or how a queued follow-up job relates to the chat. How do you structure agent traces?

#

An orchestrator agent delegates to a billing sub-agent in the same process and to a research agent that runs as a separate service. Some follow-up work is queued and runs minutes later. Conversations span many user turns.

Show answer

Use the OpenTelemetry GenAI agent conventions as the skeleton (they are marked Development, so pin a version).

- Agent invocations: one span per in-process invocation, kind INTERNAL, gen_ai.operation.name set to invoke_agent and named invoke_agent {gen_ai.agent.name}, with the agent's name, ID and version. When the application defines the control flow, for example a LangGraph StateGraph, the operation is invoke_workflow. The conventions also define a plan span for planning steps. - Inside each agent span: a gen_ai.client.inference span per model call and an execute_tool {gen_ai.tool.name} span per tool call. Set gen_ai.tool.call.id to the identifier the model supplied, so each tool span can be matched to the model turn that asked for it. Tool arguments and results are opt-in attributes and fall under your content rules. - Handoffs done through a tool call: the conventions define an 'agent as a tool' refinement of the execute-tool span for transfers, and say not to emit extra spans for them. - The remote research agent: on the caller, a gen_ai.client.invoke_agent span of kind CLIENT. Propagate trace context in the request so the remote service's spans join the same trace. - Queued follow-ups: a PRODUCER span when enqueuing, and a new trace for the job whose first span carries a link to the producer. The spec's overview says links connect causally related spans, including a long-running asynchronous operation started by a fast request. - Conversations: one trace per user turn, not one per conversation; traces lasting hours are hard to sample and to view. Tie turns together with gen_ai.conversation.id. - Per-invocation metrics: the conventions define gen_ai.invoke_agent.duration, gen_ai.invoke_agent.inference_calls and gen_ai.invoke_agent.tool_calls (count buckets 1, 2, 4 up to 128), which show runaway agents without opening traces.

The debugging path then becomes: conversation, turn trace, agent span, the tool span whose result preceded the bad answer.

Trade-offs

Rich agent spans make traces readable but add volume per run; links keep traces short but need a backend that can follow them.

Common mistake

Putting a whole multi-hour conversation, including queued jobs, into one trace, which defeats tail sampling and makes the trace view unusable.

Follow-up

Which span attributes would you index so on-call can find all runs where one specific tool returned an error?

OpenTelemetry Python: agent span and a linked queued job (sketch) · python
from opentelemetry import trace
from opentelemetry.context import Context
from opentelemetry.propagate import extract, inject
from opentelemetry.trace import Link, SpanKind

tracer = trace.get_tracer("support-agents")

def run_billing_agent(task):
    with tracer.start_as_current_span("invoke_agent billing", kind=SpanKind.INTERNAL) as span:
        span.set_attribute("gen_ai.operation.name", "invoke_agent")
        span.set_attribute("gen_ai.provider.name", "anthropic")
        span.set_attribute("gen_ai.agent.name", "billing")
        return agent_loop(task)  # model calls and execute_tool spans are children

def enqueue_followup(queue, payload):
    with tracer.start_as_current_span("send followup", kind=SpanKind.PRODUCER):
        headers = {}
        inject(headers)  # carry the producer's context in the message
        queue.put({"headers": headers, "payload": payload})

def handle_followup(message):
    producer_ctx = trace.get_current_span(extract(message["headers"])).get_span_context()
    # new trace for the job, linked to the span that queued it
    with tracer.start_as_current_span("process followup", context=Context(), kind=SpanKind.CONSUMER,
                                      links=[Link(producer_ctx)]):
        process(message["payload"])
Question 37MediumCost

Your 'average cost per agent task' dashboard is built by summing gen_ai.usage.* over every span in each stored trace and averaging over traces. It shows about 3.4 times the figure you get from the invoice divided by the number of tasks. Explain the gap and how to attribute cost per task correctly.

#

The agent framework puts each invocation's token totals on its invoke_agent span, and the model-call spans under it also carry usage. Tail sampling keeps all failed runs and 10% of successful ones. 10% of runs fail, and a failed run costs about 3 times a successful one because of retries and longer loops.

Show answer

Two errors multiply.

1. Double counting. Usage appears on the agent span and again on the inference spans under it, so summing every span counts each token twice: a factor of 2. Count tokens at exactly one level: the inference spans (one per logical model call) or the usage counters. The conventions' agent metrics follow the same principle: each inference call is counted once across the call tree, against the invocation that issued it, not its parents.

2. Sampling bias. Let a successful run cost c. True mean = 0.9 x c + 0.1 x 3c = 1.2c. Stored traces are all failures (0.10 of runs) plus 10% of successes (0.09 of runs), so their mean = (0.09 x c + 0.10 x 3c) / 0.19 = 0.39c / 0.19 = 2.05c. That is 2.05 / 1.2 = 1.71 times too high.

Together: 2 x 1.71 = 3.42, the gap you see.

Doing it properly: - Emit one unsampled usage record per task when it finishes: task ID, outcome, tokens by model and token type, number of model and tool calls, paid tool costs. Send it to the warehouse, not to metric labels. - Compute cost per task and cost per successful task (total cost divided by successful tasks), which is the number that matters when comparing models or prompts. - Include failed and retried attempts: the token metric conventions say to record contributions from failed operations too, and they cost real money. - Use traces for explaining a single expensive run, and if you must estimate from sampled traces, weight each trace by the inverse of its sampling probability. - Reconcile the per-task totals with the invoice every day.

Trade-offs

Per-task records need a small pipeline but give exact, joinable numbers; trace-derived costs are convenient but only correct with careful deduplication and weighting.

Common mistake

Averaging over tail-sampled traces as if they were a random sample of tasks.

Follow-up

How would you show cost per successful task by model and prompt version, and decide when a cheaper model is really cheaper?

Question 38HardDebugging

A customer reports that your agent issued a wrong refund yesterday. Re-running the same request today takes a different path and works. How do you make production agent runs replayable for debugging, without replaying side effects?

#

The agent calls an order API, a policy lookup and a refund tool. Prompts and tool definitions change weekly. Traces hold metadata; a restricted store can hold content.

Show answer

Why a re-run does not reproduce: - Tool results are live data: the order status and refund policy may differ today. - Versions moved: prompt, tool definitions, model snapshot, configuration. - Model output varies: sampling differs between runs, and even at temperature 0 server-side inference is not guaranteed to be deterministic, because results can depend on batching (vLLM has a batch-invariance mode for self-hosted models that addresses this).

What to record per run, in the restricted content store, linked from the trace: - Every model request in full (system prompt, messages, tool definitions, model ID, parameters such as temperature and max tokens) and every response, including tool-use blocks. - Every tool call: arguments, result, error, and the tool call ID that links it to the model turn. The GenAI conventions have opt-in gen_ai.tool.call.arguments and gen_ai.tool.call.result attributes, but large or sensitive payloads belong in the content store with a reference. - Versions and flags: prompt version, tool schema version, feature flags, time.

Three replay modes: 1. Exact replay: step through the recorded responses and tool results without calling anything. This answers 'why did it refund', for example because the policy tool returned a stale rule. 2. Counterfactual replay: feed the recorded tool results back through mocked tools, and call the model again with a fixed prompt or a new model, to test the fix on the real failing case. 3. Resume from step k: start from the recorded state at a chosen step.

Safety: in replay, side-effecting tools must be stubbed, never pointed at production; key them by tool call ID so a misconfigured replay cannot issue a real refund. Durable workflow engines rely on the same idea: Temporal records each activity's result in the event history and reuses it on replay.

Because new model calls vary, run counterfactual replays several times and report how often the fix works, not one lucky run. Recorded content is sensitive: give it its own retention and access controls.

Trade-offs

Full recording makes incidents debuggable in minutes but creates a sensitive, costly store; record everything for flagged or failed runs and a sample of the rest.

Common mistake

Debugging by re-running the request against today's live tools and models, then concluding the bug cannot be reproduced.

Follow-up

How would you turn a replayed failing run into a permanent regression test?