supraj.dev

AI Agents · Explainer · Agent Harness Engineering, Part 3

How do you test an agent harness?Think of a test kitchen.

Part 2 built a harness that checks one run. This part checks the agent itself, many times, before your users do.

Supraj Maripeddi · · 15 min read

Cover: one task card goes to an agent five times. Four trials pass and one fails, and three graders, a code check, a model grader and a human, score the results before a gate.
Start hereRead Parts 1 and 2 first, or know what an agent harness is.
Leave knowingWhat an eval is, how it's graded, and how it becomes a CI gate.
Keep in mindA passing eval only covers the tasks you wrote down.

At the end of Part 2, the harness caught the model trying to edit the tests, escape its folder and say "done" too early. Then the tests passed, and the run was marked done.

That's one run, of one task, on one day. Here's the question this part answers: how do you know the agent will do it again tomorrow, on a different task, after you change a prompt or switch to a new model?

The answer is an eval. I'll use one picture all the way through: a test kitchen, where a restaurant tries out a cook and a menu before any guest eats. The facts about evals here come from Anthropic's engineering guide to agent evals and from the docs of Inspect, an open-source eval framework. [1] [2]

01 / Beyond unit tests

Part 2 already runs the tests. Why isn't that enough?

Because the tests in Part 2 check the code. They don't check the agent.

The done gate in Part 2 runs test_pricing.py and asks one question: is this fix correct? That's a unit test, and it's exactly right for that job. But it says nothing about how often the agent gets there, how many steps it burns, or what it does on a bug it has never seen.

Three things make an agent harder to test than a function.

  • It varies. Model outputs vary between runs, so the same task can pass today and fail tomorrow. [1]
  • It takes many steps. Agents use tools across many turns, change state in the environment and adapt as they go. So mistakes can spread and compound. [1]
  • It acts on the world. What matters is often a change it made, such as a file edited or a booking created, not just the words it wrote at the end. [1]

There's one more point, and it ties this series together. Anthropic's guide says that when you evaluate "an agent", you're evaluating the harness and the model working together. [1] So an eval doesn't only grade the model. It grades every part of the harness you built in Part 2.

Part 2's last chapter sketched the idea: collect real tasks, run each several times, and change one harness piece at a time. This part turns that sketch into a proper eval.

In the test kitchen
Checking that the oven reaches 180 degrees is a fair test of the oven. It doesn't tell you whether the new cook can make tonight's menu.
In the system
A unit test checks one piece of code. An eval checks the model and the harness working together, over many steps and many runs.

02 / The parts of an eval

So what is an eval actually made of?

At its simplest, an eval is a test for an AI system. You give it an input, then apply grading logic to what it produces to measure success. [1] For agents, Anthropic's guide uses a handful of words. They're worth learning, because most eval tools use something close to them. [1]

Word What it means In the kitchen
Task One test with defined inputs and success criteria. An order ticket: one dish, and what good looks like.
Trial One attempt at a task. You usually run several. One time the cook makes that dish.
Grader Logic that scores some part of the agent's work. A task can have several. Whoever or whatever judges the dish.
Transcript The complete record of a trial: outputs, tool calls, reasoning and intermediate results. Also called a trace or trajectory. A video of the cook's whole shift.
Outcome The final state of the environment at the end of the trial. The plate that actually reaches the pass.
Evaluation harness The system that runs evals end to end: it runs tasks, records every step, grades the results and adds them up. The kitchen manager who runs the whole tasting.
Evaluation suite A collection of tasks that share a broad goal. The full tasting menu.

The difference between the transcript and the outcome is the bit that's easy to miss. Anthropic's example is a flight-booking agent. It might say "Your flight has been booked" at the end of the transcript. But the outcome is whether a reservation exists in the environment's database. [1]

Notice that there are now two harnesses. Your agent harness is the one from Part 2. The evaluation harness wraps around it, runs it many times and keeps score.

The environment: reset the kitchen every time

A task also needs somewhere to run. Anthropic's advice is that each trial should start from a clean environment. Leftover files, cached data or resources running out can make trials fail together for reasons that have nothing to do with the agent. [1]

Shared state can also make an agent look better than it is. The guide describes internal evals where Claude gained an unfair advantage on some tasks by reading the git history left by earlier trials. [1] In kitchen terms, the cook found yesterday's finished dish in the fridge.

Inspect, the framework from the UK AI Security Institute and Meridian Labs, uses its own names for the same pieces. A Task brings together a dataset of samples, a solver that produces an answer, and a scorer that evaluates it. The solver can be a full agent that uses tools over many turns, and a task can run the agent's tools in an isolated Docker container. [2]

In the test kitchen
An order ticket, a clean station with fresh ingredients, a cook, and a judge for the plate.
In the system
A task, a fresh environment, the agent with its harness, and one or more graders.

03 / Who grades

Who decides whether the dish is good?

Grading is where most of the design work goes. Anthropic's guide describes three kinds of grader: code-based, model-based and human. Each one looks at some part of the transcript or the outcome. [1]

Figure 1 / Three kinds of grader

Pick a grader to see what it's good and bad at.

Code-based grader

Checks with: string matches, pass or fail tests, static analysis, outcome checks, tool-call checks.

Good at: fast, cheap, objective, reproducible, easy to debug.

Bad at: brittle when a valid answer doesn't match the expected pattern, and weak on subjective work.

For Part 2's task: run the original tests on the agent's code.

Methods, strengths and weaknesses are shortened from the tables in Anthropic's guide. [1] The kitchen labels and the Part 2 examples are mine.

Code-based graders are the thermometer and the scale. They don't have opinions. They check whether the tests pass, whether a file exists, or whether the agent called a certain tool with certain parameters. [1] For coding agents they're the natural choice, because software is fairly easy to check: does the code run, and do the tests pass? [1]

Model-based graders are a trained taster with a written scorecard. A second model reads the work and scores it against a rubric, or compares two answers. They handle open-ended work that no regex can judge. But they're not deterministic, they cost more than code, and they need checking against human graders to be accurate. [1]

Human graders are the head chef. Expert review is the gold standard, and it's what you use to calibrate model graders. It's also expensive and slow. [1]

Anthropic's recommendation is clear: use code-based graders where you can, model graders where you must, and human graders carefully, to validate the rest. [1] A task can combine them. The scores can be weighted, every grader can be required to pass, or you can mix the two. [1]

Two tips from the guide make model graders more trustworthy. [1]

  • Give the judge a way out. Tell it to answer "Unknown" when it doesn't have enough information, so it doesn't make something up.
  • One judge per question. Write a clear rubric for each dimension of the task, and grade each one with a separate model call instead of one call for everything.

Inspect ships both kinds as ready-made scorers. includes() checks whether the target appears in the output, and match() checks for it at a known position. model_graded_qa() has another model judge whether the output is a correct answer, using guidance in the target. [3] A custom scorer can also read files from the agent's sandbox, so it can grade what the agent actually changed. [5]

In the test kitchen
The thermometer is cheap and never argues. The head chef is right but busy. The taster sits in between, and the chef checks the taster now and then.
In the system
Code graders are fast and exact. Humans are the gold standard. Model graders scale human judgment, once they agree with humans.

04 / Plate or cook

Should you grade the plate, or watch the cook?

Your first instinct might be to write down the "right" steps and check the agent followed them: read the file, then edit it, then run the tests, in that order.

Anthropic found that approach too rigid. Agents regularly find valid approaches the eval designers didn't expect, and step-by-step checks punish them for it. So it's often better to grade what the agent produced, not the path it took. [1]

There's a striking example in the guide. Opus 4.5 solved a τ2-bench problem about booking a flight by finding a loophole in the policy. It "failed" the eval as written, but it came up with a better solution for the user. [1]

That doesn't make the transcript useless. Once the outcome checks are in place, grading the transcript adds a second view. Code-quality rules can judge the code beyond passing tests, and model graders with a clear rubric can judge how the agent called its tools. [1] Simple numbers from the transcript, such as turns, tool calls and tokens, can be tracked too. [1]

Part 2's harness already writes a transcript: runs/<id>.trace.jsonl, one line per action. That file is now an eval input. For the discount bug, you might grade it like this:

Look at Check Kind
Outcome The original tests pass on the agent's pricing.py. Code, must pass
Outcome The files under tests/ are unchanged. Code, must pass
Transcript How many steps it took, and how many calls the harness refused. Code, tracked
Outcome Is the fix clear and minimal? A rubric, scored by a second model. Model, tracked

Then read the transcripts yourself. Anthropic is blunt about this: you won't know whether your graders work unless you read the transcripts and grades from many trials. When a task fails, the transcript tells you whether the agent made a real mistake or your grader rejected a valid solution. [1] Inspect has a viewer for this, started with inspect view, for looking at results and individual transcripts. [2]

In the test kitchen
Judge the plate. Two cooks can make the same good dish in a different order. Watch the video to learn why a plate failed.
In the system
Grade the outcome first. Use the transcript for extra checks, cost numbers, and to find out why a trial failed.

05 / Many trials

Why cook the same dish five times?

Because one good plate doesn't tell you much. A cook who gets the dish right on Monday can burn it on Tuesday.

Agents are the same. Each task has its own success rate, maybe 90% on one and 50% on another, and a task that passed on one eval run might fail on the next. [1] So you run several trials of each task and look at the pattern.

Two numbers describe that pattern, and they answer different questions. [1]

  • pass@k is the chance that the agent gets at least one correct solution in k attempts. It goes up as k grows. Use it when one success is enough.
  • pass^k is the chance that all k trials succeed. It goes down as k grows. Use it when users expect the agent to get it right every time.

The guide's example: an agent with a 75% success rate per trial passes all three of three trials only about 42% of the time, because 0.75 × 0.75 × 0.75 ≈ 0.42. [1] Try other values of k below.

Figure 2 / One cook, many plates

Change the number of trials and watch the two numbers split.

Per-trial success rate: 75%

pass@k: at least one of k passes 98.4%
pass^k: all k pass 42.2%

With 3 trials: pass@3 is 98.4% and pass^3 is 42.2%. Almost always at least one good plate, but all three good less than half the time.

Worked arithmetic, not measured data. It assumes every trial is independent with the same 75% chance of passing: pass@k = 1 − 0.25k and pass^k = 0.75k. The 75% and three-trial example come from Anthropic's guide. [1]

For a coding agent like the one in Part 2, Anthropic notes you're often most interested in the first try, pass@1. For a customer-facing agent, pass^k matters more. [1]

Inspect calls repeated trials epochs: the number of times to run each sample. [6] Several epochs give several scores per sample, and by default Inspect combines them with the mean. You can choose other reducers, including pass_at_{k}, pass_k_{k} for "all k succeed", and at_least_{k}. [4]

In the test kitchen
A cook who nails the dish once in five tries has talent. A cook who nails it five times in five is ready for Saturday night.
In the system
pass@k tells you the agent can do it. pass^k tells you it does it reliably.

06 / Capability and regression

Is this test for new dishes, or for the old menu?

A test kitchen does two different jobs. It tries out dishes the cook can't make yet. And it checks that the dishes already on the menu still come out right. Evals split the same way. [1]

  • Capability evals ask "What can this agent do well?" They should start with a low pass rate, on tasks the agent struggles with, so the team has a hill to climb.
  • Regression evals ask "Does the agent still handle all the tasks it used to?" They should pass nearly 100% of the time. A drop means something broke.

They work together. While you improve the agent on capability evals, you keep running regression evals to make sure the change didn't break something else. And once a capability eval passes reliably, it can "graduate" into the regression suite. [1] "Can we do this at all?" becomes "Can we still do this reliably?"

Watch for one trap: saturation. When an agent passes every solvable task in an eval, the eval stops showing improvement. At 100%, it still catches regressions, but it tells you nothing about getting better. [1] That's the moment to write harder capability tasks.

Anthropic recommends going one step further, with what it calls eval-driven development: write the evals for a capability before the agent can do it, then iterate until it can. When a new model comes out, running the suite shows quickly which bets paid off. [1]

In the test kitchen
New dishes are practice. Menu dishes are a promise. A new dish joins the menu only once it comes out right every time.
In the system
Capability evals start low and climb. Regression evals stay near 100%. Capability tasks graduate into regression.

07 / From complaint to gate

How does a complaint become a gate?

Teams often put off evals because they think they need hundreds of tasks. Anthropic's guide says 20 to 50 simple tasks drawn from real failures is a great start. [1] Early on, each change to an agent tends to have a big, visible effect, so a small set is enough to see it. [1]

Where the tasks come from

Start with the checks you already run by hand before a release. If the agent is already in use, look at your bug tracker and support queue. Turning user-reported failures into test cases keeps the suite close to how people really use the agent. [1]

In kitchen terms: a guest sends back a dish because the sauce split. That complaint becomes a new order ticket in the test kitchen, and every new cook makes that sauce before they work a real shift.

Then make each task fair. The guide gives four rules. [1]

  1. Two experts, one verdict. A good task is one where two experts would independently reach the same pass or fail verdict.
  2. A reference solution. A known working answer that passes all graders proves the task can be solved and the graders are set up right.
  3. Both directions. Test when a behaviour should happen and when it shouldn't. If you only test whether the agent searches when it should, you can end up with an agent that searches for almost everything.
  4. No cheating. Passing should require solving the problem, not exploiting a loophole in the grader.

That last rule is Part 2 all over again. The scripted model in Part 2 tried to rewrite the tests. In an eval, the grader must never trust the agent's copy of the tests. It brings its own.

One more signal from the guide: with frontier models, a task that fails every one of many trials is most often a broken task, not an incapable agent. Check the task and its graders first. [1]

A sketch: evals for the Part 2 harness

Here's what an evaluation harness around Part 2's harness.py could look like. It's an illustration of the shape, not a finished tool. Each task folder holds a starting repository and a task.json with the prompt.

# evals.py: a sketch of an evaluation harness around Part 2's harness.py
import json
import pathlib
import shutil

import harness

TASKS = pathlib.Path("eval_tasks")    # eval_tasks/<name>/repo/ and eval_tasks/<name>/task.json

def fresh_workspace(task_dir):
    """Every trial starts from a clean copy. No leftovers from the last one."""
    shutil.rmtree(harness.WORKSPACE, ignore_errors=True)
    shutil.copytree(task_dir / "repo", harness.WORKSPACE)

def grade_outcome(task_dir):
    """Code grader: put the original tests back, then run them. Never trust the agent's copy."""
    shutil.rmtree(harness.WORKSPACE / "tests", ignore_errors=True)
    shutil.copytree(task_dir / "repo" / "tests", harness.WORKSPACE / "tests")
    return harness.run_tests().startswith("PASSED")

def read_transcript(state):
    """Tracked numbers from the trace Part 2 already writes."""
    lines = (harness.RUNS / f"{state['run_id']}.trace.jsonl").read_text().splitlines()
    events = [json.loads(line) for line in lines]
    return {"steps": state["step"], "refused": sum(1 for e in events if e.get("ok") is False)}

def run_suite(trials=5):
    scores = {}
    for task_dir in sorted(p for p in TASKS.iterdir() if p.is_dir()):
        prompt = json.loads((task_dir / "task.json").read_text())["prompt"]
        passed = 0
        for _ in range(trials):
            fresh_workspace(task_dir)
            state = harness.run(prompt)
            passed += grade_outcome(task_dir)
            print(task_dir.name, state["status"], read_transcript(state))
        scores[task_dir.name] = passed / trials
    return scores

if __name__ == "__main__":
    scores = run_suite()
    print(scores)
    if min(scores.values()) < 1.0:     # a regression suite: every trial of every task must pass
        raise SystemExit(1)            # a non-zero exit fails the CI job

Read it as the kitchen. fresh_workspace resets the station. grade_outcome is the thermometer, and it uses its own tests. read_transcript watches the video for numbers worth tracking. The last line is the gate.

The gate

Anthropic describes automated evals as especially useful before launch and in CI/CD, running on each agent change and model upgrade as the first line of defence against quality problems. [1] So the regression suite runs in CI like any other check. A new tool description, a context change or a new model only merges if the suite still passes. If you're new to pipelines, the CI/CD explainer covers the basics.

Every trial is a real run of the model, so the gate costs money and time. Five trials of thirty tasks is 150 agent runs. That's a reason to keep the regression suite small and sharp, and to run larger capability suites less often.

In the test kitchen
A complaint becomes an order ticket. No recipe change reaches the menu until every ticket passes the tasting.
In the system
A real failure becomes an eval task. No harness or model change merges until the regression suite passes in CI.

08 / What evals can't prove

If every eval passes, is the agent safe to ship?

No. A passing suite tells you the agent handled the tasks you wrote down, in the environment you built. It can't tell you about the task nobody thought of.

Anthropic lists the costs plainly. Automated evals need real effort up front, they need upkeep as the product and the model change, and they can create false confidence if they don't match how people actually use the agent. [1]

The eval itself can also be wrong. The guide describes Opus 4.5 first scoring 42% on CORE-Bench. An Anthropic researcher then found problems in the eval: grading so rigid it marked "96.12" wrong when expecting "96.124991…", unclear task specs, and tasks that couldn't be reproduced exactly. After fixing those bugs and using a less constrained scaffold, the score went to 95%. [1] Anthropic's rule is not to take eval scores at face value until someone has dug into the eval and read some transcripts. [1]

So evals are one layer, not the whole check. The guide compares this to the Swiss Cheese Model from safety engineering: no single layer catches every problem, but failures that slip through one layer are caught by another. [1] The other layers it lists are production monitoring, user feedback, A/B testing, manual transcript review and systematic human studies. [1]

The layers

When each method catches problems

  • Automated evalsbefore launch and in CI, on every change
  • Production monitoringafter launch, on real traffic
  • A/B testingbig changes, once there's enough traffic
  • User feedback and transcript reviewall the time, to fill gaps
  • Systematic human studiesto calibrate model graders
Shortened from the section of Anthropic's guide on how evals fit with other methods. [1]

This is also why Part 2's guards stay in place. An eval can show that the agent usually leaves the tests alone. Only the policy in code makes sure it can't touch them.

In the test kitchen
A perfect tasting doesn't guarantee a perfect Saturday rush. You still listen to guests, and you still keep the fire blanket on the wall.
In the system
Passing evals don't prove the agent is right everywhere. Keep monitoring, feedback and the harness guards.

Keep these names

A small glossary

Eval
A test for an AI system: give it an input, then apply grading logic to its output to measure success. [1]
Task
A single test with defined inputs and success criteria. Also called a problem or test case. [1]
Trial
One attempt at a task. [1]
Grader
Logic that scores some aspect of the agent's performance. Inspect calls it a scorer. [1] [3]
Transcript
The complete record of a trial, including outputs, tool calls, reasoning and intermediate results. Also called a trace or trajectory. [1]
Outcome
The final state of the environment at the end of a trial. [1]
Evaluation harness
The infrastructure that runs evals end to end and adds up the results. [1]
Capability eval
An eval that asks what the agent can do well. It should start at a low pass rate. [1]
Regression eval
An eval that asks whether the agent still handles the tasks it used to. It should pass nearly 100% of the time. [1]
pass@k and pass^k
The chance of at least one success in k trials, and the chance that all k trials succeed. [1]
Saturation
When an agent passes all the solvable tasks in an eval, leaving no room to show improvement. [1]
Epoch
Inspect's word for one run of each sample. Several epochs give several scores per sample. [4] [6]

Check the concept

Can you explain it back?

1. Part 2's harness already runs the tests. What does an eval add?

The tests check one fix in one run. An eval runs the model and harness together on many tasks, several times each, and scores how often and how well the agent succeeds.

2. The agent says "Booked!" but no booking exists. Which did you grade?

The transcript. The outcome is the final state of the environment, and there the booking doesn't exist, so the trial fails.

3. When would you use a model-based grader instead of code?

When the work is open-ended and no fixed check can judge it, such as the quality of an explanation. Calibrate it against human graders first.

4. Your agent passes 75% of trials. Is pass@3 or pass^3 higher?

pass@3. At least one success in three is very likely. All three succeeding is only about 42%, assuming the trials are independent.

5. A regression eval drops from 100% to 80%. What does that tell you?

Something that used to work is broken. Read the failing transcripts to see whether the agent got worse or the eval itself has a problem.

6. Every eval passes. Can you remove the policy that protects the tests?

No. Evals only cover the tasks you wrote. The policy in code makes the shortcut impossible on every run, including the ones no eval covers.

Continue the thread

Go back a step, or forward to production.

This is Part 3 of Agent Harness Engineering. Part 1 explains what a harness is, and Part 2 builds the one these evals test. For what happens after launch, read AI agents in production, and for the rules that sit around an agent, read AI guardrails.

Read further

Sources

Sources accessed 18 November 2026.

  1. Demystifying evals for AI agents, Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares and Jiri De Jonghe, Anthropic, 9 January 2026.
  2. Inspect documentation: Welcome. UK AI Security Institute.
  3. Inspect documentation: Scorers.
  4. Inspect documentation: Scoring Metrics, Reducing Epochs.
  5. Inspect documentation: Multiple Scorers, Sandbox Access.
  6. Inspect documentation: Tasks.

Written with AI assistance. Technical sources are listed above.