Day 34 Theory — Advanced Agent Evaluation: Task, Tool, Trajectory and Judge Quality

Why final-answer evaluation is not enough

For a normal text-generation feature, we may care mainly about the quality of the final response.

An agent is different because it can take multiple actions before producing that response.

An agent may eventually return a correct answer after:

• calling the wrong tool,
• exposing sensitive information,
• making an unnecessary expensive model call,
• retrying a side effect,
• skipping approval,
• taking an unsafe path and then recovering.

If we evaluate only the last sentence, we can miss serious failures.

Agent evaluation therefore asks two questions:

1. Did the user get an acceptable outcome?
2. Did the system reach that outcome through an acceptable process?

Four evaluation layers

A useful decomposition is:

1. Task outcome

Did the user's goal succeed?

Examples:

• ticket classified correctly,
• password-reset request rejected when unauthorized,
• certificate renewal completed,
• answer grounded in approved documentation.

2. Tool behavior

Did the agent choose the right tool and call it correctly?

Check:

• tool selection,
• arguments,
• call count,
• authorization preconditions,
• duplicate side effects.

3. Trajectory

The trajectory is the sequence of important actions/decisions taken during the run.

For example:

classify
 -> search_kb
 -> verify_identity
 -> request_approval
 -> reset_password
 -> answer

Trajectory evaluation asks whether the path was safe, bounded and appropriate.

4. Final response quality

Was the response:

• correct,
• useful,
• grounded,
• appropriately concise,
• clear about uncertainty,
• free from sensitive data leakage?

These layers should not be collapsed into one vague score.

Exact trajectory versus acceptable trajectory

Suppose two agent runs both solve a ticket.

Run A:

classify -> retrieve -> answer

Run B:

classify -> inspect_ticket -> retrieve -> answer

Both may be valid.

If our evaluator requires one exact sequence, it can punish legitimate alternatives.

A better approach is often to define trajectory invariants.

Examples:

• identity verification must occur before privileged reset,
• approval must occur before destructive action,
• only approved tools may be called,
• no tool may be called more than the allowed number of times,
• retrieval-backed answers must cite approved evidence.

This tests what matters while allowing more than one good path.

Deterministic checks should stay deterministic

If a property can be checked exactly, do not ask an LLM judge to guess it.

Examples of deterministic checks:

assert "reset_password" not in tool_calls_without_approval
assert total_tool_calls <= configured_limit
assert response_schema_is_valid
assert tenant_id == expected_tenant

Model-based judges are most valuable when the property itself is subjective or semantic.

For example:

• answer helpfulness,
• grounded explanation quality,
• whether a summary captures the essential incident details.

This gives us an evaluation stack:

exact policy checks
      +
semantic metrics
      +
LLM judge where needed
      +
human review for calibration

LLM-as-a-judge

An LLM judge is another model asked to evaluate an output using a rubric.

A judge might receive:

User request
Reference evidence
Agent response
Rubric

and return dimensions such as:

{
  "grounded": 4,
  "complete": 3,
  "safe": 5,
  "reason": "..."
}

This can scale evaluation of properties that are difficult to express with exact rules.

But the judge is itself probabilistic.

A judge score is measurement produced by another model, not ground truth.

Judge failure modes

Positional bias

When comparing two responses, a judge may systematically favor the first or second position.

Mitigation: randomize or swap presentation order and compare stability.

Verbosity bias

A judge may prefer a longer answer because it looks more complete even when the extra text is irrelevant.

Style contamination

Polished wording can influence a judge even when factual content is weaker.

Reference leakage or prompt injection

If untrusted content is inserted into the judge prompt, it may try to influence the evaluator.

Evaluation pipelines need the same input-trust discipline as production pipelines.

Inconsistent scoring

The same example may receive materially different scores across repeated judge calls or model versions.

That is why judge quality needs measurement.

Calibrating the judge against humans

Create a set of examples labelled by qualified human reviewers.

Then compare judge decisions against those labels.

Questions include:

• How often does the judge agree with reviewers?
• Where does it systematically disagree?
• Does it preserve ranking between clearly good and bad examples?
• Does agreement differ by category?

Do not validate the judge only on easy examples.

Include difficult borderline cases.

Pairwise evaluation

Sometimes asking:

Which of response A or B is better according to this rubric?

is easier than asking for an absolute 1–5 score.

Pairwise evaluation is useful when comparing:

• prompt versions,
• model versions,
• retrieval strategies,
• agent policies.

Randomize A/B order to reduce positional bias.

Pairwise preference still needs calibration if it is used as a release gate.

Evaluation datasets need slices

An aggregate score can hide important failures.

Suppose overall success is high, but security-ticket performance is poor.

We therefore define slices such as:

password reset
VPN/certificate
knowledge question
security incident
ambiguous request
unauthorized request
prompt-injection case

Track results per slice.

A safety-critical slice can have a stricter release requirement than a low-risk informational slice.

Evaluate failures, not only happy paths

A production agent should also be tested when:

• tool times out,
• retriever returns no documents,
• approval is denied,
• user is unauthorized,
• model requests an unknown tool,
• model repeats the same call,
• remote dependency returns malformed data.

The evaluation target is the system's behavior under failure, not merely whether the model can solve ideal examples.

Offline evaluation versus production monitoring

Offline evals run on a controlled dataset before release.

Production monitoring observes real behavior after release.

They answer different questions.

Offline eval:
"Did this candidate version pass our known tests?"

Production monitoring:
"What is happening to real users now?"

Use both.

Production failures should feed back into the offline regression set when appropriate.

Evaluating trajectories without storing private chain-of-thought

Trajectory evaluation does not require private model reasoning.

Evaluate observable runtime events such as:

• agent selected,
• tool requested,
• arguments after validation/redaction,
• tool result category,
• handoff,
• approval event,
• task state,
• final output.

This is operational evidence, not hidden chain-of-thought.

A Service Desk evaluation example

User request:

"Reset the password for employee E-42."

A useful evaluation may check:

Task outcome:
- reset succeeded only if user authorized

Tool behavior:
- verify_identity called
- reset_password called at most once

Trajectory invariant:
- verify_identity occurs before reset_password
- approval occurs before reset if policy requires it

Final answer:
- confirms result without leaking credentials

Now imagine the final answer says “Password reset successfully” but the tool was called before identity verification.

Final-answer-only evaluation would pass.

Trajectory-aware evaluation correctly fails the run.

Release gates need interpretable evidence

A release should not depend on one opaque score such as:

agent_quality = 0.87

Prefer a report that explains which requirements passed or failed:

Task success: 94/100
Unauthorized action prevention: 20/20
Required approval ordering: 19/20  <- blocker
Grounded-answer judge agreement: 92%
Tool argument schema validity: 100/100

This gives engineers something actionable to fix.

Service Desk connection

Today the Service Desk moves from “does the answer look good?” to a multi-layer behavioral evaluation system.

The architectural change is:

Before
------
final response checks

After
-----
task outcome
+ tool behavior
+ trajectory invariants
+ final response quality
+ calibrated model judges where appropriate

The principle is:

Evaluate what the agent did, not only what it said. Keep exact checks deterministic, use model judges only where judgment is genuinely needed, and validate the evaluator before trusting its score.