Agent evaluation

2 min read
On trails:AGOPSstep 1/7

An agent has to be tested for goal achievement, the quality of the result, factual correctness, and safety according to the guardrails. A single prompt in and answer out is not enough, because an agent takes many steps and calls tools on the way. Testing it systematically means checking the path as well as the destination.

Flow and outcome

The path is the trajectory, the flow: which tools were called, in what order, with what arguments. Anthropic calls the complete record of one run a transcript (also called a trace or trajectory). The outcome is the final state left in the environment, which is not the same as what the agent says. A booking agent can announce that a flight is booked while no reservation exists.

A request, then step 1 and step 2 of tool calls, then the outcome. The steps together are the trajectory, the flow. The final state left in the environment is the outcome. trajectory: the flow Request user goal Step 1 tool call Step 2 tool call Outcome final state checked by tests that replay the steps checked by judge, human, user
The steps are the trajectory. What is left at the end is the outcome.
Trajectory (flow)Outcome
Looks atThe steps, tool calls and their argumentsThe result and the state it left
Tested byDev testing with tracing, pytest cases with mock toolsManual review of a test list, an LLM judge with a rubric, user feedback
CatchesWrong tool, wasted steps, loopsWrong or unsafe answer, goal not met

Two ways to test

The process has two parts. First, create a test dataset of real-world scenarios and test against it (see test datasets and mock toolsTest datasets and mock toolsReal scenarios as a dataset, and tests that swap the tools for fakes.Opens in a new tab). Second, use an evaluator LLM to score the result according to the evaluation criteria (see LLM as judgeLLM as judgeA second model scores the result against written criteria.Opens in a new tab). Output varies from run to run, so Anthropic runs several trials of each task instead of trusting one.

The catch

Quality of the result is only half of a pass. A correct answer that breaks a guardrail still fails, so safety is scored alongside quality (see guardrailsLLM guardrailsChecks around the model, so a bad input or a bad output is caught by something that does not rely on the model behaving.Opens in a new tab). And do not grade the path too rigidly. Anthropic’s advice is to grade what the agent produced, not the route it took, because agents find valid routes the test writer did not expect. Trajectory checks earn their place where the order of steps truly matters.

Sources