Why your agent evals are lying to you

The final answer is the least interesting part

When a support agent resolves a ticket correctly, most eval suites mark it green and move on. But look inside the trace and you’ll often find a different story: three redundant retrievals, a tool call that silently failed, and a plan that only worked because the customer happened to phrase their question cleanly.

Outcome-only evals reward luck. They make your pass rate look stable right up until the day the inputs shift and the hidden failures surface all at once.

Score the path, not just the destination

Trajectory evals grade each step against what a good agent should have done: did it retrieve the right policy, call the right tool with valid arguments, and stop when it had enough information? In our own benchmarks, adding step-level scoring caught 3.4× more regressions than answer-level scoring alone.

• Grade tool selection and argument validity separately

• Penalize redundant steps, not just wrong ones

• Track cost and latency as first-class eval dimensions

None of this requires a new framework. If your agent is traced, you already have the data — you just need to score it.