How to Evaluate an AI Agent: A Practical Framework

Why do most AI agents fail in production?

Most agents fail because teams confuse demos with evidence. A demo proves the agent can do the task once. An eval proves it does the task reliably across the distribution of real inputs.

RAND (2025) analysed 65 enterprise AI initiatives and found that 80.3% failed to deliver business value, twice the failure rate of non-AI IT projects. The breakdown:

• 33.8% abandoned before production
• 28.4% reach production but underdeliver
• 18.1% run but never recoup costs

The Sierra team's tau-bench paper (2024) shows the technical version of the same story: even GPT-4o solves fewer than 50% of retail tasks at pass^1, and fewer than 25% at pass^8 (eight independent attempts at the same task). Frontier agents are not reliable by default. Without an eval harness, you cannot detect regressions, you cannot defend a prompt change, and you cannot tell a stakeholder why your agent broke. The framework below fixes that.

What is the 5-step framework for evaluating an AI agent?

Run these five steps in order. Each one is non-negotiable.

1. Define one narrow task. A single agent, a single user goal, a single success definition. Multi-task agents need multi-task suites; do not fake it with one bucket.
2. Build a golden set of 50-200 real traces. Capture from production or seed from staging. Label expected tool calls and end states.
3. Pick metrics across four axes: task completion, tool selection accuracy, trajectory quality, cost/latency. Each axis has its own pass threshold.
4. Run the suite in CI on every PR. Use pytest-evals or DeepEval. Cache LLM responses. Fail the build if scores drop below threshold.
5. Gate merges on the threshold. No green eval, no merge. Track score history per metric so you can diff PR-over-PR.

The rest of this article walks each step, with the YAML config, the Python harness, and the CI job we actually ship.

Step 1: How do you define the task an agent is being evaluated on?

Pick one task with one success state. The task definition is the spec. If you cannot write the success criterion in two sentences, your agent does not know what success means either.

A good task definition has four fields:

• Inputs: schema of the user message and any context (user_id, account state, locale).
• Tools available: the exact tool list the agent is given at runtime.
• Policy constraints: things the agent must never do (charge twice, leak PII, exceed refund limits).
• Success state: a deterministic check, ideally a database diff or a structured output match.

Example for a refund agent:

Task: Issue a refund per the policy in refund_policy.md. Success: orders.status == 'refunded' AND transactions.amount_cents matches the requested amount AND no PII appears in the agent's final message.

This is the tau-bench approach: grade by end-state DB diff, not by judging the conversation. End-state grading is cheap, deterministic, and immune to LLM-judge drift.

Step 2: How do you build a golden set for AI agent evals?

Build a golden set of 50-200 hand-labelled traces before you write a single metric. This is the most important step, and the one teams skip.

Anthropic's eval team and Microsoft's Copilot team both recommend 100-150 examples as the minimum for a non-trivial domain. Diversity beats volume: a 100-case set covering 10 distinct failure modes is worth more than 1,000 cases of the same easy path.

How to source the set:

1. Pull 30-50 real traces from production (or staging if you are pre-launch).
2. Add 20-30 adversarial cases your team brainstormed: jailbreaks, ambiguous inputs, broken tool responses.
3. Add 20-30 edge cases discovered from incidents or bug reports.
4. Re-label every example with: task_id, inputs, expected_tools (ordered list), expected_end_state, policy_flags.

The Maxim AI golden-set guide (2026) recommends versioning the dataset in git and reviewing it like code. Treat it as living code: add a new case within 48 hours of every production incident, retire stale cases quarterly.

A minimal golden-set CSV row looks like:

task_id,user_input,expected_tools,expected_end_state_json,policy_flags
refund_001,"I want a refund for order 7782","[lookup_order,issue_refund]","{\"orders.status\":\"refunded\"}","no_pii"

What are the four metrics every AI agent eval needs?

Score every run across four orthogonal axes. Skipping an axis hides a failure mode.

Eval Axis	What it measures	Metric	Pass threshold
Task completion	Did the agent achieve the goal?	End-state diff vs golden state	>=85%
Tool selection accuracy	Right tool, right args?	Tool-name exact match + arg F1	>=90% / >=80%
Trajectory quality	Was the path efficient and policy-compliant?	LLM-as-judge, binary per turn	>=80% turns correct
Cost & latency	Is it economically shippable?	p50/p95 tokens, $/task, p95 ms	Set per task type

Why all four? A 2025 TRAJECT-Bench analysis found that agents with high outcome accuracy can still have low tool-selection accuracy: they brute-force the right answer through redundant tool calls, blowing latency and cost. Outcome alone does not catch this. Trajectory metrics do.

For tool selection, follow Galileo's 2026 framework: exact-match on the tool name plus an argument-level F1 score. For trajectory, score per turn with a binary pass/fail rubric, not a 1-5 scale. Microsoft's Azure AI Foundry team (2026) found LLM judges hit human-level agreement on binary calls and drift on fine-grained scales.

Should you use LLM-as-a-judge for agent evals?

Yes, but only for subjective dimensions, only with binary outputs, and only after calibrating against humans. Microsoft's evaluation-of-judges study (2026) found that frontier models hit human-level agreement on objective metrics like tool-call accuracy but become unreliable on fine-grained 1-5 scales.

The rules we follow:

• Use deterministic checks first. End-state diffs, JSON schema match, regex on PII -- always cheaper and more reliable than a judge.
• Use LLM-as-judge only for trajectory quality and policy compliance, where the answer is genuinely subjective.
• Force binary outputs. Pass/fail beats 1-5 for stability.
• Calibrate against 30-50 human-labelled examples. Compute Cohen's Kappa. If kappa < 0.7, rewrite the rubric.
• Use a 3-judge consensus for high-stakes evals. Three independent judges plus majority vote hits ~97% macro F1 in published research.

A solid judge prompt is short, has the rubric inline, gives one positive and one negative example, and asks for the verdict before the rationale. Long judge prompts produce verbose, biased rationales.

How do you configure an agent eval suite (YAML)?

Keep the eval config in a single YAML file checked into the repo. This is the contract between the eval harness and the team. We use this shape:

# evals/refund_agent.yaml
name: refund_agent_v3
task: refund
agent:
  entrypoint: agents.refund:run
  model: claude-sonnet-4.5
  tools: [lookup_order, issue_refund, send_email]

golden_set: evals/datasets/refund_v3.csv
min_cases: 120

metrics:
  - id: task_completion
    type: end_state_diff
    threshold: 0.85
  - id: tool_selection
    type: tool_match
    name_threshold: 0.90
    args_f1_threshold: 0.80
  - id: trajectory
    type: llm_judge
    judge_model: claude-sonnet-4.5
    rubric: evals/rubrics/refund_trajectory.md
    output: binary
    threshold: 0.80
  - id: cost
    type: budget
    p95_tokens: 8000
    p95_latency_ms: 12000
    max_dollars_per_task: 0.04

ci:
  fail_on: any_metric_below_threshold
  baseline_branch: main
  regression_tolerance: 0.02

The regression_tolerance matters: an eval can be noisy run-to-run, so we let scores drop up to 2 points absolute before failing the build. Anything bigger is a real regression.

How do you write an agent eval harness in Python?

Use pytest-evals or DeepEval. Both wrap the golden set as parametrized pytest cases and emit a structured report. A minimal harness using pytest-evals:

# evals/test_refund_agent.py
import pytest
from pytest_evals import eval_bag
from agents.refund import run as run_agent
from evals.judges import judge_trajectory
from evals.dataset import load_golden_set

GOLDEN = load_golden_set("evals/datasets/refund_v3.csv")

@pytest.mark.eval(name="refund_agent")
@pytest.mark.parametrize("case", GOLDEN, ids=[c.task_id for c in GOLDEN])
def test_refund_agent(case, eval_bag):
    result = run_agent(case.user_input, ctx=case.ctx)

    eval_bag.task_completion = result.end_state == case.expected_end_state
    eval_bag.tool_name_match = (
        [t.name for t in result.tool_calls] == case.expected_tools
    )
    eval_bag.args_f1 = arg_level_f1(result.tool_calls, case.expected_tool_args)
    eval_bag.trajectory_pass = judge_trajectory(
        result.trace, rubric_path="evals/rubrics/refund_trajectory.md"
    )
    eval_bag.tokens = result.usage.total_tokens
    eval_bag.latency_ms = result.latency_ms

Key patterns:

• Cache LLM responses to disk keyed by (model, prompt-hash, tool-call-hash). Without caching, every CI run costs the same as a full re-run. LangSmith's pytest integration does this out of the box.
• Run cases in parallel with pytest-xdist. Most agent calls are I/O-bound.
• Emit one JSON report per run with per-case scores, aggregate scores, and cost. This is what the dashboard reads.

How do you run agent evals in CI?

Wire the eval harness as a GitHub Action that runs on every PR touching agents/, prompts/, or evals/. Block the merge when any axis drops below threshold.

# .github/workflows/eval-on-pr.yml
name: eval-on-pr
on:
  pull_request:
    paths: ["agents/**", "prompts/**", "evals/**"]
jobs:
  evals:
    runs-on: ubuntu-latest
    timeout-minutes: 25
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -e ".[evals]"
      - name: Restore eval cache
        uses: actions/cache@v4
        with:
          path: .eval_cache
          key: evals-${{ hashFiles('evals/datasets/**') }}
      - name: Run eval suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: pytest evals/ -n 8 --eval-report=eval_report.json
      - name: Diff against main baseline
        run: python evals/scripts/diff_baseline.py eval_report.json
      - name: Comment results on PR
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: eval_report.md

The diff_baseline.py step is the gate: it pulls the latest main eval report from S3, computes the per-axis delta, and exits non-zero if any axis regressed beyond regression_tolerance. The PR comment renders the per-axis scores as a table next to main so reviewers see exactly what changed.

This is what "eval-on-PR" actually means. Not a Notion checklist. Not a manual run. A blocking CI check, every time.

What does an agent eval dashboard track?

Three things: per-axis score over time, cost per task, and the failing-cases drill-down. Everything else is decoration.

The minimum dashboard:

1. Score history, one line per axis (task completion, tool selection, trajectory, cost-pass-rate), x-axis is git SHA. The y-axis is the % of golden-set cases passing that axis. Annotate model and prompt changes.
2. Cost-per-task distribution. Histogram of $/task, labelled with median and p95. This is the metric finance asks about.
3. Failing-cases list. For the latest run, the 10 cases that failed any axis, with the trace, the expected outcome, and the actual outcome side by side. This is where debugging happens.

We build this in a thin Streamlit app that reads the JSON eval reports from S3, but Braintrust, LangSmith, and DeepEval all give you this out of the box. The vendor choice is less important than the rule: every eval run produces a structured report, every report ends up in the dashboard, every regression has an owner.

If you cannot answer "what was our task-completion score last Tuesday?" in 30 seconds, your dashboard is broken.

How do you keep an agent eval suite from rotting?

Eval rot is the silent killer. Suites pass forever, then production breaks, then nobody trusts the suite. Three rules prevent this.

1. Mine production for new cases weekly. Pull the 20 lowest-confidence traces, the 20 longest traces, and any trace flagged by an incident. Label them. Add them to the golden set. Anthropic's eval guide calls this "closing the loop" and identifies it as the single biggest predictor of long-term eval value.

2. Re-baseline whenever the world moves. Re-run the full suite on a fresh model (Claude 4.5 -> 5, GPT-4o -> 5) within 24 hours of release. Re-baseline whenever you change the prompt, the tool schema, or the policy doc. Without re-baselines, you are comparing against a ghost.

3. Assign an eval owner. One person owns the harness and the dataset. They review new cases, kill flaky ones, and report aggregate trends in weekly engineering review. No owner means no eval. Anthropic's recommendation: a dedicated evals team owns infrastructure, while domain experts contribute cases.

The goal is not a perfect score. The goal is a trusted signal that survives team turnover, model upgrades, and prompt rewrites.

Eval Axis	What it measures	Recommended Metric	Pass threshold (starting point)
Task completion	Did the agent achieve the user's goal?	End-state DB diff vs golden state (tau-bench style)	>=85% on golden set
Tool selection accuracy	Did the agent call the right tool with the right args?	Exact-match on tool name + arg-level F1	>=90% tool name, >=80% args
Trajectory quality	Was the path efficient and policy-compliant?	LLM-as-judge rubric, binary pass/fail per turn	>=80% turns rated correct
Cost & latency	Is the agent shippable economically?	p50/p95 tokens, $/task, p95 wall-clock	$ and ms budget per task type