AI Agent Evaluation: Test Before You Trust

An AI agent that impresses in a demo can still fail in daily use. It answers the common questions well, then misreads an unusual request, picks the wrong tool or invents a detail when the data is missing. Because language models are not deterministic, the same input can produce different outputs on different runs, and a small prompt change can improve one behaviour while breaking another. AI agent evaluation is how you find out, before users do, whether an agent does its job reliably enough. This guide explains how to define success, build test cases, grade results and keep evaluating as prompts, tools and models change.

Why agents need structured evaluation

Traditional software tests check that a given input always produces the same output. Agents do not work that way, and their tasks often have more than one acceptable answer. The OpenAI guide to evaluation best practices describes evals as structured tests for measuring a model's performance that help ensure accuracy, performance and reliability despite the nondeterministic nature of AI systems (OpenAI: Evaluation best practices). The same guide lists "vibe-based evals", meaning using "it seems like it's working" as a strategy, as an anti-pattern.

For an agent, evaluation must cover more than the final answer. It should check which tools were called, with which arguments, in what order, and whether the agent stopped or asked for help when it should have.

Start with success criteria

Anthropic's guide to building evaluations recommends defining success criteria that are specific and measurable before writing tests (Claude: Define success criteria and build evaluations). For an agent, write criteria for each job it does, such as:

• Correctness: the answer or action matches the data in the source system
• Tool use: the right tool is called with valid arguments, and no unnecessary or risky tools are called
• Safety: the agent refuses or escalates requests outside its scope
• Grounding: claims are supported by retrieved data, and the agent says when it does not know
• Format: outputs follow the structure downstream systems expect

Agree the criteria with the people who own the process, not just the developers. Their idea of a good answer is the one that matters.

AI agent evaluation test cases

A useful test set mirrors the real work. Anthropic's guide advises designing evals that reflect your real-world task distribution and not forgetting edge cases, such as irrelevant or nonexistent input data, overly long input, poor or harmful user input and ambiguous cases where even humans would disagree. Sources for test cases include:

• Real requests from logs, with personal data removed
• Cases written by the process owners, covering rules they care about
• Past incidents and complaints, each turned into a test
• Deliberate attempts to push the agent outside its scope

The OpenAI guide also recommends logging as you develop so you can mine your logs for good eval cases. Keep a held-out set that you do not tune prompts against, so you can check that improvements generalise.

Grade automatically where you can

Manual review does not scale to hundreds of cases on every change. Both guides recommend structuring evaluations to allow automated scoring. Anthropic's guide lists approaches such as multiple choice, string match, code-graded and LLM-graded checks, and advises prioritising volume over quality: more questions with slightly lower-signal automated grading beat fewer hand-graded ones.

In practice, combine methods:

• Code checks for anything exact: was the right tool called, is the output valid JSON, does a field match the source record
• Model-based grading for open-ended answers, using a clear rubric and a separate grading prompt
• Human review for a sample, especially for new behaviours and borderline cases

The OpenAI guide recommends using human feedback to calibrate automated scoring. Check regularly that the automated grader agrees with people on a sample of cases.

Worked example: a hypothetical evaluation plan

The table below is a hypothetical evaluation plan for an agent that answers order status questions and creates follow-up tasks in a CRM. It illustrates how to match checks to criteria, not a tested suite.

CriterionExample test caseGrading methodPass ruleCorrect statusCustomer asks about a shipped orderCode: compare answer with order recordStatus matchesRight toolRequest needs order lookupCode: inspect tool callsLookup called with correct order numberNo invented dataOrder number does not existModel grader with rubricSays it cannot find the orderScopeCustomer asks for a refundCode: no refund tool called; escalation flag setEscalated to a personTask creationSales rep asks for a reminder next TuesdayCode: inspect task argumentsCorrect date and ownerToneFrustrated customerHuman review sampleReviewer approves

Evaluate on every change

Run the evaluation set whenever you change prompts, tools, retrieval settings or the model version. The OpenAI guide recommends adopting eval-driven development, evaluating early and often, and describes evaluation as a continuous process. Compare results with the previous version case by case, not just as an overall score, because a stable average can hide new failures. Because outputs vary, run important cases more than once and look at the spread of results.

Our guide to LLM tool calling explains how tool definitions affect behaviour, which is often what evaluations reveal needs fixing.

Keep watching after launch

Pre-release tests cannot cover everything. After launch, sample real conversations for review, track signals such as escalations, user corrections and complaints, and add new failure patterns to the test set. When the agent supports an operational team, such as IT support, ask the team to flag poor answers with a simple button. Our post on AI agents for IT support describes a gradual rollout that pairs well with this kind of monitoring.

Protect data in test sets

Test sets built from real logs can contain personal or confidential data. Remove or replace it before storing cases, restrict access to the evaluation data and apply the same retention rules as your other records. Synthetic cases written to resemble real ones are a good complement, and they let you cover rare situations that may not yet appear in your logs. Label each case with its source, so you know which ones came from real requests and which were written by hand.

Frequently Asked Questions

What is AI agent evaluation?

It is a structured way of testing whether an agent does its job reliably, using a set of test cases, defined success criteria and grading methods, run whenever the agent changes.

Can evaluation be fully automated?

Much of it can, with code checks and model-based grading. Keep human review for a sample and use it to check that automated grading agrees with people.

How many test cases do we need?

Enough to cover the main tasks, the rules that matter and known edge cases. Anthropic's guide favours more cases with automated grading over fewer hand-graded ones.

When should we rerun evaluations?

On every change to prompts, tools, retrieval or model version, and regularly after launch with new cases drawn from real use.

Ship agents you can trust

We design evaluation sets, grading and monitoring for AI agents so changes are tested before they reach users. See our AI agent orchestration and RAG service, or talk to us about AI agent evaluation.

About the Author

Manav Sharma is the founder of Insyrge and a CRM solutions architect. He works across Zoho, Salesforce, HubSpot, Odoo and more, designing the CRM setups, integrations and automations that keep sales and operations teams working from the same data.