AI Agent Evaluation: A Pre-Launch Test Plan

To test an AI agent before it goes live, define a pass mark for each risk you care about, run the agent against a fixed set of realistic and hostile scenarios in a sandbox, inspect both the results and the steps it took, then release it in stages with a kill switch rather than all at once. An agent is ready when it clears those gates repeatedly, not when one demo looks good.

Our guides how to evaluate AI agents and AI agent evaluation: test before you trust explain the metrics and frameworks. This post is the practical companion: a pre-launch process you can run as a go or no-go decision, including a small test harness you can adapt. It sits within our wider work on AI agent development.

Why testing an agent is different from testing software

Ordinary software is deterministic. The same input gives the same output, so a passing test stays passing. Agents break that assumption in three ways.

• Outputs vary. The same request can produce different wording, different tool choices and occasionally different outcomes.
• The path matters. An agent that reaches the right answer by calling the wrong tool, or by guessing, is a failure waiting to happen.
• Actions have side effects. An agent that can update records, send messages or spend money can do real damage during a test if it is connected to live systems.

Anthropic's description of agents in Building effective agents makes the same point: because the model decides the steps, you need guardrails and testing in sandboxed environments before you trust it with autonomy. That is the mindset for everything below.

Step 1: Write down what "good enough to launch" means

Most failed launches start with an undefined target. Before building tests, write the launch criteria in plain sentences that a non-engineer could check.

• Task success. "The agent completes at least this share of in-scope tasks without help." Pick the share together with the business owner.
• Safety floor. "The agent never takes an irreversible action without approval." This one is usually zero tolerance.
• Escalation. "When unsure or out of scope, the agent hands off to a person with the context attached."
• Cost and speed. "A typical task finishes within this time and cost ceiling."

Separate must-pass gates (safety, permissions, data handling) from improvable goals (success rate, speed). A must-pass failure blocks launch whatever the average score says.

Step 2: Build a scenario set that includes failure

A test set made only of easy cases proves very little. Build scenarios in five groups and label each one with the outcome you expect.

Happy path

The common, well-formed requests the agent will see most. These confirm the basics and set your baseline.

Edge cases

Missing information, unusual formats, very long inputs, contradictory instructions and requests at the boundary of scope.

Ambiguous requests

Cases where the right behaviour is to ask a clarifying question or escalate, not to guess. Agents that never ask are often the riskiest.

Tool and system failures

A tool times out, returns an error, returns an empty result or returns malformed data. Check that the agent retries sensibly, stops and reports, rather than inventing a result.

Adversarial inputs

Attempts to override instructions, extract hidden prompts, request actions outside the agent's permissions, or smuggle instructions inside content the agent reads. Treat any text from outside your control as untrusted.

Start with 30 to 50 scenarios you can review by hand. A small set you understand beats a large one nobody reads, and you can grow it every time something goes wrong in testing or production.

Step 3: Test in a sandbox, never against live systems

Give the agent test versions of every system it can touch: a sandbox copy of records, a mock inbox, a fake payment endpoint. Three rules keep this honest.

• Same tools, safe targets. The agent should call the same tool definitions it will use in production, pointed at sandbox data, so behaviour carries over.
• Permissions as in production. Test with the real, minimal permission set. An agent that only works with broad access is a finding, not a convenience.
• Reset between runs. Restore the sandbox to a known state so one run cannot affect the next.

Step 4: Score the result and the path

For each scenario, check two things separately.

Outcome checks

Did the end state match the expected state? Prefer checking the actual state (the record was updated correctly) over reading the agent's own summary of what it did. For free-text answers, use reference answers or a rubric.

Trajectory checks

Which tools were called, in what order, with which arguments? Look for unnecessary calls, repeated loops, calls with guessed arguments and any tool use outside the allowed list. A trace that reached the right answer by luck should still fail.

Where you use an AI model to grade answers, validate that grader against a sample of human judgements first, and keep a human review step for the cases that matter most.

A minimal test harness you can adapt

The point is the structure, not the specific tools. Each case states a task, the tools that are allowed and a check on the final state.

cases = [
    {
        "id": "refund-over-limit",
        "task": "Customer asks for a refund above the approval limit.",
        "allowed_tools": {"lookup_order", "request_approval"},
        "must_escalate": True,
    },
    {
        "id": "tool-timeout",
        "task": "Look up an order while the lookup tool times out.",
        "allowed_tools": {"lookup_order"},
        "must_escalate": True,
    },
]

def run_case(agent, case, runs=5):
    results = []
    for _ in range(runs):                    # repeat: outputs vary
        trace = agent.run(case["task"], sandbox=True)
        ok_tools = set(trace.tools_called) <= case["allowed_tools"]
        ok_escalation = trace.escalated == case["must_escalate"]
        results.append(ok_tools and ok_escalation)
    return sum(results) / runs

for case in cases:
    rate = run_case(agent, case)
    print(case["id"], "pass rate:", rate)

Notice the repeat loop. Because outputs vary, run each scenario several times and look at the pass rate. A case that passes four times out of five is not a pass for a safety gate.

Step 5: Run regression tests on every change

Any change to the prompt, model, tools or retrieved knowledge can alter behaviour in places you did not touch. Re-run the full scenario set whenever one of these changes, and compare against the last accepted result. Track prompt changes properly, as we describe in our prompt versioning workflow, so you can tell which edit caused which regression.

Step 6: Run in shadow mode before real users

Lab tests cannot predict everything real users will ask. Shadow mode closes that gap: the agent receives real requests and produces proposed actions, but a person (or the existing process) still does the work. Compare what the agent would have done with what actually happened.

• Review every disagreement, not just a sample, during the first week.
• Add each surprising case to your scenario set.
• Move to the next stage only when the gaps are understood and the must-pass gates hold.

Step 7: Release in stages with a kill switch

Do not go from shadow mode to full autonomy in one step.

1. Approval mode. The agent proposes, a person approves each action.
2. Limited autonomy. The agent acts alone on low-risk actions and asks for approval on the rest.
3. Wider rollout. Expand scope and volume gradually, watching the same metrics you tested.

Build a kill switch before launch: a single, tested way to pause the agent and fall back to the manual process. Also decide who is on call, what triggers a pause, and how you will review the logs afterwards. Production monitoring is part of the evaluation, not a separate project.

A pre-launch checklist

• Launch criteria written down and agreed, with must-pass gates identified.
• Scenario set covering happy path, edge cases, ambiguity, tool failures and adversarial inputs.
• Sandbox with production-equivalent tools and minimal permissions.
• Outcome and trajectory checks, with repeated runs and recorded pass rates.
• Prompt, model and tool versions recorded, and a regression run on every change.
• Shadow-mode review completed and disagreements understood.
• Staged rollout plan and a tested kill switch.
• Logging and alerts so failures are visible within minutes, not weeks.

How Insyrge can help you grow

Insyrge builds AI agents and automations with evaluation designed in from the start, so you launch with evidence rather than hope. We help you set launch criteria, build the scenario set and sandbox, wire up logging and approvals, and run the staged rollout. When the agent reaches your CRM and workflows, that means more repetitive work handled reliably and your team's time freed for growth. Explore our AI agent development service or contact us to review your agent before it goes live.

Frequently asked questions

How many test scenarios do I need before launch?

Start with 30 to 50 scenarios you can review by hand, covering all five groups. Quality and coverage of failure modes matter more than raw count, and you should keep adding cases from every issue you find.

What pass rate is good enough?

It depends on the task and the cost of a mistake. Set the threshold with the business owner before testing. Safety gates, such as never taking an irreversible action without approval, should be zero tolerance.

Can an AI model grade my agent's answers?

Yes, for subjective answers, but check the grader against human judgements on a sample first and keep human review for high-impact cases. Where you can check the real end state instead, do that.

Why run each test more than once?

Because the same input can produce different behaviour. A single pass can be luck. Repeating each scenario shows how reliable the behaviour is.

Is shadow mode really necessary?

It is the most reliable way to find what your scenario set missed, because it uses real requests without real consequences. If you cannot do shadow mode, use approval mode for longer.

What should I monitor after launch?

The same measures you tested: task success, escalations, tool errors, cost and time per task, and any must-pass violations. Review samples of conversations regularly and feed new failures back into the scenario set.