What it takes to trust an agent with real work

Any decent engineer can build an agent that works in a demo. The hard part is the agent that still works on run 9,412, when the API is slow, the input is malformed and nobody happens to be looking at that particular run.

Trust in an agent isn't a feeling you build up over time. It's a short list of things you can check: a test set it passes, logs you can read, limits it can't go past, and a clear point where it stops and asks a person. Here is what we put around every agent before we call it done.

Write the exam before the agent

Before we write a prompt, we write the test. That means a set of real examples, usually a couple of hundred, each with the correct outcome attached. They come from the team's actual history: last quarter's tickets, invoices, leads, whatever the agent will handle.

Every change to a prompt, a model or a tool runs against that set. If the score drops, the change doesn't ship. If a new model comes out, we can tell within an hour whether it's better for this job, instead of arguing about it.

This sounds slow. In practice it's the fastest way to move, because it ends the loop of fixing one case and quietly breaking three others. It also gives you something concrete to show the people who'll rely on the agent. "It passes 190 of these 200 real cases, and here are the ten it gets wrong" is a far better starting point for trust than "it seemed fine when we tried it."

Keep adding to the set. Every time the agent gets something wrong in production, that input becomes a new test case.

Know when it isn't sure

An agent that's right most of the time and confidently wrong the rest is worse than useless, because nobody can tell which is which. So every agent we ship has a way of knowing when it shouldn't decide alone.

We don't rely on the model's own sense of how confident it is. Language models are not well calibrated about that. Better signals are things you can check:

A required field is missing or fails validation.
Two independent passes over the same input disagree.
The input doesn't look like anything in the test set.
The action is above a set limit: a refund over a certain amount, an email to a new customer, a change to a live record.

When any of these trip, the agent stops and hands the work to a person. The hand-off matters as much as the decision to make it. The person should get the original input, what the agent found, what it was about to do, and why it stopped. A good hand-off saves them most of the work. A bad one ("something went wrong, please check") makes them start from zero and wonder why the agent exists.

The aim is to involve people where their judgment adds something, and nowhere else.

Logs a person can read in a minute

Every run records:

Field	Why it's there
Input	So you can rerun the exact case
Each step and tool call	So you can see where it went off course
Model and prompt version	So you can tie a failure to a change
Cost and time per step	So you can spot loops and slow tools
Outcome	Done, handed off, or failed, with the reason

The test for good logs is simple. When someone says "the agent did something odd with this order," can you find that run and see exactly what happened in under a minute? If you need to ask an engineer to dig through server output, the logs aren't done.

Retries for the boring failures

Most failures are dull. A timeout, a rate limit, an expired token, a third-party API having a bad few minutes. These should be retried automatically, with a growing wait between attempts, and only escalated if they keep failing. Nobody should be paged for a timeout.

Two details make retries safe. First, separate failures that might fix themselves from ones that won't. A rate limit will clear. A malformed input won't, so retrying it just burns money. Second, make sure a retried step can't do its job twice. If the agent sends an email or creates a record, it needs a way to know it already did, so a retry doesn't send the customer two copies.

A budget for every run

Agents can loop. A tool returns something unexpected, the model tries again, gets the same result, tries again. Left alone, that run can burn through far more than the task is worth.

So every run gets limits:

A maximum number of steps.
A maximum spend.
A maximum time.

Hitting any of them ends the run and flags it. Set the limits from the test set: look at what normal runs cost and allow a comfortable margin above that. You want to find a runaway in the daily report, not on the invoice.

A daily report someone reads

Finally, a short summary each day: how many runs, how many finished, how many were handed off and why, what failed, what it cost. One screen, no charts that need explaining.

The report does two jobs. It catches drift early, like a rise in hand-offs that tells you the inputs have changed. And it keeps the team's trust grounded in what actually happened. If the report is boring, the system is working.

A worked example

Say a finance team wants an agent to match supplier invoices to purchase orders. Here's how each layer looks:

Test set: 200 past invoices with the correct match, including the messy ones: partial deliveries, credit notes, invoices that reference two orders.
Hand-off rules: if the totals differ, if no order matches, or if the invoice is above a set amount, the agent stops. The person gets the invoice, the closest candidate orders, and the specific mismatch.
Logs: each run shows which orders the agent looked at, what it compared, and why it picked the one it did.
Retries: if the accounting system times out, the agent waits and tries again. If an invoice PDF can't be read, it goes straight to a person.
Budget: normal runs take a handful of steps, so a run that hits twenty is stopped and flagged.
Daily report: matched, handed off, failed, cost.

None of this is clever. That's the point. The finance team doesn't need to trust the model. They need to trust the system around it, and every part of that system is something they can inspect.

What to do this week

Pick one agent you run, or plan to, and pull 50 real past examples with the right answer for each. That's the start of your test set.
Write down the three situations where you'd want a person to decide instead of the agent. Turn each into a rule the agent can check.
Open the logs for one recent run and time how long it takes to understand what happened. If it's more than a minute, fix the logging first.
Set a step limit and a spending limit on every run, based on what normal runs cost.
Start a one-screen daily report, even if it's just a query that someone looks at each morning.