"How reliable is the agent?" is a question I get asked in almost every kickoff call, and it's the wrong question — not because it doesn't matter, but because reliability isn't one number you can hand over. An agent can be extremely reliable at drafting a reply and completely unreliable at deciding whether to send it. Autonomy needs to be earned task by task, not granted to the system as a whole.
Four levels, not one score
The framework we use internally has four tiers, and most production systems we build live across several of them at once, depending on the specific action being taken.
Tier one is suggest-only: the agent proposes an action, a human approves or edits every single instance before anything happens. Tier two is act-with-review: the agent takes the action immediately, but a human reviews a sample or reviews anything above a risk threshold, after the fact. Tier three is act-with-exception: the agent runs unsupervised except when it hits a defined set of conditions that trigger a handoff. Tier four is fully autonomous: no routine human check at all, reserved for actions that are cheap to reverse and easy to detect if they go wrong.
How we decide where a task sits
The tier isn't chosen by gut feel. We look at three things for each distinct action the agent might take: the cost of a wrong action, how easy that wrong action is to detect after the fact, and how expensive it is to reverse. A wrongly-drafted email that a human reviews before sending is nearly free to get wrong. A wrongly-issued refund is not — it's detectable, but reversing it costs goodwill even when you catch it.
High cost, low detectability, expensive to reverse: that combination stays at tier one regardless of how good the model's accuracy numbers look in testing. Low cost, high detectability, cheap to reverse: that's where tier three or four earns its place quickly, often within the first few weeks of production data.
Reliability is earned with data, not assumed with confidence
We don't move a task up a tier because the model "feels" ready. We move it because we have a few weeks of production logs at the lower tier showing what the actual error rate looks like against real inputs, not the evaluation set we built beforehand. Real traffic finds edge cases evaluation sets don't, every time, without exception.
This is also why we push back on clients who want to launch a voice agent straight into tier three or four for anything with real consequences. It's not caution for its own sake — it's that you can't know the error rate of a system against your specific, messy, real-world inputs until it has actually seen them, and the only safe way to let it see them is with a human still in the loop.
What this looks like month over month
In practice, a system we ship in month one might have five distinct actions, three of them at tier one and two at tier two. By month three, if the logs support it, two of those tier-one actions have moved to tier two, and one of the tier-two actions has earned tier three. The system gets more autonomous over time, but the autonomy tracks evidence, and it's reversible — if error rates drift after a model update or a change in the business, a task can drop back down a tier just as easily as it moved up.
That's the part people find reassuring once they see it in practice: autonomy in a well-built system isn't a one-time decision made in a planning meeting. It's a setting that moves in both directions based on what's actually happening in production.