Approve the AI Agent Workflow and Its Controls Together

Suppose the pilot deck says the model gets 98 percent of refund decisions right. Before you nod, multiply.

At 10,000 decisions a month, a 2 percent error rate implies about 200 wrong decisions, if that rate holds in production. These are illustrative numbers, not a benchmark.

That isn't necessarily 200 mistakes reaching customers. Some may be blocked or corrected. The approval question is how many get through, what harm they cause and whether the checking can keep up.

I'd approve the workflow and its controls together. Model accuracy matters. But if your risk case depends on review, you need evidence that the review can find the material errors and handle the workload.

First establish whether the percentage describes model proposals or final decisions after review. Don't credit the same checks twice. And don't treat a result from a narrow, curated pilot as a measured production rate. If production accuracy is unknown, write unknown and show any planning assumptions separately.

Two hundred errors is not two hundred reviews

Wrong decisions don't arrive labeled. If a person must inspect every proposal, the review workload is 10,000 cases, not 200.

Now extend the hypothetical. Suppose an automated screen sends 10 percent of cases to people: 1,000 reviews a month. Your team can complete 20 reviews an hour and has 30 hours available. That's capacity for 600 cases. The queue grows by 400, even though comparing 600 reviews with 200 expected errors might have looked reassuring.

Nor will those 1,000 flagged cases necessarily contain all 200 errors. Some flags will be false alarms. Some wrong decisions will pass the screen. Processing the entire queue would still not establish that the workflow is safe.

Keep two comparisons separate:

• Review workload against review capacity: can the assigned checks happen in time?
• Errors escaping the whole process against acceptable risk: are the remaining consequences tolerable and permitted?

For work assigned to people, review capacity is the rate they can sustain at the required quality multiplied by the hours actually available. Time the real job, including retrieving records, checking policy and resolving discrepancies. Counting clicks on an approval button won't do.

Reserve capacity for corrections and quality sampling, too. Then test peak demand and response deadlines, not just monthly averages. A reviewer available tomorrow cannot prevent a payment released tonight. Decide whether overdue cases wait, move to another qualified reviewer or return to the existing process. Don't let a full queue quietly become permission to bypass the check.

Make the output cheaper to check

Reading everything may be too slow or expensive at the intended volume. More reviewers can help, but they aren't the only lever.

In the refund example, I'd enforce amount limits and duplicate-payment rules in ordinary code wherever the policy can be expressed reliably. Let AI interpret an ambiguous request where that capability is useful. Don't ask it to improvise a rule the business already knows.

Give the reviewer a compact record containing the proposed decision, relevant transaction, applicable policy and supporting evidence. Make discrepancies easy to locate. A polished paragraph that forces someone to reconstruct the case is an expensive review interface.

AI-assisted checking can reduce that work, but it adds another system to evaluate. A model checking another model can share its blind spots or reward a convincing explanation rather than a correct decision. A different model family may help diversify the check; it doesn't establish that the checker is independent or effective.

Test the complete checking path on cases with confirmed mistakes and on correct cases, including the ordinary work that will dominate the queue. Keep some cases out of tuning. Measure which important errors remain undetected, which correct cases trigger unnecessary review and how long resolution takes.

Have qualified people examine disputed results. A finding absent from your answer key might be a new defect, not a false alarm. And a proposed correction needs checking too.

Sample decisions that passed without human review, as well as those that were flagged. Looking only at the review queue cannot tell you what the screen missed. Sampling helps estimate what escapes; it does not protect every individual transaction. Where a failure must be prevented, the relevant control belongs before the action.

The Error Ledger should change the approval

For the next consequential rollout or expansion, I'd ask the workflow owner for a short Error Ledger, prepared with operations and the relevant quality or risk specialists. This is a proposed operating practice, not a validated protocol or a certification.

Put intended monthly and peak volume at the top. Record the error-rate estimate, what was tested, when, and the uncertainty around it. Show model errors separately from errors that remain after the controls have run.

For each material error type, describe the consequence, the control and the evidence that the control works. An unjustified refund, a wrongly denied refund and disclosure of customer information are different failures. Don't average them into one reassuring quality score. Estimate financial costs where useful, but keep safety, privacy and legal obligations explicit. No severe error found in a small test is not evidence that the risk is zero.

Beside that, show all work routed to people, the capacity available and the time the cases can wait. Include the hours spent investigating false alarms and correcting mistakes. This is where the 400-case gap in our example has to be resolved.

Put the economics on the same page. Compare the proposed workflow with the current process on comparable work, including the current process's own errors, delays and controls. Count the model, automated checks, human review, rework and recovery costs. If checking consumes the time saved, don't sell the project as labor savings. Better quality or faster service might still justify it, but make that case explicitly.

In The AI Answer Your Board Actually Wants, I recommend measuring the share of a workload handled without a person and assigning an owner to its exceptions. This adds the operating test: can those exceptions be handled, and what is the system failing to recognize as an exception?

Let the evidence change the approval. Fund additional review if capacity is the real gap. Improve the input or checking process if avoidable rework is consuming the hours. Narrow the decisions the agent can make automatically, or cap the rollout while the team learns.

Accepting some residual error can also be reasonable where the consequences and your obligations allow it. Record the remaining exposure, who can accept it and the monitoring required. That is different from signing off on an unmonitored flow simply because the risk has been named. If a material risk cannot be controlled within acceptable and permitted bounds, defer or reject the rollout.

The page does not replace security, privacy or sector-specific approval. It makes the operating assumptions behind this particular approval visible.

Better accuracy can change the answer

Within our illustration, improving accuracy from 98 percent to 99.5 percent cuts expected wrong decisions from 200 to 50 at the same volume. That's a real improvement in the arithmetic. If testing supports lighter review, reduce it.

But increase volume to 40,000 decisions and that lower error rate implies 200 wrong decisions again. Expand into harder cases and the old rate may no longer apply. Revisit the calculation when volume, task mix, model or controls change. Better models are a reason to update the decision, not a reason to stop measuring it.

Keep the burden proportionate. For genuinely low-volume, low-consequence, reversible work, a formal ledger may add little. Keep the fast path and the basic safeguards. Low volume alone, however, does not make a consequential decision harmless.

For a live workflow, assign someone to monitor escaped errors by type, review delays and recovery work. Set triggers for reducing volume or pausing automation, with a named person able to act. Make ordinary problems easy to report inside the tool, and route serious safety, privacy or compliance concerns through the established incident process. A thumbs-down button is useful feedback, not an incident response plan.

Before the next expansion, ask for evidence that the current controls are working, not merely that every box on the page is filled.

Then multiply before you nod. Ask for the review queue and the errors that escape it.

Frequently asked questions

Our AI model is 98 percent accurate at 10,000 decisions a month. Should I approve the workflow for production?

Approve the workflow and its controls together, not the accuracy number on its own, because at 10,000 decisions a month a 2 percent error rate implies about 200 wrong decisions if that rate holds in production. The approval question is how many of those get through, what harm they cause and whether the checking can keep up, and if your risk case depends on review, you need evidence that the review can find the material errors and handle the workload. Accepting some residual error can be reasonable where the consequences and your obligations allow it, as long as you record the remaining exposure, who can accept it and the monitoring required, but if a material risk cannot be controlled within acceptable and permitted bounds, defer or reject the rollout.

Can I assume the pilot's 98 percent accuracy will hold in production?

No, don't treat a result from a narrow, curated pilot as a measured production rate. First establish whether the percentage describes model proposals or final decisions after review, and don't credit the same checks twice. If production accuracy is unknown, write unknown and show any planning assumptions separately.

If the model only makes about 200 wrong decisions a month, can't our reviewers just check those cases?

They can't, because wrong decisions don't arrive labeled: if a person must inspect every proposal, the review workload is all 10,000 monthly decisions, not 200. An automated screen that sent 10 percent of cases to people would mean 1,000 reviews a month, but a team completing 20 reviews an hour with 30 hours available can handle only 600, so the queue would grow by 400, and the flagged cases would not necessarily contain all the errors anyway, since some flags would be false alarms and some wrong decisions would pass the screen. Keep two comparisons separate: whether the assigned checks can happen in time, which is review workload against review capacity, and whether the errors escaping the whole process leave consequences that are tolerable and permitted.

How do I know if our review capacity is actually enough?

Your review capacity is the rate your reviewers can sustain at the required quality multiplied by the hours actually available, so time the real job, including retrieving records, checking policy and resolving discrepancies, because counting clicks on an approval button won't do. Reserve capacity for corrections and quality sampling, and test peak demand and response deadlines rather than just monthly averages, since a reviewer available tomorrow cannot prevent a payment released tonight. Decide whether overdue cases wait, move to another qualified reviewer or return to the existing process, so a full queue doesn't quietly become permission to bypass the check.

How can we make the AI's decisions easier and cheaper to review?

Enforce amount limits and duplicate-payment rules in ordinary code wherever the policy can be expressed reliably, and let the AI interpret ambiguous requests where that capability is useful, instead of asking it to improvise a rule the business already knows. Give the reviewer a compact record containing the proposed decision, the relevant transaction, the applicable policy and the supporting evidence, and make discrepancies easy to locate, because a polished paragraph that forces someone to reconstruct the case is an expensive review interface. AI-assisted checking can reduce that work, but it adds another system to evaluate, since a model checking another model can share its blind spots or reward a convincing explanation rather than a correct decision, and a different model family may help diversify the check without establishing that the checker is independent or effective.

How do we test whether our checks actually catch the important errors?

Test the complete checking path on cases with confirmed mistakes and on correct cases, including the ordinary work that will dominate the queue, keep some cases out of tuning, and measure which important errors remain undetected, which correct cases trigger unnecessary review and how long resolution takes. Have qualified people examine disputed results, because a finding absent from your answer key might be a new defect rather than a false alarm, and a proposed correction needs checking too. Sample decisions that passed without human review as well as the ones that were flagged, since looking only at the review queue cannot tell you what the screen missed, though sampling estimates what escapes rather than protecting every individual transaction, so where a failure must be prevented, the relevant control belongs before the action.

What should I ask the workflow owner for before approving a consequential AI rollout?

Ask for a short Error Ledger, prepared with operations and the relevant quality or risk specialists; it's a proposed operating practice, not a validated protocol or a certification, and it doesn't replace security, privacy or sector-specific approval. It should put intended monthly and peak volume at the top, record the error-rate estimate with what was tested, when and the uncertainty around it, show model errors separately from errors that remain after the controls have run, and describe each material error type's consequence, control and evidence that the control works, keeping safety, privacy and legal obligations explicit instead of averaging different failures into one reassuring quality score. It should also show all work routed to people, the capacity available and how long cases can wait, including the hours spent investigating false alarms and correcting mistakes, and put the economics on the same page by comparing the proposed workflow with the current process on comparable work and counting the model, automated checks, human review, rework and recovery costs; then let the evidence change the approval, funding more review if capacity is the real gap, improving the input or checking process if avoidable rework is consuming the hours, narrowing what the agent can decide automatically, or capping the rollout while the team learns.

What should we monitor once the AI workflow is live?

Assign someone to monitor escaped errors by type, review delays and recovery work, and set triggers for reducing volume or pausing automation, with a named person able to act. Make ordinary problems easy to report inside the tool, and route serious safety, privacy or compliance concerns through the established incident process, because a thumbs-down button is useful feedback, not an incident response plan. Before the next expansion, ask for evidence that the current controls are working rather than merely that every box has been filled, and revisit the calculation whenever volume, task mix, model or controls change, since better models are a reason to update the decision, not a reason to stop measuring it.