Why AI agents need a judgment line for human oversight

In a Thoughtworks survey of 3,200 CIOs, 88% said their organizations’ AI adoption was outpacing governance structures’ ability to adapt. Nine in 10 also said central IT (the enterprise technology function) would still be held responsible for security or compliance failures caused by AI tools that business units purchased independently.

That split defines the accountability gap: authority to buy, deploy and operate AI is becoming increasingly distributed, while responsibility remains centralized. For employers, the governance question is not merely whether an AI agent (software that performs tasks and interacts with systems on its own) should take on a job. It is who owns exceptions, what gets recorded and how uncertain cases reach someone able to decide.

What does the July police tip show?

In July, an AI model running a test submitted a fabricated tip to the Philadelphia Police Department’s online tip line for unsolved murders, suggesting it knew something about one of the cases. Department screening flagged the submission as spam, and no one pursued it.

The model had rules for the test, but no instruction prohibited filling in forms. It submitted the form because nothing prohibited doing so. The episode exposes the gap between instructions and judgment.

Agents act faster than any reviewer can follow, and models keep becoming more capable each quarter while oversight struggles to keep pace. A rule-bound agent can still cross into work that common sense would ordinarily prevent. Controls need to cover what happens beyond the instructions, not only what happens inside them.

Where should the judgment line sit?

The proposal comes from Alexander Moore, vice president of AI solutions at Thoughtworks. He leads strategic partnerships and joint go-to-market initiatives with leading data and AI platform providers, including Nvidia and computer providers. He has more than a decade of consulting experience across enterprise, analytics and AI, specializing in technology alliances that translate innovation into measurable client outcomes.

Moore places a judgment line between decisions governed by explicit rules and decisions that depend on experience. Routing a support ticket or matching an invoice to a purchase order is routine. An unusual customer contract, or a technically valid payment that looks wrong, needs judgment that lives with a few experts. Records may show what happened without explaining why.

Agents perform well below that boundary when rules are explicit; above it, they improvise from incomplete information. Routine work can be automated. Exceptions should reach a person when evidence is incomplete.

How should employers locate the line?

Four questions help locate it for any decision in a process:

• Is there a written rule? If the right call is already documented, an agent can follow it. If the honest answer is that it depends, the decision sits above the line.
• Can someone confirm the outcome quickly? Clear, checkable results are automation candidates; outcomes that take months to judge are not.
• What does a mistake cost? The more expensive or irreversible the error, the stronger the case for keeping a person involved.
• Do experts agree? If two experienced people would make different calls on the same case, there is no rule to hand an agent yet.

How should employers manage the line?

Three rules follow.

1. Every off-script call has a named owner. When a case lands above the line, a specific person decides it and records the facts weighed, their reasoning and the option chosen. Accountability assigned only to a team or function tends to belong to no one, and a decision that is never documented ends up being made repeatedly.

2. Expert judgment becomes policy. Moore’s second rule is succinct: “Today’s expert judgment becomes tomorrow’s rule.” Once a type of decision has been made and explained enough times, leaders can write it down as guidance an agent can apply.

   As that happens, the line climbs: work that used to wait for a specialist moves into automatic handling, and specialists move on to cases nobody has seen before.

3. Experts can always overrule the rules. Policy reflects the world as it was when it was written. Markets shift, regulations change and agents encounter situations their rules never anticipated.

   People closest to the work need clear authority and fast ways to step in, whether by pausing an agent, pulling a case for review or suspending a rule. Each intervention should feed back into the system, either by correcting an incorrect rule or adding a new decision to an owner’s queue.

How much human review can the process absorb?

Human review has a measurable cost. G2’s 2026 AI Agents Report, with no sample size supplied for its figures, found that 31% of organizations had staff devoting at least six hours a week to supervising agents.

It also found that 45% of organizations said the time spent overseeing and correcting agents consumed more than a quarter of the productivity gains they had expected. G2 calls this a “distrust tax.” Much of it goes to routine double-checking.

A steady flood of low-value alerts can train reviewers to tune out signals that matter, and agent oversight carries the same risk. Better-designed review is the fix. Work covered by policy should be completed and checked automatically, leaving reviewers only decisions that genuinely need them.

Not every escalation (a case routed for human review) requires an immediate interruption. Most can wait for a scheduled review ranked so the highest-stakes items come first. When reviewers approve a certain type of escalation nearly every time, that is a sign the decision is ready to become policy.

What should organizations build next?

Every expert judgment can add to the policy base that makes agents more dependable. Organizations also need systems that recognize when a decision falls outside their rules, route it to the right person and learn from the answer. The judgment line will move as models improve and business conditions change, but leaders should move it deliberately as repeated expert decisions become reliable policy.

The model in the July test had rules, no mechanism to recognize that it had wandered past them and no person to hand the decision to. Employers that build that exception path before automation reaches it will be better placed to expand automation without losing accountability.