Investieren in Dubai als Deutscher: warum wir über die Holding gehen | Umair Zaffar

Human in the Loop Does Not Scale: How to Design Oversight That Does

“A human checks every output” gets regulated AI projects approved. At volume it is the first promise that breaks. Oversight that survives scale: risk tiers, triggers, sampling, escalation and a log you can trust months later.

Every regulated AI project I have been part of starts with the same reassurance. The model proposes, a human decides, nothing leaves the building unchecked. The problem is that it describes a pilot with two hundred cases a week, and nobody rewrites it when the system handles twenty thousand. AI governance that only works at pilot volume is not governance, it is a slide.

Why “a human checks every output” collapses

At low volume, full review is cheap and honest. Then volume grows tenfold and the review team does not. Review time per case drops below the time it takes to read it. Approval rates climb toward one hundred percent, because approving is one click and rejecting means a written justification. The human ends up formally in the loop and practically out of it, while the audit trail still shows a reviewer name on every decision. That is the worst outcome, because it looks like control. So the question is which human, on which decisions, with what information.

Risk tiers per decision type, not per system

The first mistake is to classify the system instead of the decisions it makes. One assistant in a claims department may draft an acknowledgment letter, propose a payout amount and flag suspected fraud. Three decision types, three different consequences. I tier them separately.

  • Tier 1, reversible and low impact. An output the recipient can see through and correct, such as a draft reply. Automated with random sampling, no review before release.
  • Tier 2, consequential but correctable. A recommendation that changes money, time or access. Released only by a human who sees the model’s reasoning, sources and policy applied, not just the result.
  • Tier 3, hard to reverse or legally protected. Denying a loan, rejecting an applicant, touching health data. The model prepares, a human decides, a second human sees a defined share of cases.

The tiers are versioned and registered together with the model. The rule from our governance layer applies here too: no registration, no deployment. A decision type without a tier does not go live.

Triggers: confidence, disagreement and honest random sampling

Tiers set the amount of oversight. Triggers decide which case lands in front of a person.

Confidence signals. A model’s own confidence is weak and often poorly calibrated, but not useless. A low-confidence case in tier 2 goes to review before release. High confidence does not earn trust, it only lowers priority.

Disagreement signals. Much stronger. Run a second, cheaper check: a rules engine, a classifier, a second model with a different prompt, or a check that the cited sources support the output. When the two disagree, a human looks. This catches the cases where confidence fails: the model is sure and wrong.

Random sampling. It finds what no trigger anticipated. The rate must be statistically meaningful for the volume: calculate it from the error rate you want to detect and the confidence you need, not from how many reviewers are free. For tier 3 it is a fixed floor per reviewer per week, so nobody can quietly turn it down to zero. And the system draws the sample, not the team, because reviewers pick the interesting cases and the quiet errors live in the boring ones.

Escalation: thresholds, pagers and when four eyes actually help

Review handles single cases, escalation handles patterns. Every trigger feeds a counter with thresholds, and every threshold has a name attached.

  • Rejection rate on sampled cases exceeds the agreed band for a day: product owner and model owner get a ticket, the decision type stays live.
  • Disagreement rate doubles against its baseline within an hour: the on-call engineer is paged, which usually means a source, a prompt or a model version changed unnoticed.
  • A single tier 3 case is overturned for discrimination: the decision type is paused, the risk function is informed the same day, and nothing resumes until the surrounding cases have been checked.

Pausing has to be one action by one person. If stopping a decision type needs a change request, you do not have a kill switch.

Two signatures on every output do not double the quality, they halve the attention. I apply four-eyes rules only where a second person changes the outcome: tier 3 rejections, overturned cases on a protected ground, the first hundred cases after a model or policy change.

What gets logged per decision

Months later, an internal auditor or a supervisor must be able to reconstruct a single decision without the engineer who built the system. The minimum per decision:

  • A hash of the inputs, so the exact case can be verified without personal data in the log, consistent with pseudonymisation.
  • Model identifier and version, including the provider’s version string, because “the model” changes under your feet.
  • Prompt template version and policy version that applied, as registered, not as remembered.
  • The retrieved sources with document versions, since a correct answer against an outdated policy is still a wrong decision.
  • The output, the confidence and disagreement signals, and the tier and trigger that routed the case.
  • The reviewer action: approved, corrected, rejected or escalated, with a reason category and the time spent. Time per review is the earliest warning you have for rubber-stamping.

Closing the loop: AI governance that learns from reviewers

Oversight that only produces approvals is a cost. Oversight that changes the system is an investment. Every corrected or rejected case gets a reason category that maps to one of three places for the fix: the prompt, the retrieval corpus, or the policy. Changes to prompts and retrieval get a new version and go through the same registration as a new model. And the cases that triggered a change become regression tests for the next version.

The failure modes nobody puts on the slide

Reviewer fatigue is not a character flaw, it is a scheduling outcome. Caps on cases per reviewer per hour and rotation between decision types do more than training.

Rubber-stamping is measurable. If approval rate and time per case move toward one hundred percent and zero seconds, the loop is dead. I treat both as production metrics with alerts.

Automation bias is the hardest. People defer to a confident system. Showing sources and reasoning helps, a bare recommendation with a green badge hurts. Injecting known-wrong cases into the review queue and measuring whether reviewers catch them is unpopular and the only reliable test I know.

Open questions

  • Sampling rates and thresholds that are sound today may not hold when models change weekly. How oversight budgets should adapt to drift without a human recalculating them is open.
  • Disagreement between two models catches errors only if their failure modes are independent. How to measure that independence in production is a research question with practical value.
  • The reasoning a model shows a reviewer is itself a model output. Whether it reflects what drove the decision or is a plausible story written afterwards decides how much it protects against automation bias, and we cannot verify that today.
  • A log lets you reconstruct a decision. It does not say whether the model would have answered differently under different wording. How much robustness evidence a decision record should carry is unresolved.

Frequently asked questions

Does the EU AI Act require a human in the loop?

The EU AI Act, in force since August 2024, phases in its obligations over the coming years and requires human oversight for high-risk systems. It describes what oversight has to achieve, not how many people have to click. Tiers and triggers meet that goal better than a signature on every output.

How do I set the random sampling rate?

From the error rate you want to detect and the confidence you need at your volume, not from available capacity. If that is more than the team can review, the tier or the volume is wrong, not the rate.

What is the first thing to log if we start tomorrow?

Model version, policy version and reviewer action per decision. Those three already tell you whether a problem lies with the model, the rules or the people.

Related: AI Governance · EU AI Act · AI Consultant

Leave Comment