The Human in the Loop Has an Information Problem

Oct 30

“Human in the loop” has become one of the most familiar phrases in artificial intelligence governance. When an AI system informs a consequential decision, a human reviewer is often presented as the safeguard. The machine proposes. The human decides. The tech gods have received their offering.

The logic is intuitive. A system may automate analysis, ranking, prediction, or recommendation, while a person remains responsible for exercising judgment.

But what, exactly, makes that judgment meaningful?

A reviewer may know what the AI recommends without understanding the basis for the recommendation. They may know that the system has limitations without knowing which limitations matter in the case in front of them. They may have permission to override an output without knowing when an override is appropriate.

They may even have access to the right documentation without enough time, context, or knowledge to make sense of it.

In those circumstances, the human is present. Whether the oversight works is another question.

A human-in-the-loop designation is a starting assumption about how a safeguard will work. Organizations should be prepared to test that assumption.

We already know how to ask better questions

The underlying problem is familiar to anyone who works in risk, audit, compliance, or internal controls. A policy can exist and still fail. A reviewer can be assigned and still miss the problem. An approval process can operate exactly as designed without adequately addressing the risk it was meant to control.

That is why assessing a control—a safeguard intended to prevent or detect a problem—requires more than confirming that it exists.

Is it designed to address the relevant risk? Can the person responsible actually perform it? Does it happen consistently and early enough to make a difference? What evidence shows that it works? What happens when it fails?

AI governance needs the same discipline.

The question “Is there a human in the loop?” tells us something about the workflow. It tells us considerably less about whether we can rely on the person’s review.

A breathing human is a remarkably low bar for a governance framework.

What meaningful oversight requires

Human oversight is often described in terms of authority: someone must be able to question, override, or stop the system. That authority matters. But a person also needs to understand enough about the system and the decision to recognize when intervention is warranted.

This does not mean every reviewer must become a machine-learning engineer. It means the knowledge required for oversight should match the responsibility assigned to the reviewer.

A recruiter, a clinician, and an insurance claims reviewer may need very different information. Each needs a practical understanding of the system’s relevant capabilities, its limitations, and the circumstances in which relying on it could cause harm.

General AI literacy can help. Employees should understand that AI systems can produce incorrect outputs, that privacy risks exist, and that organizational policies govern how tools may be used. But knowing that “AI can be wrong” does not tell someone how to recognize a consequential error in their own work.

That is where information literacy becomes central.

The information problem

Information literacy is the ability to evaluate information: where it comes from, how credible it is, what it establishes, what its limitations are, and what remains uncertain.

Access to information and the ability to evaluate it are different things. We can retrieve an answer in seconds without knowing whether its evidence supports its conclusion. AI makes that distinction especially important because it can deliver information that has already been selected, organized, and interpreted for us.

There is enormous value in that capability. But a coherent answer can conceal uncertainty. A numerical score can suggest more precision than the underlying assessment deserves. An explanation can sound persuasive without giving a reviewer what they need to test the recommendation.

We spend a great deal of time asking what artificial intelligence can know, interpret, and decide. I wonder whether we have devoted enough attention to what the people overseeing it need to understand.

Consider an AI-assisted hiring system.

The system reviews applications and ranks candidates. Before anyone is rejected, a recruiter must review the recommendations. On paper, that sounds reassuring.

Now consider the recruiter’s experience.

One candidate receives a score of 91. Another receives a 74. Hundreds of applications are waiting. The organization purchased the system from a respected vendor. Leadership expects it to make recruiting more efficient. The recruiter has been told that the scores are advisory and may be overridden.

But what would tell the recruiter that an override is appropriate?

Do they know which factors materially influenced the score? Whether employment gaps could disadvantage caregivers? How the system treats nontraditional educational backgrounds? Whether résumé formatting affects the result? Do they understand what testing was performed and where the system’s performance may be less reliable?

A girl has questions.

Without relevant information, the recruiter may have authority to challenge a recommendation without a useful basis for doing so.

Providing that information is only the first step. We also need to know whether the recruiter can interpret it.

Can they distinguish between strong average performance and reliable performance for the particular candidates being assessed? Can they recognize when a score suggests unjustified certainty? Can they explain why they accepted or rejected a recommendation using evidence beyond the recommendation itself?

Those are questions about information literacy within a specific decision-making role.

Experience, confidence, and completion of a training module do not, by themselves, demonstrate that capability. We need to see how people apply what they know.

When review becomes agreement

We often speak about human judgment as though its independence comes automatically with the human.

But the recruiter may be evaluating a candidate through the lens the AI has already supplied. A low score becomes a reason to search for weaknesses. A high score becomes a reason to confirm strengths. The reviewer can still make a decision, but the system may have shaped how they approach it.

That possibility deserves attention because benefiting from a system’s capabilities and being able to oversee it are different responsibilities.

A person may reasonably use an AI system to identify patterns they could not identify independently. Oversight then requires a way to evaluate whether those patterns support the decision being made. Depending on the use case, that might involve examining source evidence, checking relevant limitations, consulting someone with additional expertise, or escalating a concern.

The reviewer does not need to reproduce the system’s entire analysis. They need a credible way to recognize when reliance is unwarranted and a practical route to act on that concern.

Information literacy is necessary for this kind of oversight. It cannot, however, compensate for a process that makes intervention impossible or costly.

Timing matters

Imagine an AI system ranks 10,000 loan applications. The bottom 6,000 are automatically removed from consideration. A human reviews the remaining 4,000 before final decisions are made.

Calling the process human-supervised leaves an important question unanswered: who reviewed the exclusion of the first 6,000 applicants?

For those applicants, the consequential outcome may already have been determined before the reviewer entered the process.

The useful governance question is: at what point can the human still materially change the outcome?

A review conducted after the system has filtered the population, narrowed the available options, or established the default recommendation may offer less protection than its description suggests.

Timing also affects judgment. Once a reviewer sees an AI-generated assessment, that assessment can become an anchor. Instead of beginning with “What does the evidence show?” the reviewer may begin with “Do I have enough evidence to disagree with the AI?”

Those questions place the burden of justification in different places. The design of the review should account for that difference.

Authority must work in practice

Now suppose the reviewer recognizes a problem. What happens next?

Consider an insurance claims team measured partly on processing volume. Following the AI recommendation takes one click. Overriding it requires additional documentation, managerial review, and several extra minutes of work.

The employee may have formal authority to intervene. The workflow still makes agreement easier.

This does not establish that the employee will always defer to the system. It does show why an assessment of oversight must examine the conditions under which decisions are made.

Can the reviewer override the recommendation? Are they willing and able to do so in practice? What happens when they exercise that authority?

Those are different questions.

If questioning the system consistently produces delay, criticism, or an unmanageable backlog, written permission may offer less protection than the organization assumes.

The responsibility belongs partly to the institution designing the workflow. We cannot reasonably rely on independent judgment while making its exercise a professional liability.

Automation bias is a design concern

Automation bias—the tendency to give undue weight to an automated recommendation—is often discussed as a weakness the individual reviewer must overcome.

Remain vigilant. Challenge the model. Exercise independent judgment. Fight the good fight.

These expectations deserve more support than a slide in annual training.

If a system has usually been correct, people have reasons to trust it. If leadership repeatedly emphasizes its accuracy, employees may give its recommendations additional weight. If accepting an output is easy and questioning it is burdensome, the process creates a reason to accept it.

Trust can be justified. Oversight requires people to recognize the limits of that trust.

Organizations should therefore examine how interfaces, workloads, performance measures, and management behavior affect review. Does the system expose uncertainty in a useful way? Can reviewers consult evidence beyond the output? Is escalation practical? Do managers support well-founded disagreement?

A warning about automation bias is useful. A workflow that supports meaningful challenge is more useful still.

What organizations should test

A more rigorous assessment begins with a practical question: can this human review be relied upon for the purpose assigned to it?

That assessment should consider several connected conditions.

Knowledge and judgment. Does the reviewer understand the system’s relevant capabilities and limitations? Can they recognize circumstances that call for further review?

Information. Does the reviewer receive relevant evidence in a form they can evaluate? Can they distinguish what the evidence supports from what remains uncertain?

Authority and escalation. Can they override, pause, stop, or escalate the process as appropriate to their role? Is there someone equipped to respond when the issue exceeds their expertise?

Timing. Does intervention occur while it can still change the consequential outcome?

Capacity and incentives. Is there enough time to perform the review? Do workload expectations and performance measures support thoughtful challenge?

Accountability. Is it clear who is responsible for the decision and for addressing failures in the oversight process?

Observed behavior. Do people actually perform the review as intended, including when the system appears confident or the workload is high?

These conditions interact. A knowledgeable reviewer with no time may struggle to perform the control. A well-informed reviewer without practical authority may recognize a problem they cannot resolve. Formal authority without relevant information leaves the reviewer guessing.

No single training certificate or workflow diagram demonstrates that the whole arrangement works.

Design is only the beginning

Even a thoughtfully designed process can weaken over time.

Documentation becomes cumbersome. New employees receive abbreviated training. The model changes, but the guidance does not. Workload increases. Review time shrinks. An exception becomes a habit.

Organizations should therefore examine how oversight operates after implementation, using evidence appropriate to the use case. That could include sampled decisions, scenario exercises, reviewer interviews, escalation records, observed work practices, and analysis of errors or near misses.

Individual measures require interpretation. A low override rate could mean the system is performing well. It could also mean reviewers rarely challenge it. A high rate could indicate attentive review, poor system performance, or unclear guidance.

The number alone does not answer the question. We need to understand the decisions behind it.

This brings us back to information literacy. The organization assessing human oversight must also know what its own evidence does and does not establish.

The Question of Self-Governance

Perhaps the most difficult question is what makes us capable of exercising the judgment we expect AI oversight to provide.

Humans bring bias, overconfidence, habits, and sensitivity to institutional pressure into any decision. These vulnerabilities existed long before artificial intelligence. AI changes the setting in which they operate.

We are asking people to evaluate systems that can process more information than they can examine independently and produce recommendations with remarkable speed and apparent confidence. We then expect those people to recognize when the recommendation should be questioned.

That is a substantial responsibility.

Do we recognize when our confidence exceeds our understanding? Are we willing to question conclusions that confirm what we already believe? Can we distinguish trust supported by evidence from trust sustained by convenience? Do we understand how our incentives and environments influence our decisions?

Self-governance, in this sense, involves recognizing our limitations and creating structures that account for them. That responsibility belongs to institutions as well as individuals.

If we intend to rely on human judgment as a safeguard, we should invest in understanding, developing, and testing it. We should provide information people can use, prepare them for the decisions they face, and make it possible to act when something warrants concern.

The objective is to preserve the capacity for meaningful human judgment in systems that increasingly shape how we understand the world and make decisions.

We cannot assume that capacity simply because a human is present.

Maybe governing artificial intelligence begins with something more fundamental: learning how to govern ourselves.