Human oversight has an attention budget
Adding another check does not create another supply of human attention.
In brief: Human oversight is often strengthened by adding reviews, warnings and approval steps. Each may be sensible on its own. Together, they can consume the attention needed to notice when an AI system is wrong. Research on safety clutter, alert fatigue and aviation procedures suggests that oversight should be designed as work performed under real conditions, not counted as controls on a diagram.
Imagine a caseworker using an AI system to draft decisions about benefit claims. The organisation requires the caseworker to inspect the source data, review the draft, check the explanation, consider possible discrimination, complete a risk checklist and record why the recommendation was accepted or changed.
Every requirement has a defensible purpose. In combination, they turn human oversight into several minutes of screen switching and form filling for every case. Under workload pressure, the visible checks are completed quickly. The less visible work, noticing that one claimant’s circumstances do not fit the system’s assumptions, receives whatever attention remains.
The problem is not simply that the caseworker is busy. The oversight arrangement itself is deciding what deserves attention. Activities that leave an auditable trace can crowd out the harder cognitive work that the controls were meant to support.
When safety work becomes safety clutter
Safety researchers use the term safety clutter for procedures, documents, roles and activities performed in the name of safety that do not contribute to operational safety. The point is not that procedure is bad, or that frontline workers should be liberated from tedious accountability. It is that a safety activity needs to be evaluated by what it contributes in practice, including what it displaces.
This matters for AI governance because controls are often treated as additive. A review stage, an explanation, another notification and a mandatory record are each entered into the system as extra protection. Their combined demand on the same person is rarely represented. The governance diagram acquires more reassuring boxes while the operator acquires more tabs.
More warnings can produce less warning
Consider a content moderator receiving warnings for possible hate speech, self-harm, misinformation, personal data and uncertain policy classification. Most alerts concern marginal cases or turn out not to require action. After hundreds of repetitions, the alert no longer means “stop and examine this carefully.” It becomes another obstacle to clear before continuing.
Healthcare has a well-established version of this problem. Alert fatigue occurs when the volume of safety alerts, many of them clinically inconsequential, desensitises people who must respond to them. Exposure changes behaviour. An alert can retain its formal status while losing its practical ability to redirect attention.
The obvious response is to tell people to remain vigilant. This is not much of a design strategy. If a system repeatedly teaches someone that its warnings carry little information, eventual inattention is not a mysterious defect in the human component.
A checklist has to fit the situation
Aviation offers a more developed way of thinking about the problem. Checklists are important safety tools, but their value depends on their content, timing and fit with the conditions in which crews must use them.
After US Airways Flight 1549 lost thrust in both engines shortly after take-off in 2009, the crew had only a few minutes before ditching in the Hudson River. The available dual-engine-failure checklist was designed principally for failures at high altitude. The US Federal Aviation Administration’s case summary describes it as overly complex and lengthy for the low-altitude situation the crew faced.
The lesson is not that expert intuition beats procedure. A checklist designed for a different time horizon can consume scarce capacity without supplying the information needed now. NASA’s guidance on emergency and abnormal checklist design therefore considers workload, stress, time pressure and the operational context in which a procedure must be used. A safeguard cannot be evaluated separately from the situation in which a person is expected to enact it.
Approval can become continuation behaviour
Now imagine an employee supervising an AI agent. The agent must request approval before each consequential action. This appears to provide strong human control. During one task, however, it produces twenty approval requests. Each comes with logs, an explanation and a recommended action.
By request fourteen, approval has become continuation behaviour. The interface has secured a sequence of human clicks without securing much human scrutiny.
This is particularly important for agentic systems because their activity may unfold across many steps. Adding an intervention point after every step may create theoretical control while making considered intervention less likely. The person needs enough information to understand what is happening, enough authority to stop it and enough time and attention to recognise when stopping matters.
Oversight should be evaluated as work
The presence of more human checks does not tell us whether a system has stronger human oversight. We also need to ask what each check requires the person to notice, how often it occurs, what else demands their attention and whether the information arrives while intervention is still useful.
This changes the design question. Instead of asking how many safeguards surround an AI system, we can examine how those safeguards allocate attention. Some checks should be removed. Some warnings should be reserved for cases where action is genuinely required. Some reviews should be triggered by the possible consequence of an error rather than by model uncertainty alone. Procedures may need different versions for different operating conditions.
Human attention is part of the governance system. Treating it as unlimited does not produce more oversight. It produces oversight that is increasingly visible on paper and increasingly difficult to perform.