Loading page…
The framework mapping finds recurring safeguards, not proof of their effects. Patil’s concern is that review, escalation and audit rules can discourage workers from acting on expertise.
Listen to this story
AI oversight rules can preserve accountability yet make workers less likely to trust their own assessment, argues Shefali V. Patil. Her review of 14 governance frameworks found recurring requirements for human review, escalation, documentation and risk-based oversight; she warns these can anchor decisions on AI alerts, shift authority away from frontline staff and reward defensible records over situational judgment. The review maps design patterns, but does not establish that they cause worse outcomes; longitudinal evidence is not yet available.
Patil says analysts may treat an AI alert as the starting point for review—and interpret the absence of an alert as reassurance.
Nurses using Epic-based sepsis-detection tools may avoid challenging alerts when doing so requires escalation to supervising physicians.
The article cites an estimate that 90% to 95% of anti-money-laundering transaction alerts are false positives, each still requiring investigation and documentation.
Shefali V. Patil and collaborators mapped 14 AI governance frameworks and found a recurring reliance on human oversight, escalation, documentation and risk-based review. In a November 2 article in MIT Sloan Management Review, Patil argues that these safeguards can also send workers a counterproductive message: follow the process rather than exercise independent judgment.
The ongoing collaboration covers financial services, health care, cybersecurity and cross-sector governance. Its mapping identifies common design choices; it does not demonstrate that those choices cause worse outcomes. Patil explicitly notes that longitudinal evidence—tracking effects over time—is not yet available to systematically document the consequences of these emerging AI governance designs.
Her argument draws on research into accountability and organizations that manage catastrophic risks, such as nuclear plants and air traffic control centers. That work emphasizes protecting frontline judgment and giving authority to people closest to a problem. The frameworks she examined pursue similar aims through four recurring practices:
Keeping a human in the loop is intended to preserve responsibility rather than let people blame the machine. But assigning analysts to approve or reject AI alerts can narrow their role, Patil argues. Instead of forming an independent assessment, they begin with the system’s conclusion and look for reasons to challenge it.
That shift can affect cases the AI never flags, too. An analyst may read silence as reassurance and become less likely to investigate a missed threat. Patil says less experienced analysts are especially vulnerable because they have less developed judgment for separating meaningful signals from noise.
When AI outputs arrive before human evaluation, the system’s assessment becomes the cognitive anchor.
Shefali V. Patil, MIT Sloan Management Review
Escalation rules create a different tension. They give workers a formal way to challenge AI, but often require a supervisor or committee to authorize a departure. Requiring approval can turn an exercise of professional judgment into a request for permission. Patil argues that this moves disputed decisions toward people farther from the unfolding situation.
Her clinical example concerns nurses using Epic-based sepsis-detection tools. Patil reports that many refrain from challenging alerts when doing so requires escalating to supervising physicians—not because they agree with the recommendation, but because interrupting or challenging higher-status colleagues is burdensome. Others disengage from the alerts and defer responsibility upward.
Documentation serves a real purpose: decisions must be traceable and open to review. Patil’s concern is that the requirement can redirect attention from what fits the situation to what will look reasonable afterward. Workers may reason backward from a defensible explanation rather than forward from the conditions they face.
The workload can be substantial. The article cites an estimate that 90% to 95% of anti-money-laundering transaction-monitoring alerts are false positives, yet each must be investigated and documented for audit purposes. That example illustrates the burden of explaining decisions, rather than establishing the behavioral effects of AI governance.
The potential loss is experienced judgment that does not fit neatly into a form. Patil draws on decision-making research describing rapid pattern recognition built through practice. Such judgments can be difficult to explain afterward without being arbitrary or irrational. If workers avoid judgments they cannot easily document, she argues, organizations may become less sensitive to emerging danger even as their records become more complete.
Loading discussion...
Join the conversation
Explain when approval adds safety and when it gets in the way.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.