Automation Bias Prevention
Added May 2026
Implement measures to detect and counteract the tendency for human reviewers to defer to AI recommendations without adequate critical evaluation.
Objective
Maintain the integrity of human oversight by ensuring reviewers are actively engaging with AI outputs rather than reflexively approving them.
Maturity Levels
Initial
No awareness of automation bias risk; override rates are not tracked.
Developing
Training materials mention automation bias but no structural interventions are in place.
Defined
Override rates are tracked by reviewer and use case; unusually low rates trigger investigation.
Managed
Periodic calibration exercises test reviewer accuracy; automation bias is a standing item in governance reviews.
Optimizing
Review workflow design is continuously refined based on bias detection data; A/B testing (comparing two versions of the review screen) is routine.
Evidence Requirements
What an auditor or assessor would expect to see for this control.
- —Monthly override rate reports by reviewer, team, and use case showing trend over time
- —Calibration exercise results showing detection rates for deliberately planted errors by reviewer, with pass/fail against defined threshold
- —Investigation records for any reviewer or team whose override rate fell below the defined alert threshold
- —Approval screen settings confirming friction controls are active (mandatory rationale field, one-click approval disabled for outputs the AI flags as uncertain)
- —Training records showing completion of automation bias (the tendency to accept AI output without checking it) awareness module for all active reviewers
Implementation Notes
Key steps
- Track override rates (how often reviewers reject or change an AI recommendation) by reviewer, team, and use case. A sustained rate below 2-3% for high-stakes decisions warrants investigation.
- Introduce friction by design (deliberate extra steps before approval): require reviewers to input a brief rationale before approving, rather than offering one-click approval.
- Run periodic calibration tests: present reviewers with known-incorrect AI outputs and measure detection rates.
- Rotate reviewers on high-volume queues so they do not slip into routine approval.
Example Implementation
HR team using AI resume screening across three hiring groups
Automation Bias Monitoring: Resume Screening Queue
Monthly report to AI Governance Committee:
| Team | Queue Volume | Override Rate | Flag Threshold | Status |
|---|---|---|---|---|
| Engineering Hiring | 340 | 4.1% | < 2% | OK |
| Sales Hiring | 280 | 1.6% | < 2% | Review required |
| Operations | 190 | 5.3% | < 2% | OK |
Friction controls in place:
- Reviewers must select a disposition before approving: STRONG_FIT | ADEQUATE | BORDERLINE | STRETCH
- One-click approval disabled for all queues with AI confidence < 0.90
- Calibration exercise run quarterly: reviewers assess 10 seeded known-incorrect outputs; detection rate < 70% triggers mandatory re-training