Reviewer Competency Requirements
Define minimum competency requirements for humans who review, approve, or override AI-generated outputs in high-risk contexts.
Objective
Ensure that human oversight is substantive rather than procedural by requiring reviewers to have the domain knowledge necessary to critically evaluate AI outputs.
Maturity Levels
Initial
No competency requirements exist; any employee can be assigned to review AI outputs.
Developing
Competency expectations are informally understood but not documented or enforced.
Defined
Written competency requirements are defined per use case, with a documented training curriculum and sign-off process.
Managed
Competency is assessed before assignment and re-evaluated periodically; records are maintained.
Optimizing
Competency framework is updated as AI capabilities evolve; training effectiveness is measured through calibration exercises.
Evidence Requirements
What an auditor or assessor would expect to see for this control.
- —Competency matrix documenting domain and AI literacy requirements per use case, version-controlled and approved
- —Training completion certificates for all assigned reviewers, retained in HR system with completion date
- —Reviewer role assignment records confirming certification prerequisite was verified before assignment
- —Recertification status report showing current, lapsed, and upcoming-due certifications by reviewer
- —Calibration exercise or assessment results used to verify competency at initial certification and recertification
Implementation Notes
Key steps
- Define competency requirements per use case, not generically — a loan underwriter reviewing AI credit decisions needs different skills than a clinician reviewing AI diagnostic suggestions.
- Distinguish between domain competency (understanding the subject matter) and AI literacy (understanding model limitations, hallucination risk, and confidence signals).
- Document training completion and maintain records — this is a key audit evidence requirement under EU AI Act Art. 26.
- For agentic systems, include training on how to interpret agent action logs, not just final outputs.
Example Implementation
Financial services firm with AI-assisted fraud detection and credit scoring
Reviewer Competency Matrix
| Use Case | System | Domain Requirement | AI Literacy Requirement | Recertification |
|---|---|---|---|---|
| Fraud Detection | TxnGuard v3 | AML analyst Level II+ | Module 1 + Module 2 | Annual |
| Credit Scoring | LoanScoreAI | Licensed underwriter | Module 1 + Module 3 | Annual + supervisor sign-off |
| Document Extraction | DocParse | Operations analyst | Module 1 | Annual |
AI Literacy Modules:
- Understanding model outputs, confidence scores, and failure modes (required for all reviewers)
- Recognizing distribution shift and out-of-distribution inputs
- Interpreting attribution/explanation outputs for model-specific systems
Records: Training completion certificates retained in HR system; reviewer assignments blocked until current certification confirmed
Control Details
- Control ID
- HOC-005
- Domain
- Human Oversight
- Typical owner
- HR / AI Governance Team
- Implementation effort
- Medium effort
- Agent-relevant
- Yes