# AI System Audit Checklist and Findings Report Template How to audit one AI system against its obligations: a scope and methodology template, a document review checklist by tier, a bias-testing verification worksheet, a human oversight interview guide, and a findings template. **Who this is for:** The internal auditor or external assessor examining a specific AI system. Source playbook: https://aigovernance.com/playbook/audit-ai-system-for-compliance --- ## Audit scope and methodology template _Sets what is being audited, against what, and how, before fieldwork._ ### Template > Agreed with the system owner before fieldwork. - **System and version:** - **Obligations in scope:** the specific regulations, internal standards, and control set - **Audit questions:** the 5-8 questions the audit will answer - **Methods:** document review, re-performance (bias tests, sample decisions), interviews, system observation - **Sampling approach:** how decisions and controls are selected for testing, not owner-chosen - **Out of scope:** and why - **Timeline, team, and reporting route** ### Worked example - **System:** resume-screener v2.4.0 - **Obligations:** EU AI Act Chapter III Section 2; NYC LL144; internal controls HOC-001/002, MON-003, ALC-001/002. - **Audit questions:** Is classification correct? Is bias testing run and acted on? Is human oversight meaningful in practice? Are decisions reconstructable? Is the documentation current? - **Methods:** document review; re-perform disparate-impact on a fresh sample; trace 25 random decisions; interview 5 reviewers; observe the review UI. - **Sampling:** decisions selected by the auditor via random seed from the full log; controls tested on the auditor's sample. - **Out of scope:** the base model vendor's internals (covered by vendor DD). - **Timeline:** 3 weeks; report to the Audit Committee. ### Acceptance criteria - Scope names the specific obligations and controls, not "compliance" in general. - Sampling is auditor-controlled, not owner-provided. - The audit questions are answerable with the chosen methods. --- ## Document review checklist _The documents that should exist for the system's tier, with a status per document._ ### Template | Document | Required at this tier? | Present | Current | Adequate | Notes | |---|---|---|---|---|---| | Risk assessment (pre-deployment) | | Y/N | Y/N | Y/N | | | Model card | | | | | | | Data governance record | | | | | | | Bias / fairness evaluation | | | | | | | Human oversight arrangement | | | | | | | Technical documentation (Annex IV, if EU high-risk) | | | | | | | Change / version history | | | | | | | Monitoring plan and recent results | | | | | | | Incident history | | | | | | ### Worked example | Document | Required? | Present | Current | Adequate | Notes | |---|---|---|---|---|---| | Risk assessment | Yes | Y | Y | Y | signed 2026-09 | | Model card | Yes | Y | N | Partial | still describes v2.3 | | Bias evaluation | Yes | Y | Y | Y | 2026-08 | | Human oversight arrangement | Yes | N | - | - | finding: not written up | | Annex IV technical doc | Yes | Y | Partial | Partial | sections 6-7 missing | | Monitoring plan + results | Yes | Y | Y | Partial | monthly only; no alerting | ### Acceptance criteria - Every required document is checked for presence, currency, and adequacy separately. - "Present but stale" and "present but inadequate" are captured, not just present or missing. - Each shortfall becomes a finding. --- ## Bias testing verification worksheet _The auditor re-performs, rather than relying on the owner's report._ ### Template | Step | Owner-reported | Auditor re-performed | Match? | Note | |---|---|---|---|---| | Groups tested | | | | | | Disparate-impact ratios | | | | | | Subgroup error rates | | | | | | Threshold used | | | | | | Remediation claimed | verify it is actually in the deployed model | | | | ### Worked example | Step | Owner-reported | Auditor re-performed | Match? | Note | |---|---|---|---|---| | Groups tested | sex, race/ethnicity, age band | same, plus sex x race | partial | owner did not test the intersection | | Disparate-impact ratio (sex) | 0.88 | 0.86 on the auditor sample | close | within sampling variance | | Threshold | 0.80 | 0.80 | yes | | | Remediation (tenure-weight cap) | applied | confirmed in the deployed config | yes | | ### Acceptance criteria - The auditor runs the test on an independent sample, not the owner's dataset. - Claimed remediations are verified in the deployed system, not just in a report. - Discrepancies between reported and re-performed results are documented. --- ## Human oversight interview guide _Questions for reviewers that reveal whether oversight is real._ ### Template > Interview 3-5 reviewers separately. Open questions, then probes. - Walk me through the last case you reviewed. What did you see on screen? - How much time do you usually spend per case? What happens if you run out of time? - When did you last override the AI? What made you do it? - What would you need to override that you do not have now? - Are you measured on speed or volume? Does that affect how you review? - What are the AI's known weaknesses for this decision type? - What happens after you make a decision? Who checks it? ### Worked example **Themes from 5 reviewer interviews (resume-screener):** - 3 of 5 could not name a recent override; "the scores are usually right". - Average self-reported time: 40-60 seconds; UI auto-advances at 60s. - All 5 are on a daily throughput target. - 2 of 5 could describe a known model weakness; 3 could not. - No second check after a decision unless the candidate complains. Assessment: oversight is not currently meaningful. Corroborates the override-rate data. ### Acceptance criteria - Reviewers are interviewed individually, not in a group with their manager. - Answers are corroborated against override-rate and timing data. - The guide probes for time pressure, throughput targets, and knowledge of model weaknesses. --- ## AI audit findings template _Each finding: what, the evidence, severity, owner, and a remediation date._ ### Template | ID | Finding | Evidence | Obligation / control affected | Severity | Owner | Remediation | Due | |---|---|---|---|---|---|---|---| | A-1 | | | | high / medium / low | | | YYYY-MM-DD | ### Worked example | ID | Finding | Evidence | Affected | Severity | Owner | Remediation | Due | |---|---|---|---|---|---|---|---| | A-1 | Human oversight is not meaningful: reviewers do not have time or authority in practice | 5 interviews; override rate 4% vs 11% median; 60s auto-advance | EU AI Act Art. 14; HOC-002 | High | Head of Talent | remove auto-advance; raise per-case time; drop throughput target for this queue | 2026-11-15 | | A-2 | Model card describes v2.3, model is v2.4 | doc review | ALC-004 | Medium | DS team | refresh on the versioning trigger | 2026-10-15 | | A-3 | Monitoring has no alerting; monthly only | monitoring config | MON-002/003 | Medium | ML Platform | add thresholds + alerts | 2026-11-01 | ### Acceptance criteria - Each finding cites specific evidence and the exact obligation or control it breaches. - Severity, owner, and a due date are set for every finding. - Findings are tracked to closure and re-tested. --- ## Governance controls this kit produces evidence for - **ALC-005**: The whole kit is a regulatory audit-readiness exercise and its evidence for one system. - **MON-003**: The bias verification worksheet independently checks fairness monitoring. - **HOC-003**: The oversight interview guide tests whether the output review workflow is effective. - **CMP-007**: The document review checklist maps to EU AI Act conformity requirements. - **ALC-002**: Tracing sample decisions tests the high-risk audit trail.