# AI Bias Testing Templates and Remediation Checklist A run-it-yourself bias testing protocol: a disparate-impact analysis with a four-fifths calculator, a subgroup confusion-matrix template, a remediation decision tree keyed to the source of bias, and an audit trail that records before-and-after metrics. **Who this is for:** The data scientist or model risk analyst who has to test a system that affects people and defend the result. Source playbook: https://aigovernance.com/playbook/algorithmic-bias-detection-and-mitigation --- ## Disparate impact analysis _Selection rate by group, ratio against the most-selected group, screened with the four-fifths rule._ ### Template | Group | N | Selected (positive outcome) | Selection rate | Ratio vs. highest group | Pass four-fifths (>= 0.80)? | |---|---|---|---|---|---| | | | | rate | ratio | Y / N | | | | | | | | | | | | | | | ### Worked example | Group | N | Selected | Selection rate | Ratio vs. highest | Pass four-fifths? | |---|---|---|---|---|---| | Group A | 8,200 | 2,050 | 0.250 | 1.00 (highest) | Y | | Group B | 3,100 | 682 | 0.220 | 0.88 | Y | | Group C | 1,450 | 261 | 0.180 | 0.72 | N | Screen result: Group C fails the four-fifths screen (0.72). Proceed to subgroup confusion matrix and source diagnosis. ### Acceptance criteria - Selection rates are computed for every relevant protected group, and for key intersections where sample size allows. - The four-fifths ratio is calculated against the highest-selected group, and any failing group is flagged for diagnosis. - Sample sizes are shown, and groups too small to test are named rather than dropped silently. --- ## Confusion matrix by subgroup _For classifiers: error rates per group, so a model can be "accurate overall" and still fail one group._ ### Template | Group | TPR (recall) | FPR | FNR | Precision | Notes | |---|---|---|---|---|---| | | | | | | | | | | | | | | | Overall | | | | | | ### Worked example | Group | TPR | FPR | FNR | Precision | Notes | |---|---|---|---|---|---| | Group A | 0.84 | 0.09 | 0.16 | 0.81 | baseline | | Group B | 0.83 | 0.10 | 0.17 | 0.80 | within tolerance | | Group C | 0.68 | 0.11 | 0.32 | 0.79 | FNR nearly 2x baseline: qualified people in Group C are missed more often | | Overall | 0.82 | 0.10 | 0.18 | 0.80 | | For scoring models, replace with a score-distribution comparison (mean, spread, and the rate above the decision threshold per group). ### Acceptance criteria - Error rates are reported per group, not only in aggregate. - The metric that matters for the use case is identified (false negatives for opportunity-allocation, false positives for punitive decisions). - A gap greater than the pre-set tolerance is carried into source diagnosis. --- ## Bias remediation decision tree _Which mitigation to try, based on where the bias comes from._ ### Template | Diagnosed source | Signs | Remediation options | Re-test focus | |---|---|---|---| | Training data representation | Under-represented group; label scarcity | Resample, targeted data collection, reweighting | Selection rate parity | | Label bias | Historical labels encode past discrimination | Relabel, proxy-label audit, outcome redefinition | Both selection rate and error rates | | Proxy variables | A feature correlates with a protected attribute and carries little independent signal | Drop or transform the feature; monitored group-aware constraint | Error-rate parity | | Model design / threshold | Single global threshold; loss not aligned to fairness goal | Post-processing (group thresholds where lawful), calibrated equalized odds | The metric tied to the chosen fairness definition | | Deployment context | Model fine on test data, biased in the live pipeline | Fix upstream data capture; re-test on production sample | Live-sample disparate impact | ### Worked example > Diagnosis and outcome for the Group C four-fifths failure. | Step | Finding | |---|---| | Signs | FNR ~2x baseline, FPR balanced | | Diagnosed source | Label scarcity plus a proxy feature ("years at a single employer") that adds little signal once experience is controlled for | | Remediation | Drop the tenure proxy; add targeted labeled data for Group C roles | | Re-test focus | Selection-rate ratio and FNR gap | | Result after retrain | Ratio 0.72 to 0.86; FNR gap 0.16 to 0.05 | ### Acceptance criteria - The source of bias is diagnosed before a mitigation is chosen, not guessed. - The chosen technique is lawful in the jurisdiction (group-specific thresholds are restricted in some). - Re-test targets the specific metric the remediation was meant to move. --- ## Bias testing audit trail _The defensible record: what was tested, what failed, what was done, and the before-and-after numbers._ ### Template | Date | System / version | Groups tested | Screen result | Diagnosis | Remediation | Before | After | Re-test date | Sign-off | |---|---|---|---|---|---|---|---|---|---| | | | | | | | | | | | ### Worked example | Date | System / version | Groups tested | Screen result | Diagnosis | Remediation | Before | After | Re-test date | Sign-off | |---|---|---|---|---|---|---|---|---|---| | 2026-06-12 | resume-screener v2.3 | sex, race/ethnicity, age band | sex ratio 0.83 fail | tenure proxy + label scarcity | drop tenure weight; add data | ratio 0.83, FNR gap 0.14 | ratio 0.90, FNR gap 0.05 | 2026-07-15 | R. Nkemelu | | 2026-08-30 | resume-screener v2.4 | sex, race/ethnicity, age band | pass (lowest ratio 0.88) | n/a | none | ratio 0.88 | n/a | 2026-09-30 | R. Nkemelu | ### Acceptance criteria - Every test run is recorded, including passes. - Before and after metrics are captured for every remediation. - Each entry has a named sign-off and a scheduled re-test date. --- ## Governance controls this kit produces evidence for - **MON-003**: The protocol and audit trail are the bias and fairness monitoring record, with a re-test cadence. - **HOC-001**: Testing is scoped to systems that a risk classification marks as affecting people. - **ALC-004**: The audit trail is explainability and fairness documentation for the system file. - **MGV-006**: The disparate-impact and subgroup metrics feed a benchmark-aligned evaluation framework. - **CMP-007**: The audit trail supplies the bias-testing evidence an EU AI Act conformity assessment requires for a high-risk system.