All governance templates →← How do we detect and mitigate algorithmic bias?

Implementation Kit

AI Bias Testing Templates and Remediation Checklist

A run-it-yourself bias testing protocol: a disparate-impact analysis with a four-fifths calculator, a subgroup confusion-matrix template, a remediation decision tree keyed to the source of bias, and an audit trail that records before-and-after metrics.

Who this is for: The data scientist or model risk analyst who has to test a system that affects people and defend the result.

Download the kit (Markdown) ↓4 artifacts. Every table also copies as CSV.

1. Disparate impact analysis

Spreadsheet

Selection rate by group, ratio against the most-selected group, screened with the four-fifths rule.

Template

Group	Selection rate	Ratio vs. highest group	Pass four-fifths (>= 0.80)?
<group A>	rate	ratio	Y / N
<group B>
<group C>

Worked example

Group	N	Selected	Selection rate	Ratio vs. highest	Pass four-fifths?
Group A	8,200	2,050	0.250	1.00 (highest)	Y
Group B	3,100	682	0.220	0.88	Y
Group C	1,450	261	0.180	0.72	N
Screen result: Group C fails the four-fifths screen (0.72). Proceed to subgroup confusion matrix and source diagnosis.

Acceptance criteria

✓Selection rates are computed for every relevant protected group, and for key intersections where sample size allows.
✓The four-fifths ratio is calculated against the highest-selected group, and any failing group is flagged for diagnosis.
✓Sample sizes are shown, and groups too small to test are named rather than dropped silently.

2. Confusion matrix by subgroup

Spreadsheet

For classifiers: error rates per group, so a model can be "accurate overall" and still fail one group.

Template

Group	TPR (recall)	FPR	FNR	Precision	Notes
<group A>
<group B>
Overall

Worked example

Group	TPR	FPR	FNR	Precision	Notes
Group A	0.84	0.09	0.16	0.81	baseline
Group B	0.83	0.10	0.17	0.80	within tolerance
Group C	0.68	0.11	0.32	0.79	FNR nearly 2x baseline: qualified people in Group C are missed more often
Overall	0.82	0.10	0.18	0.80
For scoring models, replace with a score-distribution comparison (mean, spread, and the rate above the decision threshold per group).

Acceptance criteria

✓Error rates are reported per group, not only in aggregate.
✓The metric that matters for the use case is identified (false negatives, meaning qualified people wrongly turned down, for opportunity-allocation; false positives, meaning people wrongly flagged, for punitive decisions).
✓A gap greater than the pre-set tolerance is carried into source diagnosis.

3. Bias remediation decision tree

Spreadsheet

Which mitigation to try, based on where the bias comes from.

Template

Diagnosed source	Signs	Remediation options	Re-test focus
Training data representation	Under-represented group; label scarcity	Resample, targeted data collection, reweighting	Selection rate parity
Label bias	Historical labels encode past discrimination	Relabel, proxy-label audit, outcome redefinition	Both selection rate and error rates
Proxy variables	A feature correlates with a protected attribute and carries little independent signal	Drop or transform the feature; monitored group-aware constraint	Error-rate parity
Model design / threshold	Single global threshold; loss not aligned to fairness goal	Post-processing (group thresholds where lawful), calibrated equalized odds	The metric tied to the chosen fairness definition
Deployment context	Model fine on test data, biased in the live pipeline	Fix upstream data capture; re-test on production sample	Live-sample disparate impact

Worked example

Diagnosis and outcome for the Group C four-fifths failure.

Step	Finding
Signs	FNR ~2x baseline, FPR balanced
Diagnosed source	Label scarcity plus a proxy feature ("years at a single employer") that adds little signal once experience is controlled for
Remediation	Drop the tenure proxy; add targeted labeled data for Group C roles
Re-test focus	Selection-rate ratio and FNR gap
Result after retrain	Ratio 0.72 to 0.86; FNR gap 0.16 to 0.05

Acceptance criteria

✓The source of bias is diagnosed before a mitigation is chosen, not guessed.
✓The chosen technique is lawful in the jurisdiction (group-specific thresholds are restricted in some).
✓Re-test targets the specific metric the remediation was meant to move.

4. Bias testing audit trail

Spreadsheet

The defensible record: what was tested, what failed, what was done, and the before-and-after numbers.

Template

Date	System / version	Groups tested	Screen result	Diagnosis	Remediation	Before	After	Re-test date	Sign-off

Worked example

Date	System / version	Groups tested	Screen result	Diagnosis	Remediation	Before	After	Re-test date	Sign-off
2026-06-12	resume-screener v2.3	sex, race/ethnicity, age band	sex ratio 0.83 fail	tenure proxy + label scarcity	drop tenure weight; add data	ratio 0.83, FNR gap 0.14	ratio 0.90, FNR gap 0.05	2026-07-15	R. Nkemelu
2026-08-30	resume-screener v2.4	sex, race/ethnicity, age band	pass (lowest ratio 0.88)	n/a	none	ratio 0.88	n/a	2026-09-30	R. Nkemelu

Acceptance criteria

✓Every test run is recorded, including passes.
✓Before and after metrics are captured for every remediation.
✓Each entry has a named sign-off and a scheduled re-test date.

Governance controls this kit produces evidence for

Completing the artifacts above gives you a head start on the evidence requirements for these controls.

MON-003

AI Bias and Fairness Monitoring

The protocol and audit trail are the bias and fairness monitoring record, with a re-test cadence.

HOC-001

AI System Risk Classification

Testing is scoped to systems that a risk classification marks as affecting people.

ALC-004

AI Explainability Documentation

The audit trail is explainability and fairness documentation for the system file.

MGV-006

RAI Benchmark-Aligned Evaluation Framework

The disparate-impact and subgroup metrics feed a benchmark-aligned evaluation framework.

CMP-007

EU AI Act Conformity Assessment and FRIA Process

The audit trail supplies the bias-testing evidence an EU AI Act conformity assessment requires for a high-risk system.

This kit backs one playbook. Read the full guidance for the reasoning behind each artifact.