AI Fairness Metric Selection Worksheet and Test Plan

The step before testing: choosing the fairness definition that fits your use case, since you cannot satisfy all of them at once. A selection worksheet, an intersectional test plan, and a re-test schedule. Pair it with the Bias Testing Kit for the mechanics.

Who this is for: The model owner and legal partner deciding what "fair" means for this specific system before any metric is computed.

Download the kit (Markdown) ↓3 artifacts. Every table also copies as CSV.

1. Fairness definition selection worksheet

Spreadsheet

Walks from use-case properties to a recommended fairness metric, with the trade-off made explicit.

Template

Question	Answer	Implication
Is the decision allocating an opportunity (hiring, credit, housing) or a burden (fraud flag, audit, moderation)?		Opportunity: watch false negatives. Burden: watch false positives.
Are base rates genuinely equal across groups, or is that assumption contested?		If base rates differ and are legitimate, demographic parity may be the wrong target
Is there a legally recognised protected class and a disparate-impact standard that applies?		The four-fifths rule becomes a hard screen, not just a diagnostic
Do individuals reasonably expect similar people to be treated similarly?		Adds an individual-fairness check on top of a group metric
Can we lawfully use group-aware post-processing in this jurisdiction?		Constrains which remediations are available
Recommended primary metric:		e.g. equalized odds, demographic parity, calibration within groups
Secondary checks:
Accepted trade-off:		which fairness property we are knowingly not optimising, and why

Worked example

Question	Answer	Implication
Opportunity or burden?	Opportunity (hiring screen)	Prioritise false-negative parity
Base rates equal?	Contested; historical hiring skew present	Demographic parity risky as a sole target
Protected class + disparate-impact standard?	Yes (sex, race; Title VII, NYC LL144)	Four-fifths is a hard screen
Individual-fairness expectation?	Yes	Add a "similar applicants, similar scores" spot check
Group-aware post-processing lawful?	Restricted for hiring in the US	Prefer pre-processing and feature fixes
Recommended primary metric	Equalized odds (focus on FNR parity)
Secondary checks	Four-fifths selection-rate screen; score-distribution comparison
Accepted trade-off	Not enforcing strict demographic parity; documented with Legal

Acceptance criteria

✓The primary fairness metric is chosen and written down before any results are seen.
✓The worksheet names the fairness property being traded away and why.
✓Legal has reviewed the choice where a protected class is involved.

2. Intersectional test plan

Spreadsheet

Which group combinations to test, and which are too small to be reliable.

Template

Attribute combination	Test?	If not, why
sex	Yes
race/ethnicity	Yes
age band	Yes
sex x race/ethnicity	Yes / No	sample size
sex x age band	Yes / No
race/ethnicity x age band	Yes / No

Worked example

Combination	N in eval set	Test?	Note
sex	38,000	Yes
race/ethnicity	38,000	Yes
age band	38,000	Yes
sex x race/ethnicity	~1,100 smallest cell	Yes	report with confidence intervals
sex x age band	~2,400 smallest cell	Yes
race/ethnicity x age band	~140 smallest cell	No	cell too small; monitor in production aggregate

Acceptance criteria

✓Single attributes are always tested; intersections are tested where the smallest cell supports it.
✓Untested combinations are named with the reason, and picked up in production monitoring.
✓Results for small cells are reported with uncertainty, not as point estimates.

3. Fairness re-test schedule

Spreadsheet

When the chosen metrics are recomputed, tied to the model review cycle.

Template

System	Primary metric	Routine cadence	Trigger events	Owner	Next run
<system>	<metric>	monthly / quarterly	model change; data refresh; population shift; incident	<name>	YYYY-MM-DD

Worked example

System	Primary metric	Routine cadence	Trigger events	Owner	Next run
resume-screener	equalized odds (FNR parity) + four-fifths screen	monthly	any model version; quarterly data refresh; complaint	DS team	2026-10-01

Acceptance criteria

✓Re-test cadence matches the model review cadence at minimum.
✓Trigger events force an out-of-cycle run.
✓Results land in the bias testing audit trail.

Governance controls this kit produces evidence for

Completing the artifacts above gives you a head start on the evidence requirements for these controls.

MON-003

AI Bias and Fairness Monitoring

The chosen metric plus the re-test schedule define what fairness monitoring runs and when.

HOC-001

AI System Risk Classification

The worksheet ties the fairness approach to the system's risk classification and use-case type.

MGV-006

RAI Benchmark-Aligned Evaluation Framework

The selected fairness definition is the fairness axis of a benchmark-aligned evaluation framework.

This kit backs one playbook. Read the full guidance for the reasoning behind each artifact.