# AI Fairness Metric Selection Worksheet and Test Plan The step before testing: choosing the fairness definition that fits your use case, since you cannot satisfy all of them at once. A selection worksheet, an intersectional test plan, and a re-test schedule. Pair it with the Bias Testing Kit for the mechanics. **Who this is for:** The model owner and legal partner deciding what "fair" means for this specific system before any metric is computed. Source playbook: https://aigovernance.com/playbook/measuring-and-mitigating-algorithmic-bias --- ## Fairness definition selection worksheet _Walks from use-case properties to a recommended fairness metric, with the trade-off made explicit._ ### Template | Question | Answer | Implication | |---|---|---| | Is the decision allocating an opportunity (hiring, credit, housing) or a burden (fraud flag, audit, moderation)? | | Opportunity: watch false negatives. Burden: watch false positives. | | Are base rates genuinely equal across groups, or is that assumption contested? | | If base rates differ and are legitimate, demographic parity may be the wrong target | | Is there a legally recognised protected class and a disparate-impact standard that applies? | | The four-fifths rule becomes a hard screen, not just a diagnostic | | Do individuals reasonably expect similar people to be treated similarly? | | Adds an individual-fairness check on top of a group metric | | Can we lawfully use group-aware post-processing in this jurisdiction? | | Constrains which remediations are available | | **Recommended primary metric:** | | e.g. equalized odds, demographic parity, calibration within groups | | **Secondary checks:** | | | | **Accepted trade-off:** | | which fairness property we are knowingly not optimising, and why | ### Worked example | Question | Answer | Implication | |---|---|---| | Opportunity or burden? | Opportunity (hiring screen) | Prioritise false-negative parity | | Base rates equal? | Contested; historical hiring skew present | Demographic parity risky as a sole target | | Protected class + disparate-impact standard? | Yes (sex, race; Title VII, NYC LL144) | Four-fifths is a hard screen | | Individual-fairness expectation? | Yes | Add a "similar applicants, similar scores" spot check | | Group-aware post-processing lawful? | Restricted for hiring in the US | Prefer pre-processing and feature fixes | | **Recommended primary metric** | Equalized odds (focus on FNR parity) | | | **Secondary checks** | Four-fifths selection-rate screen; score-distribution comparison | | | **Accepted trade-off** | Not enforcing strict demographic parity; documented with Legal | | ### Acceptance criteria - The primary fairness metric is chosen and written down before any results are seen. - The worksheet names the fairness property being traded away and why. - Legal has reviewed the choice where a protected class is involved. --- ## Intersectional test plan _Which group combinations to test, and which are too small to be reliable._ ### Template | Attribute combination | Estimated N in eval set | Test? | If not, why | |---|---|---|---| | sex | | Yes | | | race/ethnicity | | Yes | | | age band | | Yes | | | sex x race/ethnicity | | Yes / No | sample size | | sex x age band | | Yes / No | | | race/ethnicity x age band | | Yes / No | | ### Worked example | Combination | N in eval set | Test? | Note | |---|---|---|---| | sex | 38,000 | Yes | | | race/ethnicity | 38,000 | Yes | | | age band | 38,000 | Yes | | | sex x race/ethnicity | ~1,100 smallest cell | Yes | report with confidence intervals | | sex x age band | ~2,400 smallest cell | Yes | | | race/ethnicity x age band | ~140 smallest cell | No | cell too small; monitor in production aggregate | ### Acceptance criteria - Single attributes are always tested; intersections are tested where the smallest cell supports it. - Untested combinations are named with the reason, and picked up in production monitoring. - Results for small cells are reported with uncertainty, not as point estimates. --- ## Fairness re-test schedule _When the chosen metrics are recomputed, tied to the model review cycle._ ### Template | System | Primary metric | Routine cadence | Trigger events | Owner | Next run | |---|---|---|---|---|---| | | | monthly / quarterly | model change; data refresh; population shift; incident | | YYYY-MM-DD | ### Worked example | System | Primary metric | Routine cadence | Trigger events | Owner | Next run | |---|---|---|---|---|---| | resume-screener | equalized odds (FNR parity) + four-fifths screen | monthly | any model version; quarterly data refresh; complaint | DS team | 2026-10-01 | ### Acceptance criteria - Re-test cadence matches the model review cadence at minimum. - Trigger events force an out-of-cycle run. - Results land in the bias testing audit trail. --- ## Governance controls this kit produces evidence for - **MON-003**: The chosen metric plus the re-test schedule define what fairness monitoring runs and when. - **HOC-001**: The worksheet ties the fairness approach to the system's risk classification and use-case type. - **MGV-006**: The selected fairness definition is the fairness axis of a benchmark-aligned evaluation framework.