AI Explainability Templates and Validation Checklist

Setting and delivering the right level of explanation per system. A requirements matrix, a customer-facing adverse-action explanation template, a regulator explanation document, and a validation checklist that asks whether an explanation is actually faithful to the model.

Who this is for: The owner deciding what explanation each AI system must produce, for whom, and how it is validated.

Download the kit (Markdown) ↓4 artifacts. Every table also copies as CSV.

1. Explainability requirements matrix

Spreadsheet

System by risk tier, mapped to the explanation type and audiences it owes.

Template

System	Risk tier	Explanation approach	Audiences owed an explanation	Legal driver
<system>	Minimal / Limited / High	none / global feature importance / per-decision factors / inherently interpretable	customer / regulator / internal reviewer	GDPR Art. 22 / FCRA / EU AI Act Art. 13 / none

Worked example

System	Risk tier	Explanation approach	Audiences	Legal driver
resume-screener	High	per-decision top factors + counterfactual; inherently interpretable scoring head	internal reviewer; candidate on request	EU AI Act Art. 13-14; local hiring transparency
credit-decisioning	High	per-decision reason codes	customer (adverse action); regulator	FCRA / ECOA; GDPR Art. 22
support-copilot	Limited	none per-decision; system-level description	internal	none

Acceptance criteria

✓Every system has a stated explanation approach tied to its risk tier.
✓The audiences are named, and the legal driver for each is identified.
✓Where the law requires a specific format (FCRA reason codes, GDPR meaningful information), the matrix says so.

2. Customer-facing explanation template

Document

Plain-language, adverse-action-style explanation an individual can act on.

Template

Plain language. No model jargon. Actionable where possible.

Decision: what was decided
Main reasons: the top 3-4 factors, in everyday terms, most important first
What would have helped: the closest change that could have altered the outcome, if one exists
Human involvement: whether a person reviewed it and how to reach one
How to contest: the appeal or reconsideration route, with a timeframe
Data used: the categories of information considered, and how to request the specifics

Worked example

Decision: Your application was not advanced to interview at this stage.
Main reasons: (1) The listed experience was below what the role requires. (2) Fewer of the role's core skills were evidenced in the application. (3) No portfolio or work samples were included.
What would have helped: Adding work samples and listing the core skills explicitly tends to change this outcome for similar applications.
Human involvement: A recruiter reviewed this decision. You can reply to this email to reach them.
How to contest: Request a reconsideration within 14 days by replying with any additional information.
Data used: Your application form and any attached documents. To request the specific factors, contact privacy@company.example.

Acceptance criteria

✓The explanation uses everyday language and avoids feature names or scores.
✓It states whether a human was involved and how to reach one.
✓It gives a concrete route and timeframe to contest.
✓It is generated as part of the decision flow, not written by hand after a complaint.

3. Regulator / auditor explanation document

Document

The technical explanation a regulator expects: how the system reaches decisions, and the evidence the explanations are sound.

Template

One per high-risk system. Part of the documentation package.

System and version:
Decision logic overview: model type, inputs, how outputs map to decisions
Explanation method: inherently interpretable, or the post-hoc method (SHAP, LIME, reason codes) and its configuration
Fidelity evidence: how you validated that explanations reflect actual model behaviour, with results
Worked examples: 2-3 real decisions with their explanations
Known limits of the explanation: where it is approximate or can mislead
Change control: how explanations are kept in sync when the model changes

Worked example

System: resume-screener v2.4.0
Decision logic: VendorLM-4 features feed a linear scoring head; score bands map to advance / hold / review.
Explanation method: the scoring head is inherently interpretable (signed coefficients); per-decision output lists the top contributing features and a nearest counterfactual.
Fidelity evidence: because the head is linear, per-decision attributions are exact for the head; feature-embedding stability tested by perturbation (mean rank change 0.4 across 1,000 cases).
Worked examples: dec-...823 (score 2, factors: experience, skill match), dec-...901 (score 4), dec-...733 (override case).
Known limits: attributions describe the scoring head, not the upstream embedding; portfolio content is not an input, so the model cannot weigh it.
Change control: any change to features or the head triggers regeneration of the explanation spec and this document.

Acceptance criteria

✓The document explains the actual decision logic, not a generic description of ML.
✓It includes evidence that explanations are faithful, not just that they are produced.
✓It states where the explanation is approximate or could mislead.

4. Explanation validation checklist

Spreadsheet

Confirms an explanation method actually represents the model before you rely on it.

Template

Check	Result	Evidence
Removing the top-cited feature materially changes the output	Y / N	perturbation test
Explanations are stable across near-identical inputs	Y / N	stability test
Explanations agree with a second method (e.g. SHAP vs. permutation importance) at the global level	Y / N	comparison
Counterfactuals offered are actually attainable and do flip the outcome when applied	Y / N	counterfactual check
The explanation covers the whole pipeline, or its scope limit is disclosed	Y / N	scope note
Non-experts can act on the customer-facing version (tested with real users)	Y / N	user test

Worked example

Check	Result	Evidence
Top-feature removal changes output	Y	mean score shift 0.7 band on removal
Stable across near-identical inputs	Y	rank change < 1 for 96% of pairs
Agrees with a second method globally	Y	top-5 features match permutation importance
Counterfactuals attainable and effective	Partial	82% flip when applied; 18% blocked by interaction effects, disclosed
Whole-pipeline or scope disclosed	Y	scope limited to scoring head, stated
Non-experts can act on it	Y	9/10 test users identified a next step

Acceptance criteria

✓The method passed a perturbation or ablation test showing cited features actually drive the output.
✓Explanation stability was measured, not assumed.
✓The customer-facing version was tested with people who are not model experts.

Governance controls this kit produces evidence for

Completing the artifacts above gives you a head start on the evidence requirements for these controls.

ALC-004

The matrix, regulator document, and validation checklist are the explainability documentation for each system.

HOC-001

The required explanation level is derived from each system's risk classification.

HOC-003

Per-decision factor explanations are the information the output review workflow depends on.

CMP-007

The regulator explanation document supplies the transparency evidence an EU AI Act conformity assessment requires.

ALC-002

Worked-example explanations attached to decisions strengthen the high-risk audit trail.

This kit backs one playbook. Read the full guidance for the reasoning behind each artifact.

Decide what to implement next

Assess your governance gaps, then create an action plan with owners and target dates. Build and export without an account; sign in when you want to save your plan.

Start the AI governance assessment →