AI Decision Logging
Added May 2026
Log inputs, outputs, model versions, confidence scores, and context for every AI decision affecting people or business outcomes.
Objective
Create an auditable record of AI-driven decisions to support regulatory compliance, incident investigation, and accountability.
Maturity Levels
Initial
AI decisions are not logged; only application-level errors are captured.
Developing
Some AI outputs are logged but coverage is incomplete and log format is inconsistent.
Defined
A defined logging schema captures inputs, outputs, model version, and timestamp for all in-scope AI decisions.
Managed
Logs are centralized, searchable, and reviewed regularly; completeness is verified through sampling.
Optimizing
Decision logs feed automated monitoring, drift detection (flagging model drift, when accuracy slips over time), and model performance dashboards.
Evidence Requirements
What an auditor or assessor would expect to see for this control.
- —Log schema documentation (the agreed log format) defining required fields, storage type, access controls, and retention period
- —Sample log entries confirming all required fields are populated for a representative set of decisions
- —Storage integrity configuration confirming append-only (new entries can be added but existing ones cannot be changed) or immutable log storage with appropriate access controls
- —Log completeness sampling report showing what percentage of in-scope decisions are being captured
- —Retention compliance evidence confirming logs are maintained for the required period
Implementation Notes
Key steps
- Define your minimum log record: input (or a hash, a fingerprint that confirms the input without revealing it, if personally identifiable information (PII) must be excluded), output, model version/ID, timestamp, user/session identifier, and confidence score where available.
- Log at the AI decision point, not at the application layer (the wider software the AI sits inside); application logs often omit the information needed for AI accountability.
- For high-risk systems, log the full input context: documents fetched to inform answers in RAG (retrieval-augmented generation) systems, or tool results from agent systems that take actions.
- Ensure logs are immutable (cannot be changed once written) and access-controlled; the ability to delete or modify decision logs is a material compliance risk.
Example Implementation
Insurance company using AI to assist underwriters with risk scoring
AI Decision Log Schema: Underwriting Risk Scorer
Minimum log record (one entry per scoring request):
| Field | Example Value | Notes |
|---|---|---|
| log_id | ulid-01HXYZ... | Unique, append-only identifier |
| session_id | sess-abc123 | Links to underwriter session |
| model_id | risk-scorer-v2.4.1 | Full version including prompt hash |
| input_hash | sha256:3a9f... | SHA-256 of full input; raw input stored separately in PHI store |
| risk_score | 0.73 | Model output |
| confidence | 0.88 | Model confidence |
| top_features | ["prior_claims", "industry_code"] | Top 3 contributing features |
| timestamp | 2026-05-09T14:22:01Z | UTC |
| underwriter_id | uw-00412 | Reviewer identity |
Storage: Append-only; no delete or update; access restricted to Compliance, AI Governance, and Legal
Retention: 7 years from decision date (state insurance regulation requirement)