# AI Model Drift Monitoring Templates and Response Protocol A monitoring setup that catches a model degrading before users do. A metrics specification by model type, an alert threshold worksheet, an ownership register, and a drift response protocol with yellow and red actions. **Who this is for:** The ML or platform owner responsible for a model once it is in production. Source playbook: https://aigovernance.com/playbook/model-drift-monitoring --- ## Monitoring metrics specification _What to measure, by model type. Input drift, output drift, performance, and (where people are affected) fairness._ ### Template | Metric | Model types | What it detects | Source | Cadence | |---|---|---|---|---| | Input feature drift (PSI / KL) | all | inputs shifting away from training distribution | inference logs vs. training baseline | daily | | Prediction distribution shift | all | output mix changing | inference logs | daily | | Performance vs. ground truth | where labels arrive | accuracy / error decay | labels joined to predictions | weekly or on label arrival | | Calibration | scoring models | predicted vs. actual rates diverging | outcomes | weekly | | Fairness metrics | people-affecting | disparity emerging | scored outcomes by group | monthly | | Latency / error rate | all | serving degradation | serving metrics | real-time | ### Worked example | Metric | For fraud-scoring | Baseline | Cadence | |---|---|---|---| | Input drift (PSI) | on 12 key features | training distribution snapshot | daily | | Score distribution | share above the hold threshold | 4.2% at launch | daily | | Precision / recall | vs. confirmed-fraud labels (arrive ~10 days later) | P 0.88 / R 0.71 | weekly | | Calibration | predicted vs. actual fraud rate by score band | curve at launch | weekly | | Adverse-impact ratio | by protected group on holds | 0.94 at launch | monthly | ### Acceptance criteria - Every production model has a metrics spec appropriate to its type. - A training-time baseline is stored for every drift metric. - People-affecting models include a fairness metric on the same cadence as review. --- ## Alert threshold setting worksheet _Turns each metric into a yellow (watch) and red (act) threshold, justified rather than guessed._ ### Template | Metric | Baseline | Normal variation (from history) | Yellow threshold | Red threshold | Rationale | |---|---|---|---|---|---| | | | | | | | ### Worked example | Metric | Baseline | Normal variation | Yellow | Red | Rationale | |---|---|---|---|---|---| | Input drift PSI (max feature) | 0 | up to 0.10 in a stable month | 0.15 | 0.25 | PSI > 0.25 historically preceded a precision drop | | Score share above hold threshold | 4.2% | 3.8-4.8% | outside 3.5-5.5% for 2 days | outside 3-6% for 1 day | fraud rate does move; sustained shift is the signal | | Precision | 0.88 | 0.85-0.90 weekly | < 0.85 | < 0.80 | 0.80 is the business floor for analyst load | | Adverse-impact ratio | 0.94 | 0.90-0.97 | < 0.88 | < 0.80 | 0.80 is the four-fifths line | ### Acceptance criteria - Thresholds are set from historical variation and a business floor, not round numbers alone. - Every metric has both a yellow and a red threshold. - Each threshold has a one-line rationale. --- ## Monitoring ownership register _Each system to its monitoring owner and the review cadence, so alerts have a destination._ ### Template | System | Monitoring owner | On-call route | Dashboard review cadence | Escalation contact | |---|---|---|---|---| | | | | weekly / monthly | | ### Worked example | System | Monitoring owner | On-call route | Review cadence | Escalation | |---|---|---|---|---| | fraud-scoring | ML Platform (D. Reyes) | pager: ml-oncall | weekly dashboard review | Head of Risk Ops | | resume-screener | DS team (R. Nkemelu) | slack: #ds-alerts | monthly + on any yellow | Head of Talent | ### Acceptance criteria - Every production model has a named monitoring owner and an alert route. - Dashboard reviews happen on a set cadence, not only when alerts fire. - An escalation contact is named for red events. --- ## Drift response protocol _What happens at yellow and at red: who is told, what is checked, and the response options._ ### Template | Level | Trigger | Notify within | Diagnostic steps | Response options | |---|---|---|---|---| | Yellow | any metric past its yellow threshold | 1 business day | check for data pipeline change, seasonality, upstream vendor change; widen sampling | monitor closely; schedule retrain; adjust threshold if a legitimate shift | | Red | any metric past red, or yellow sustained N days | same day, to escalation contact | full diagnosis; compare to last known good; check for incident | roll back to last good version; disable and fall back to manual/rules; expedited retrain; pause if people are harmed | ### Worked example > Red event, 2026-09-05, fraud-scoring. | Step | What happened | |---|---| | Trigger | precision fell to 0.78 over two weekly readings; input drift PSI 0.27 on "merchant category" | | Notify | Head of Risk Ops, same day | | Diagnosis | a payments partner changed merchant-category coding; the feature meaning shifted | | Response | rolled back to the rules-engine fallback for affected categories; expedited retrain on remapped categories | | Outcome | back on the model 4 days later at P 0.87; logged as a Sev-2 incident; post-incident review scheduled | ### Acceptance criteria - Yellow and red each have a notification target and a time. - Diagnostic steps come before response, so you do not retrain on a pipeline bug. - Red responses include a fallback that does not depend on the degraded model. --- ## Governance controls this kit produces evidence for - **MON-002**: The metrics spec and thresholds are the model drift detection mechanism. - **MON-001**: The stored training-time baselines are the AI performance baseline. - **MON-005**: The review cadence and response protocol are continuous model evaluation. - **MON-004**: Prediction-distribution and calibration monitoring are output anomaly detection. - **CHM-003**: The red-level rollback and fallback options are the model rollback and emergency shutdown path.