MON · Monitoring & DriftMON-002Medium effort

Model Drift Detection

Monitor production AI systems for data drift, concept drift, and output distribution shifts that indicate degraded or changed model behavior.

Objective

Detect when AI model behavior has meaningfully changed from its deployment baseline, triggering investigation and remediation before harm occurs.

Maturity Levels

Initial

No drift monitoring exists; model degradation is discovered reactively.

Developing

Some output metrics are monitored but input distribution and concept drift are not assessed.

Defined

Automated drift detection monitors input distributions, output distributions, and key performance metrics with defined alert thresholds.

Managed

Drift alerts are triaged and resolved within defined SLAs; patterns are analyzed to identify root causes.

Optimizing

Drift detection models are themselves evaluated for accuracy; thresholds are tuned based on false positive/negative rates.

What an auditor or assessor would expect to see for this control.

—Drift detection configuration documenting methods, metrics monitored, and alert thresholds for each system
—Periodic drift reports showing statistical distribution comparisons against the reference baseline
—Alert records for drift events including detection date, severity, and response timeline
—Root cause analysis records for significant drift events, with remediation actions and outcomes
—Retraining or recalibration records triggered by confirmed drift, including test results before redeployment

Monitor three types of drift separately: data drift (input distribution changes), concept drift (the relationship between inputs and outputs changes), and prediction drift (output distribution changes).
For LLM-based systems, traditional statistical drift detection does not apply directly — monitor proxy metrics: output length distributions, topic distributions, refusal rates, and human feedback signals.
Set drift alerts that notify the model owner, not just the monitoring dashboard — unactioned alerts are as bad as no alerts.
Document what actions are taken in response to drift alerts: investigation, retraining, rollback, or escalation.

Product recommendation model experiencing seasonal purchasing pattern shifts

Monitored drift types and methods:

Drift Type	Method	Metric	Alert Threshold	Cadence
Data drift (input)	Population Stability Index on top 20 features	PSI	> 0.2 (any feature)	Daily
Prediction drift	KS test on score distribution	KS statistic	p < 0.01	Daily
Business outcome drift	CTR on recommended products	Click-through rate	> 15% drop from 7-day rolling avg	Hourly
Latency drift	p95 inference time	ms	> 200ms	Real-time

Alert routing: All alerts → #ml-monitoring Slack channel + model owner email

Response SLAs:

Business outcome alert: investigate within 2 hours; escalate if not resolved in 4
Data/prediction drift: investigate within 1 business day; present findings in weekly model review

Seasonal adjustment: Baseline updated each quarter to account for known seasonal patterns (documented in monitoring runbook)