Behavioral Anomaly Detection for Agentic Systems
Added May 2026
Monitor AI agents for unexpected action sequences, data or system access, and behavior inconsistent with assigned tasks.
Objective
Detect likely prompt injection (hidden instructions planted in content an agent reads), unintended goals, unlocked restricted abilities, or self-expanding scope in live agents before any harmful or unauthorized action.
Maturity Levels
Initial
Agent behavior is not monitored beyond basic error logging; anomalies are only discovered through downstream effects.
Developing
Some alerting exists for obvious failures (tool errors, timeouts) but behavioral patterns are not analyzed.
Defined
Baseline behavioral profiles are established for each agent; alerts fire when action sequences, resource access patterns, or output characteristics fall outside the baseline.
Managed
Anomaly signals are triaged by severity; low-severity findings feed into periodic reviews while high-severity findings trigger immediate investigation and potential halt.
Optimizing
Behavioral baselines update continuously with supervised learning (automated learning guided by human-labeled examples); anomaly detection is integrated with the kill-switch (emergency stop) so confirmed deviations trigger automatic containment.
Evidence Requirements
What an auditor or assessor would expect to see for this control.
- —Behavioral baseline documentation for each production agent including expected tool call patterns, access scopes, and output characteristics
- —Monitoring system configuration showing alert thresholds and severity tiers
- —Alert history with triage records showing how anomalies were investigated and resolved
- —Evidence that agents cannot access monitoring logs (tamper protection)
- —Review records showing baseline updates and rationale for threshold adjustments
Implementation Notes
Key steps
- Define the expected behavioral envelope (the range of normal behavior) for each agent: which tools (software functions or systems it can use) it should call, in what order, with what frequency, and accessing which data sources.
- Establish baselines (records of normal behavior) from observed production behavior over a representative period; flag unusual departures from them.
- Monitor at minimum: tool call sequences and frequency, data access volume and scope, patterns in the agent's output, and how long actions take.
- Define alert tiers: informational (review in next cycle), warning (investigate within 24 hours), critical (halt and investigate immediately).
- Consider whether your monitoring infrastructure is agent-accessible; agents should not be able to read or modify their own behavioral logs.
Example Implementation
Research agent that queries internal document stores and drafts summaries for analysts
Research Agent Behavioral Baseline: Established 2026-04-01
Expected patterns:
- Tool calls: document_search → document_fetch → draft_summary (this sequence, 95%+ of sessions)
- Documents fetched per session: 1–12 (alert at >25)
- Data sources accessed: internal docs only (alert if external API calls detected)
- Output length: 200–800 tokens (alert if >2000 tokens)
- Session duration: 30–180 seconds (alert if >600 seconds)
Alert fired 2026-05-22: Session fetched 47 documents, accessed 3 sources outside expected scope. Investigation: Prompt injection in user query caused agent to enumerate document library. Halted. User query pattern added to input filter.