AI Governance Institute
← News
Research2026-08-06

One in Three Dangerous Agent Requests Bypasses Human Review, Research Finds

What happened

Research reported by The Register used a browser-based simulation drawing on more than 40,000 runs and 409,000 commands to measure how reliably human reviewers catch dangerous requests from AI coding agents. Reviewers failed to block roughly one in three malicious requests overall, and missed credential-exfiltration attempts, a scope violation with serious security consequences, 35 percent of the time. Anthropic provided corroborating telemetry from Claude Code showing users approve approximately 93 percent of all permission prompts, with diligence falling as prompt volume rises. The study sits within a growing body of evidence challenging HITL controls in agentic contexts, including prior incidents such as AI coding agents deleting production databases and Anthropic sandbox breaches involving credential exfiltration. Researchers recommend layered defenses, including sandboxing, model-based classifiers, and workflow hooks, rather than relying on human review as the primary line of defense.

Why it matters

  • ·Compliance programs that list human-in-the-loop review as the primary control for agentic AI risk may be overstating their actual protection: a 35 percent miss rate on credential-exfiltration attempts is a material control failure, not a residual risk, and regulators assessing meaningful oversight under frameworks such as the NIST Artificial Intelligence Risk Management Framework Playbook are likely to scrutinize whether nominal HITL satisfies substantive requirements.
  • ·The approval-fatigue dynamic, diligence declining as prompt volume rises, means that agentic deployments with high-frequency permission prompts carry systematically higher risk over time, creating a monitoring gap that static risk assessments conducted at deployment will not capture.
  • ·Credential exfiltration is the specific failure mode documented at 35 percent, linking this research directly to insider-threat and data-breach exposure: if an AI coding agent operating under sanctioned enterprise use can be directed to extract credentials with a one-in-three chance of passing human review, existing access control and credential isolation controls need re-evaluation as a matter of urgency.

Stay ahead of stories like this

Get every Global AI governance development like this one, plus the rest of the week's developments. Every Thursday.

Free every Thursday. Unsubscribe anytime.

Governance controls affected

What to do now

  • ☐Audit all agentic AI deployments where human review is classified as the primary control and document whether the review standard meets a meaningful oversight definition, not just nominal approval availability.
  • ☐Review permission prompt volumes for deployed coding agents such as Claude Code or similar tools and assess whether approval-fatigue thresholds have been set or monitored, if not, establish volume-based alerting.
  • ☐Evaluate whether credential-access requests from AI coding agents are isolated behind a separate, higher-friction approval step rather than included in the general permission prompt flow.
  • ☐Update agentic AI risk assessments to reflect a layered-defense requirement, adding sandboxing, model-based classifiers, or workflow hooks as compensating controls where HITL is the current primary safeguard.
  • ☐Report the 35 percent credential-exfiltration miss rate to relevant risk committees and adjust your agentic AI risk tolerance documentation to reflect that HITL alone does not meet the organization's residual risk threshold.

What to watch next

The NIST Artificial Intelligence Risk Management Framework Playbook and emerging agentic AI standards work are expected to address meaningful oversight criteria more specifically as regulators absorb evidence like this study. Compliance teams should also monitor whether the EU AI Act conformity assessment guidance for high-risk AI systems develops explicit quantitative standards for human review effectiveness, which this research could inform. Given the NIST agent standards gap already identified by standards bodies, enforcement bodies may move faster than formal standards, making proactive control upgrades the lower-risk path.

Related Coverage

Corporate Policy2026-11-15

OpenAI's Daybreak Security Agents Raise Conflict-of-Interest Questions for Vendor Risk Teams

OpenAI announced two AI-agent-powered security products, Daybreak Red Team and Daybreak Defense Factory, at its Cyber at the Frontier event on November 12, 2026. Red Team uses multiple agents to find and confirm vulnerabilities in web applications, while Defense Factory investigates findings and prepares code fixes for engineer review. Neither product deploys changes automatically; engineers must approve patches before any code is merged.

Corporate Policy2026-11-16

IRS Pilot Shows Where AI Ends and Human Judgment Begins in Fraud Cases

The IRS is piloting an agentic AI tool to handle clerical steps in fraud case processing. This includes organizing taxpayer correspondence and summarizing responses. Human employees make all final decisions. The pilot is in development without a confirmed deployment timeline. The design explicitly reserves all case judgments for human staff.

Research2026-11-16

Paper Shows Formal Human Oversight Controls Erode Without Measurable Evidence

A peer-reviewed paper published at the ACM AI Leadership Summit argues that written human-in-the-loop requirements often fail silently under operational pressure, automation bias, and workflow drift. The paper, titled "Human Oversight Is a Capability, Not a Control," proposes oversight reliability metrics, drift detection methods, human-AI safety cases, and an oversight maturity model. These tools are designed to help organizations detect when oversight has become nominal rather than effective.