Microsoft's run-assert-eval Cuts Agent Violations From 30% to 5.9%, With Audit Proof
What happened
Microsoft published Introducing run-assert-eval: Find the risk, fix it, prove it on September 24, 2026. The announcement describes a tool that integrates three previously separate responsible AI components into one automated workflow. The tool first maps the ways an AI agent could cause harm (threat modeling). It then measures how often the agent actually violates defined policies (evaluation). Finally, it generates machine-readable enforcement rules that govern the agent at runtime. In a billing-support agent test case, the baseline violation rate for cross-account data disclosure was 30%; after the tool generated and applied its enforcement rules, that rate fell to 5.9%. The tool runs against the same frozen set of test cases before and after remediation. This produces auditable evidence that a specific control change caused a specific reduction in violations. This approach follows a broader industry shift documented in Agent Governance Shifts From Logging to Pre-Action Authorization Evidence. That shift sees regulators and auditors moving beyond asking whether policies exist toward asking whether they demonstrably work.
Why it matters
- ·Regulators and auditors increasingly expect quantified evidence that AI controls work, not just documentation that they exist. A tool that generates before-and-after violation rates against a frozen test set provides traceable evidence. That is exactly what frameworks like ISO/IEC 42001:2023 - Artificial Intelligence Management System and emerging agent-specific guidance expect compliance teams to produce.
- ·The tool's runtime enforcement layer converts governance policy into machine-readable rules the agent follows during live operation. This directly addresses the gap that Frontier Agents Fail Policy Tests at Scale, Exposing a Pre-Deployment Gate Gap documented: prose-based AI policies frequently do not survive contact with actual agent behavior. Organizations that rely only on written policies without tested runtime enforcement now have a clear benchmark to close against.
- ·Ownership of this tooling class is unresolved in most enterprise governance programs. The workflow spans engineering, responsible AI, and compliance functions. Without a named owner and a defined process for incorporating test results into audit packages and model change records, the evidence the tool generates may not reach the people who need it, creating a documentation gap even when the underlying control is working.
Governance controls affected
What to do now
- ☐Ask your AI engineering or platform team whether any AI agents currently in production have been tested to measure how often they violate their own stated policies, and request the results in writing.
- ☐Identify which team owns the gap between a written AI governance policy and a tested, enforced runtime rule: if no team owns it, escalate that accountability gap to the AI governance committee.
- ☐Review your model deployment approval process to confirm it requires a quantified violation-rate result, not just a sign-off that testing occurred, before an agent goes live.
- ☐For any agent handling sensitive customer data, ask what the acceptable violation rate threshold is and who approved that number, then compare it to actual measured rates if they exist.
- ☐Confirm that evidence from pre-deployment agent testing (including before-and-after violation rates) is captured in your AI model registry or audit package, not left only in engineering repositories.
What to watch next
Run-assert-eval writes governance rules machines can enforce and measures their effect before deployment. This pattern is converging with regulatory expectations in the EU, the US, and Singapore. Watch for EU AI Act Implementation Timeline: High-Risk Deadline Deferred to 2 December 2027 conformity assessment guidance. It may specify whether runtime enforcement evidence satisfies technical documentation requirements for high-risk agent systems. The NIST Artificial Intelligence Risk Management Framework Playbook is also likely to be updated with agent-specific measurement guidance as the field matures. Compliance teams should also monitor financial regulators. Building on SR 26-2 Forces Banks to Rethink Model Governance From Inventory to Board Oversight, those regulators may begin requiring quantified control-effectiveness evidence for AI agents in model risk management submissions.
Stay ahead of stories like this
Get every Global AI governance development like this one, plus the rest of the week's developments. Every Thursday.