AI Governance Institute
← News
Research2026-10-29

Top AI Scores 10.8% on Tax Work, Exposing Professional Judgment Gap

What happened

On October 27, 2026, Surge AI published DAYJOB: Tax, a benchmark evaluating AI agents on 50 professional tax assignments drawn from individual returns, partnerships, international tax, and corporate transactions. The benchmark tested whether agents exercise judgment a licensed professional would apply. This included recognizing material issues a client did not raise, resolving conflicting evidence, and meeting professional standards without explicit prompting. Scoring used a 95% rubric threshold intended to reflect what would be required for client-facing use. Claude Sonnet 5.5 led all tested models with a 10.8% pass rate, meaning roughly nine out of ten tasks would fail a professional review. These results add to a growing body of evidence. Earlier findings show that 90% of US work hours lack AI benchmark coverage. Together, they suggest enterprise AI reliability claims often rest on narrow or non-representative evaluations.

Why it matters

  • ·Firms deploying AI in tax, audit, or legal work face professional liability exposure if outputs fail the standard of care. A sub-11% pass rate on realistic tasks is not a defensible reliability baseline for client-facing use.
  • ·Compliance programs using AI-assisted review in regulated domains, such as tax reporting or financial disclosures, must now ask whether their oversight model assumes more AI competence than the evidence supports.
  • ·Procurement and vendor due diligence processes that rely on vendor-supplied benchmarks may not capture task-specific failure rates. DAYJOB: Tax shows that domain-specific, judgment-focused evaluation produces very different results than general capability benchmarks.

Stay ahead of stories like this

Get every Global AI governance development like this one, plus the rest of the week's developments. Every Thursday.

Free every Thursday. Unsubscribe anytime.

Governance controls affected

What to do now

  • ☐Ask every team using AI for tax, audit, legal, or other professionally licensed work to identify what benchmark or evaluation data was used to validate reliability before deployment, and whether that evaluation tested professional judgment tasks rather than general question-answering.
  • ☐Review your human oversight model for AI-assisted professional work: confirm that reviewers are checking AI outputs against professional standards, not just checking for obvious errors, and document that review process.
  • ☐Update your AI vendor due diligence questionnaire to require domain-specific, task-realistic evaluation results, not just general benchmark scores, before approving AI tools for use in regulated professional services.
  • ☐Assess whether any current AI deployments in tax or compliance functions carry implicit professional liability and determine whether your firm's errors-and-omissions or professional liability insurance covers AI-assisted work product.
  • ☐Inventory any marketing or client-facing claims your firm makes about AI capability in professional services and verify those claims can be supported by task-specific evaluation evidence at the professional standard your clients would expect.

What to watch next

Regulators overseeing licensed professions, including accounting, law, and financial advice, are beginning to scrutinize AI use in professional work product. Expect professional licensing bodies and sector regulators to develop their own reliability thresholds, which may differ materially from vendor benchmark marketing. Compliance teams should also monitor broader AI disclosure frameworks. The EU AI Act (Regulation (EU) 2024/1689) covers high-risk systems. Watch whether enforcement guidance extends these requirements to AI used in professional services contexts.

Related Coverage

Research2026-11-16

Paper Shows Formal Human Oversight Controls Erode Without Measurable Evidence

A peer-reviewed paper published at the ACM AI Leadership Summit argues that written human-in-the-loop requirements often fail silently under operational pressure, automation bias, and workflow drift. The paper, titled "Human Oversight Is a Capability, Not a Control," proposes oversight reliability metrics, drift detection methods, human-AI safety cases, and an oversight maturity model. These tools are designed to help organizations detect when oversight has become nominal rather than effective.

Corporate Policy2026-11-14

FAA's $875M AI Traffic System Rolls Out Without Controller Union Input

The Federal Aviation Administration has deployed its $875 million SMART air traffic management system to all 22 air route traffic control centers across the US. This expands from an initial pilot covering Washington, DC airspace. The system uses cloud-based AI to combine weather, flight path, traffic flow, and staffing data to recommend routes and arrival sequencing. Human FAA specialists validate recommendations before they reach local controllers, who retain final authority over whether to accept them. The National Air Traffic Controllers Association, representing more than 15,000 members, stated it was not consulted in the design, testing, or implementation of SMART.

Corporate Policy2026-11-10

Anthropic's Fable 5.2 and Mythos 5.2 Create a Two-Tier Access Control Problem

Anthropic released Claude Fable 5.2 and Claude Mythos 5.2 on November 9, 2026, two variants of the same frontier model optimized for advanced technical work. Fable 5.2 is available generally with enhanced cybersecurity and biosecurity restrictions. Mythos 5.2 carries fewer constraints and is restricted to organizations that pass Anthropic's Life Sciences and Cyber Verification Programs. Compliance teams must now account for which variant their organization or vendors can access, and whether internal controls justify that access tier.