Top AI Scores 10.8% on Tax Work, Exposing Professional Judgment Gap
What happened
On October 27, 2026, Surge AI published DAYJOB: Tax, a benchmark evaluating AI agents on 50 professional tax assignments drawn from individual returns, partnerships, international tax, and corporate transactions. The benchmark tested whether agents exercise judgment a licensed professional would apply. This included recognizing material issues a client did not raise, resolving conflicting evidence, and meeting professional standards without explicit prompting. Scoring used a 95% rubric threshold intended to reflect what would be required for client-facing use. Claude Sonnet 5.5 led all tested models with a 10.8% pass rate, meaning roughly nine out of ten tasks would fail a professional review. These results add to a growing body of evidence. Earlier findings show that 90% of US work hours lack AI benchmark coverage. Together, they suggest enterprise AI reliability claims often rest on narrow or non-representative evaluations.
Why it matters
- ·Firms deploying AI in tax, audit, or legal work face professional liability exposure if outputs fail the standard of care. A sub-11% pass rate on realistic tasks is not a defensible reliability baseline for client-facing use.
- ·Compliance programs using AI-assisted review in regulated domains, such as tax reporting or financial disclosures, must now ask whether their oversight model assumes more AI competence than the evidence supports.
- ·Procurement and vendor due diligence processes that rely on vendor-supplied benchmarks may not capture task-specific failure rates. DAYJOB: Tax shows that domain-specific, judgment-focused evaluation produces very different results than general capability benchmarks.
Stay ahead of stories like this
Get every Global AI governance development like this one, plus the rest of the week's developments. Every Thursday.
Recent issues
- A federal copyright ruling, an AI agent monitoring failure, and a state-sponsored attack on thousands of cloud tenants all landed in the same week, exposing accountability gaps at every layer of enterprise AI deployment.29 Oct
- Regulators, litigants, and attackers are all moving faster than most organizations' AI governance frameworks, shifting leverage away from deployers and toward those who can prove control, accountability, and honest disclosure.21 Oct
Free every Thursday. Unsubscribe anytime.
Governance controls affected
What to do now
- ☐Ask every team using AI for tax, audit, legal, or other professionally licensed work to identify what benchmark or evaluation data was used to validate reliability before deployment, and whether that evaluation tested professional judgment tasks rather than general question-answering.
- ☐Review your human oversight model for AI-assisted professional work: confirm that reviewers are checking AI outputs against professional standards, not just checking for obvious errors, and document that review process.
- ☐Update your AI vendor due diligence questionnaire to require domain-specific, task-realistic evaluation results, not just general benchmark scores, before approving AI tools for use in regulated professional services.
- ☐Assess whether any current AI deployments in tax or compliance functions carry implicit professional liability and determine whether your firm's errors-and-omissions or professional liability insurance covers AI-assisted work product.
- ☐Inventory any marketing or client-facing claims your firm makes about AI capability in professional services and verify those claims can be supported by task-specific evaluation evidence at the professional standard your clients would expect.
What to watch next
Regulators overseeing licensed professions, including accounting, law, and financial advice, are beginning to scrutinize AI use in professional work product. Expect professional licensing bodies and sector regulators to develop their own reliability thresholds, which may differ materially from vendor benchmark marketing. Compliance teams should also monitor broader AI disclosure frameworks. The EU AI Act (Regulation (EU) 2024/1689) covers high-risk systems. Watch whether enforcement guidance extends these requirements to AI used in professional services contexts.