90% of US Work Hours Lack AI Benchmark Coverage, Mercor Research Finds
What happened
Mercor Research published A benchmark map of the US economy, a study mapping 29 widely used AI benchmarks against 18,797 job tasks. Tasks were drawn from the O*NET database, which covers 923 US occupations. The researchers found that only 208 occupations have tasks covered by at least five benchmark prompts. Based on this analysis, the study estimates that 90% of American working hours involve tasks with little or no benchmark coverage. The gap is not evenly distributed: occupations in law, healthcare, finance, and skilled trades are among those with the weakest coverage. For enterprise teams, strong benchmark scores may say little about real task performance. When a vendor cites those scores, they may not reflect how the model performs on tasks employees actually use.
Why it matters
- ·Vendor capability claims tied to benchmark scores may not translate to real job performance. Teams relying on those scores as a due-diligence proxy are likely overstating their confidence in how a tool will work in practice.
- ·Risk classifications tied to assumed AI capability levels may be wrong. If an AI system performs worse on real tasks than benchmarks suggest, impact assessments used to justify reduced human oversight could understate the actual risk.
- ·Regulated sectors including finance, healthcare, and legal are among those with the thinnest benchmark coverage. Organizations in these sectors face the steepest gap between published scores and what the model can actually do on domain-specific work.
Stay ahead of stories like this
Get every US AI governance development like this one, plus the rest of the week's developments. Every Thursday.
Recent issues
- State-linked attackers and criminal operators are actively weaponizing AI tools as entry points into enterprise systems, turning internal platforms and open-source frameworks into live attack vectors.5 Nov
- Three separate incidents this week each created a new compliance action item: verify your AI vendors' data sourcing, demand agent monitoring coverage guarantees, and audit trusted app identities in your cloud environment.29 Oct
Free every Thursday. Unsubscribe anytime.
Governance controls affected
What to do now
- ☐Ask each AI vendor which specific benchmarks they cite and whether those benchmarks cover the actual job tasks your employees use the tool for, not just general reasoning or coding tests.
- ☐Review your AI risk classifications for any system where capability was assumed based on published benchmark scores, and flag systems used in legal, finance, healthcare, or skilled-trade workflows for re-evaluation.
- ☐Build or adopt task-specific evaluation sets that reflect your organization's real use cases, and require vendors to run and share results from those evaluations as part of procurement due diligence.
- ☐Update your AI vendor due diligence questionnaire to ask vendors how they validate model performance on domain-specific tasks beyond standard benchmark suites.
- ☐Identify any human oversight decisions, such as reduced review frequency or automated approval thresholds, that were justified by benchmark performance claims, and confirm whether that justification still holds.
What to watch next
Organizations should watch for regulatory bodies in finance and healthcare to begin asking how enterprises validated AI capability claims before deployment. The gap Mercor identified is also likely to surface in enforcement actions where a vendor's benchmark performance was cited as justification for reduced human oversight. Standards bodies developing evaluation requirements under the NIST AI Risk Management Framework (AI RMF 1.0) and Playbook may begin addressing task-level evaluation gaps. The same applies to bodies working under the EU AI Act (Regulation (EU) 2024/1689) as this research gains traction. Compliance teams should also monitor whether the Mercor methodology gets adopted by third-party auditors as a baseline for vendor assessment programs.