90% of US Work Hours Lack AI Benchmark Coverage, Mercor Research Finds

What happened

Why it matters

• ·Vendor capability claims tied to benchmark scores may not translate to real job performance. Teams relying on those scores as a due-diligence proxy are likely overstating their confidence in how a tool will work in practice.
• ·Risk classifications tied to assumed AI capability levels may be wrong. If an AI system performs worse on real tasks than benchmarks suggest, impact assessments used to justify reduced human oversight could understate the actual risk.
• ·Regulated sectors including finance, healthcare, and legal are among those with the thinnest benchmark coverage. Organizations in these sectors face the steepest gap between published scores and what the model can actually do on domain-specific work.

Governance controls affected

What to do now

• ☐Ask each AI vendor which specific benchmarks they cite and whether those benchmarks cover the actual job tasks your employees use the tool for, not just general reasoning or coding tests.
• ☐Review your AI risk classifications for any system where capability was assumed based on published benchmark scores, and flag systems used in legal, finance, healthcare, or skilled-trade workflows for re-evaluation.
• ☐Build or adopt task-specific evaluation sets that reflect your organization's real use cases, and require vendors to run and share results from those evaluations as part of procurement due diligence.
• ☐Update your AI vendor due diligence questionnaire to ask vendors how they validate model performance on domain-specific tasks beyond standard benchmark suites.
• ☐Identify any human oversight decisions, such as reduced review frequency or automated approval thresholds, that were justified by benchmark performance claims, and confirm whether that justification still holds.

What to watch next

Organizations should watch for regulatory bodies in finance and healthcare to begin asking how enterprises validated AI capability claims before deployment. The gap Mercor identified is also likely to surface in enforcement actions where a vendor's benchmark performance was cited as justification for reduced human oversight. Standards bodies developing evaluation requirements under the NIST AI Risk Management Framework (AI RMF 1.0) and Playbook may begin addressing task-level evaluation gaps. The same applies to bodies working under the EU AI Act (Regulation (EU) 2024/1689) as this research gains traction. Compliance teams should also monitor whether the Mercor methodology gets adopted by third-party auditors as a baseline for vendor assessment programs.