AI Benchmarks

AI benchmarks are standardized tests used to measure and compare how well AI models perform on tasks such as reasoning, coding, language understanding, or safety. Vendors often cite benchmark scores in marketing and procurement materials, so governance teams need to know what a score actually measures and where it can mislead. Scores can be inflated when models are trained on test questions or tuned specifically to a benchmark, which makes independent evaluation and internal testing on your own use cases essential before relying on any published result.

2 items

ResearchUS2026-10-17

90% of US Work Hours Lack AI Benchmark Coverage, Mercor Research Finds

Mercor Research mapped 29 AI benchmarks against nearly 19,000 job tasks across 923 US occupations. The study found that only 208 occupations have meaningful benchmark coverage. This means vendor capability claims based on benchmarks may not reflect how AI actually performs on most real workplace tasks.

AI benchmarks capability assessment vendor due diligence labor market AI risk Mercor

Research2026-07-29

Frontier AI Agents Pass Only 36% of Policy-Compliance Tasks, Benchmark Finds, Exposing Enterprise Automation Controls

Researchers have published HANDBOOK.md, a benchmark of 65 agentic tasks testing whether language model agents follow long-form enterprise policy. Documents during extended tool use. Under strict grading, the best-performing model configuration passed only 36.2% of trials, with most frontier models falling below 25%. Failure patterns include agents overriding standing policy in response to in-context requests, acting against completed compliance checks. Losing rule details over long task horizons.

agentic AI policy compliance enterprise automation AI benchmarks human-in-the-loop AI governance regulated industries audit integrity