METR Finds AI Excels at Coding but Falls Short on Judgment Tasks
What happened
METR's updated time horizon report expands its benchmark suite to include harder, more open-ended tasks alongside the verifiable ones in the original plot. Task difficulty is measured by how long a human takes to complete each task; models are then rated by the length of tasks they can successfully finish. On structured tasks like coding and math, where it is possible to check automatically whether an answer is correct, frontier models continue improving on an exponential curve. On judgment-laden tasks, results differ sharply: not one of the AI-designed evaluations was considered useful by METR employees, and models fell short of human performance on scoring philosophical arguments. The most advanced models, including Opus 5.5, reached what METR calls "borderline above the bar" on evaluation design. This marks a clear improvement from one year ago, when every model scored below that threshold. Still, no model reached a level METR employees found usable. METR notes that most completion time estimates for harder tasks are based on extrapolation rather than direct measurement. Paying qualified humans to attempt tasks requiring hundreds of hours is prohibitively expensive.
Why it matters
- ·Human oversight classifications built on general AI benchmark scores may misassign risk levels. Because AI performs very differently on verifiable tasks versus judgment-based ones, compliance teams should map oversight requirements to task type rather than to a single capability tier.
- ·The referenced separate METR report claiming AI may be doubling the pace of AI development at Anthropic's IPO-stage company raises recursive self-improvement concerns. This updated report confirms the judgment gap separating current models from fully autonomous AI research and development workflows.
- ·Capability thresholds are shifting even on hard-to-verify tasks. The improvement from "clearly below the bar" to "borderline above the bar" in one year means static human oversight policies will become outdated quickly. Periodic reassessment is required rather than a one-time classification.
Stay ahead of stories like this
Get every Global AI governance development like this one, plus the rest of the week's developments. Every Thursday.
Recent issues
- Regulators, courts, and markets are all testing whether AI companies' accountability structures hold up under real pressure, not just in policy documents.19 Nov
- The next quarter will test whether compliance programs can keep pace with AI systems that breach containment, reshape capital markets, and enable fraud at a scale no prior control framework anticipated.12 Nov
Free every Thursday. Unsubscribe anytime.
Governance controls affected
What to do now
- ☐Review every AI use case in your inventory and tag each one as either 'verifiable output' (such as code generation or data extraction, where correctness can be checked automatically) or 'judgment-based output' (such as research direction, strategic analysis, or compliance interpretation), and confirm that judgment-based use cases have mandatory human review at the output stage.
- ☐Ask the teams using AI for research, compliance analysis, or strategic planning to identify which specific tasks they currently allow AI to complete without human review, and compare those tasks against METR's finding that AI still falls short of human performance on designing evaluations and exercising research judgment.
- ☐Update your AI risk classification criteria to require a periodic review, at least annually, of which use cases remain in lower-oversight tiers, since METR's data shows capability on judgment tasks is improving and a use case safely classified today may cross a governance threshold within twelve months.
- ☐Brief the board or AI governance committee on the finding that benchmark progress on coding does not translate directly to progress on judgment tasks, and confirm that any internal narrative framing AI as capable of autonomous research is qualified against METR's specific evidence.
- ☐If your organization uses AI in any AI research and development workflow, assess whether your human oversight policy accounts for the possibility that the AI is contributing to its own capability improvements, given the separate METR finding referenced in the report about AI potentially doubling development pace.
What to watch next
METR has indicated that predicting progress on judgment-laden tasks will remain difficult by their nature. Compliance teams should monitor future METR updates for movement in the 'borderline above the bar' category. Such movement would signal that autonomous AI research workflows are closer to human-level capability. Separately, federal and state regulators considering human oversight requirements may reference capability benchmarks when setting thresholds for mandatory human review. This includes those developing rules under the EU AI Act (Regulation (EU) 2024/1689) and emerging US frameworks. Organizations that have made public commitments about AI oversight should track this research as evidence that those commitments may need updating as the judgment-task frontier moves.
