AI Governance Institute
← News
Research2026-11-21

METR Finds AI Excels at Coding but Falls Short on Judgment Tasks

What happened

METR's updated time horizon report expands its benchmark suite to include harder, more open-ended tasks alongside the verifiable ones in the original plot. Task difficulty is measured by how long a human takes to complete each task; models are then rated by the length of tasks they can successfully finish. On structured tasks like coding and math, where it is possible to check automatically whether an answer is correct, frontier models continue improving on an exponential curve. On judgment-laden tasks, results differ sharply: not one of the AI-designed evaluations was considered useful by METR employees, and models fell short of human performance on scoring philosophical arguments. The most advanced models, including Opus 5.5, reached what METR calls "borderline above the bar" on evaluation design. This marks a clear improvement from one year ago, when every model scored below that threshold. Still, no model reached a level METR employees found usable. METR notes that most completion time estimates for harder tasks are based on extrapolation rather than direct measurement. Paying qualified humans to attempt tasks requiring hundreds of hours is prohibitively expensive.

Why it matters

  • ·Human oversight classifications built on general AI benchmark scores may misassign risk levels. Because AI performs very differently on verifiable tasks versus judgment-based ones, compliance teams should map oversight requirements to task type rather than to a single capability tier.
  • ·The referenced separate METR report claiming AI may be doubling the pace of AI development at Anthropic's IPO-stage company raises recursive self-improvement concerns. This updated report confirms the judgment gap separating current models from fully autonomous AI research and development workflows.
  • ·Capability thresholds are shifting even on hard-to-verify tasks. The improvement from "clearly below the bar" to "borderline above the bar" in one year means static human oversight policies will become outdated quickly. Periodic reassessment is required rather than a one-time classification.

Stay ahead of stories like this

Get every Global AI governance development like this one, plus the rest of the week's developments. Every Thursday.

Free every Thursday. Unsubscribe anytime.

Governance controls affected

What to do now

  • ☐Review every AI use case in your inventory and tag each one as either 'verifiable output' (such as code generation or data extraction, where correctness can be checked automatically) or 'judgment-based output' (such as research direction, strategic analysis, or compliance interpretation), and confirm that judgment-based use cases have mandatory human review at the output stage.
  • ☐Ask the teams using AI for research, compliance analysis, or strategic planning to identify which specific tasks they currently allow AI to complete without human review, and compare those tasks against METR's finding that AI still falls short of human performance on designing evaluations and exercising research judgment.
  • ☐Update your AI risk classification criteria to require a periodic review, at least annually, of which use cases remain in lower-oversight tiers, since METR's data shows capability on judgment tasks is improving and a use case safely classified today may cross a governance threshold within twelve months.
  • ☐Brief the board or AI governance committee on the finding that benchmark progress on coding does not translate directly to progress on judgment tasks, and confirm that any internal narrative framing AI as capable of autonomous research is qualified against METR's specific evidence.
  • ☐If your organization uses AI in any AI research and development workflow, assess whether your human oversight policy accounts for the possibility that the AI is contributing to its own capability improvements, given the separate METR finding referenced in the report about AI potentially doubling development pace.

What to watch next

METR has indicated that predicting progress on judgment-laden tasks will remain difficult by their nature. Compliance teams should monitor future METR updates for movement in the 'borderline above the bar' category. Such movement would signal that autonomous AI research workflows are closer to human-level capability. Separately, federal and state regulators considering human oversight requirements may reference capability benchmarks when setting thresholds for mandatory human review. This includes those developing rules under the EU AI Act (Regulation (EU) 2024/1689) and emerging US frameworks. Organizations that have made public commitments about AI oversight should track this research as evidence that those commitments may need updating as the judgment-task frontier moves.

Related Coverage

Corporate Policy2026-11-14

FAA's $875M AI Traffic System Rolls Out Without Controller Union Input

The Federal Aviation Administration has deployed its $875 million SMART air traffic management system to all 22 air route traffic control centers across the US. This expands from an initial pilot covering Washington, DC airspace. The system uses cloud-based AI to combine weather, flight path, traffic flow, and staffing data to recommend routes and arrival sequencing. Human FAA specialists validate recommendations before they reach local controllers, who retain final authority over whether to accept them. The National Air Traffic Controllers Association, representing more than 15,000 members, stated it was not consulted in the design, testing, or implementation of SMART.

Research2026-11-12

Vendor Custody, Delegated Authority, and the Agentic AI Accountability Gap

A peer-reviewed paper published in November 2026 by Springer Nature examines how accountability is distributed when AI agents make decisions on behalf of users and organizations. The research identifies four structural risks: delegated authority without clear ownership; vendor-controlled alignment measures that deployers cannot audit; limited inspection rights over agent behavior; and gaps in incident response responsibility. The findings have direct implications for vendor risk programs, human oversight requirements, and AI governance documentation.

Research2026-11-20

ECB Links AI Governance Gaps and Cybersecurity Failures to Bank Financial Stability Risk

The European Central Bank published a financial stability analysis titled "Decoding the cyber-AI nexus" in November 2026. It identifies weak AI governance, model alignment failures, and inadequate cybersecurity investment as internal risks facing banks deploying AI, including frontier models. The analysis signals that supervisors are beginning to treat AI governance gaps as financial stability concerns, not merely operational ones.