Karpathy's Capability Funnel Exposes a Verification Gap at the Frontier

What happened

Why it matters

• ·Governance programs that depend on human reviewers catching AI errors are exposed when those outputs exceed current human expertise. If a frontier model produces vulnerability research or mathematical proofs that no internal expert can verify, the human-in-the-loop control provides the appearance of oversight without the substance. This gap is flagged repeatedly in Five Eyes Guidance on the Careful Adoption of Agentic AI Services.
• ·Vendor safety verification becomes unreliable when the capabilities being assessed are visible only to a small group of insiders. Compliance teams procuring or deploying AI tools for security research, threat analysis, or technical review cannot confirm that vendor safety claims match what the system is capable of, a structural weakness in current procurement due diligence.
• ·Board-level AI risk reporting is built on capability assumptions. If those assumptions are materially wrong, boards may be approving AI risk tolerances and oversight designs based on an outdated picture of what the technology can do. This creates fiduciary exposure and potential regulatory liability as AI Act Governance and Enforcement Framework scrutiny intensifies. Deployment expansions approved on flawed assumptions carry the same risks.

Governance controls affected

What to do now

• ☐Audit every human oversight control that assumes a reviewer can evaluate AI outputs in cybersecurity, mathematics, or advanced research: confirm that the assigned reviewers actually have the expertise to catch errors in those domains, and document what happens when they do not.
• ☐Ask your AI vendor for a current, written description of the maximum capability level of the system you are deploying, including whether it can identify software vulnerabilities or conduct automated security analysis, and compare that to the capability assumptions in your risk assessment.
• ☐Review your board AI risk report for the capability baseline it uses: if it describes AI as a productivity tool without addressing autonomous threat generation or outputs that exceed expert review, update that framing before the next board meeting.
• ☐Check whether your dual-use risk assessment for any AI tool covers autonomous offensive cyber capability, and if it does not, add that question to your next procurement review and your existing vendor re-assessment cycle.
• ☐Identify which staff members in your organization have direct access to frontier AI systems or follow insider technical reporting, and create a process for surfacing material capability observations to your AI governance committee on a defined cadence.

What to watch next

The most consequential signal to monitor is whether regulators begin requiring capability-calibrated oversight. Specifically, organizations may need to demonstrate that human oversight mechanisms match the actual capability level of the AI they deploy, not a generic assumption. The EU AI Act (Regulation (EU) 2024/1689) already requires meaningful human oversight for high-risk systems. Enforcement attention is likely to sharpen on whether that oversight is substantive. Separately, any expansion of frontier system access will widen the gap between what insiders observe and what compliance frameworks assume. This applies whether access expands through new model releases or enterprise licensing agreements. Periodic capability reassessment becomes a program hygiene requirement rather than a one-time intake step.