Catastrophic Risk

Catastrophic risk refers to the possibility that advanced AI systems could cause large-scale, widespread harm, such as mass casualties, major economic disruption, or the loss of human control over critical systems. In AI governance, the term is used to justify stricter oversight of the most capable frontier models, including pre-deployment safety testing, incident reporting, and security requirements. For compliance and risk teams, it signals where regulators and frontier developers are concentrating their expectations, and where internal risk registers may need to address low-probability but high-impact scenarios.

1 item

Corporate PolicyUS2026-10-21

Anthropic's Draft Bill Would Block AI Models Deemed Likely to Kill 50 or More People

Anthropic has circulated a draft federal bill that would allow a court to block an AI model's release. This applies if independent evaluators determine it poses a foreseeable, imminent risk of killing more than 50 people, causing over one billion dollars in property damage, or enabling biological weapons or uncontrolled cyberattacks. Evaluators would register with the U.S. Commerce Department and must notify it within 72 hours of identifying such risks. Only a federal judge, not a government agency acting alone, could halt a release.

Anthropic AI safety legislation catastrophic risk federal regulation third-party evaluation model risk pre-release review evaluator independence