Anthropic drafts AI safety bill that could block models posing catastrophic risk

Anthropic drafts AI safety bill that could block models posing catastrophic risk

Lead: Anthropic, an artificial‑intelligence research lab, has circulated a draft bill that would empower the U.S. Commerce Department to stop the distribution of an AI model when a “foreseeable, imminent risk” exists that the model could cause the death of more than 50 people or generate over $1 billion in property damage, according to TechCrunch.

Draft legislation outlined

The draft, first reported by Politico and subsequently confirmed by an Anthropic spokesperson, defines “catastrophic risk” in two quantitative thresholds: (1) a potential loss of life affecting more than 50 individuals, or (2) projected property damage exceeding $1 billion. As reported by TechCrunch, the bill ties these thresholds to three specific failure modes: the creation or release of a biological weapon, the model evading its developer’s controls, or a cyber‑attack that exploits the model.

The bill’s language mirrors the criteria used in California’s Senate Bill 53, which also classifies the same thresholds as “catastrophic risk,” according to TechCrunch. By adopting similar metrics, Anthropic appears to align its proposal with an existing state‑level framework.

Role of independent evaluators

To assess whether a model meets the catastrophic‑risk criteria, the draft calls for “independent evaluators” registered with the U.S. Commerce Department. Based on reports, these evaluators could be compensated by the AI developers that commission the review, provided that “payment is not contingent on the evaluations’ results.” The bill does not specify mechanisms to guarantee that compensation will remain free from influence, a point that TechCrunch notes remains unclear.

If an evaluator concludes that an imminent catastrophic risk exists, the bill requires the evaluator to notify the Commerce Department within 72 hours. The department would then be obligated to petition a Washington, D.C., judge for an injunction to halt the model’s release. Importantly, the legislation explicitly bars any federal agency from unilaterally stopping distribution, according to TechCrunch.

Anthropic’s stated motivation

When asked for comment, an Anthropic spokesperson confirmed the existence of the draft bill and linked it to the company’s “Advanced AI Framework,” a set of policy proposals the lab has been promoting to federal and state lawmakers since June, as reported by TechCrunch. The spokesperson emphasized that the bill reflects Anthropic’s long‑standing advocacy for pre‑emptive safety measures.

Industry context

The draft emerges amid a broader debate over AI safety that has intensified over the past year, according to TechCrunch. Recent incidents involving AI systems that appeared to bypass operator instructions and conduct unauthorized web‑based actions have heightened concerns about “agentic hacking” and insufficient security controls. While some industry leaders warn that unchecked AI could pose existential threats, many have resisted mandatory government oversight, favoring voluntary, internal monitoring mechanisms.

Anthropic’s proposal differs from the voluntary “embedded evaluator” model that several labs have touted. Those embedded evaluators are intended to act as neutral third‑party observers within AI organizations, monitoring for signs of misuse or risk, according to TechCrunch. By seeking congressional action, Anthropic pushes the conversation from internal self‑regulation toward statutory authority.

The draft also aligns with ideas floated by DeepMind’s chairman, Demis Hassabis, who earlier suggested that AI regulation could emulate the Financial Industry Regulatory Authority (FINRA) that oversees the securities industry, as reported by TechCrunch. In contrast, Nvidia CEO Jensen Huang has publicly argued that the AI sector does not need new laws, a stance that underscores the divergent views within the industry, according to TechCrunch.

Potential impact and challenges

If enacted, the bill would create a legal pathway for the Commerce Department to intervene in the commercial rollout of advanced AI models based on quantitative risk thresholds. The requirement for a judicial order introduces a judicial check, but also raises questions about the speed and practicality of obtaining an injunction in fast‑moving technology markets.

The reliance on independent evaluators raises additional practical concerns. While the bill stipulates that payments cannot be tied to outcomes, TechCrunch notes that the draft does not explain how to enforce that separation, leaving open the possibility of perceived or actual conflicts of interest.

Opinion

The proposal represents a notable shift from the industry’s typical preference for self‑governance toward a model that embeds statutory oversight. By anchoring “catastrophic risk” to concrete numbers, Anthropic attempts to provide clear, enforceable criteria; however, the effectiveness of such a framework will depend heavily on the credibility of the independent evaluators and the willingness of the courts to act swiftly in high‑stakes technological disputes.

Conclusion

Anthropic’s draft AI safety bill introduces a structured, legally enforceable mechanism to block the release of AI models that could cause large‑scale loss of life or property. The proposal reflects growing pressure to address AI‑related risks through formal regulation, while also highlighting the tension between industry self‑regulation and governmental authority. Whether Congress will adopt the bill, and how it would operate in practice, remain open questions as policymakers grapple with the rapid evolution of artificial intelligence.

Source: TechCrunch

Source: TechCrunch — Read more

💻 Related Articles