Rogue AI Must Threaten 50 Lives Before Being Shut Down, Anthropic Suggests

Anthropic is pushing legislation that would only allow the US government to shut down a rogue AI model when at least 50 people are at risk of imminent serious harm or death.

The AI firm has been quietly lobbying congress to adopt its draft bill, a copy of which was obtained by Politico.

Under the bill, a state-led “blocking order” can also be triggered if there is a risk of an AI model causing at least $1bn (£750m) in damage to property. 

The rules laid out in the draft would prevent US states from enforcing tougher safety laws and make it harder for courts to put the brakes on dangerous AI systems.

They set a high threshold for federal intervention in the case of potential “catastrophic harm”. 

Federal officials, for example, would have to ascertain that any threat posed by AI was “imminent” and that no alternative could prevent catastrophe before intervening.

In a Substack post, Zvi Mowshowitz, a writer who focuses on AI, described this as an “absurdly high bar”.

The draft bill reads: “The term ‘catastrophic risk’ means a foreseeable and material risk that a frontier developer’s development, storage, use or deployment of a frontier model will materially contribute to the death of, or serious injury to, more than 50 people, or to more than $1bn in damage to… property arising from a single incident”.

AI safety experts, including Nathan Calvin, Nat Purser and Nate Soares have warned such a bill would weaken existing safeguards while limiting federal oversight.

Mowshowitz said the bill would make people less safe than if legislators did “nothing”. He argued that Anthropic appeared more concerned about the government enforcing safety rules than ensuring it had the power to hold the wider AI industry to account.

Anthony Cimino, head of federal affairs at Anthropic, said on X that the language around risk thresholds in the draft was “pulled directly from existing standards in California law” and would not absolve companies of liability nor prevent the government from using other tools at its disposal to protect the public.

But he acknowledged that it may be “time to re-evaluate thresholds to make sure the national standard is as strong as it can be.”

Last month, former Anthropic employee Jacob Coxon warned that AI could wipe out humanity within a decade. 

The researcher’s comments triggered a series of similar warnings from other AI colleagues. But observers have cast doubt on the claims, as Fortune AI’s editor, Jeremy Kahn, noted, with some viewing them as a “cynical ploy… intended to distract regulators and the public from the real and concrete risks that already exist”.