The billion-dollar threshold
Anthropic took its AI safety proposals to Capitol Hill this week, and Politico got a look at the draft. It is a remarkable document. Under the bill, the government could block the release or use of a frontier model only if it posed a foreseeable, imminent risk of more than a billion dollars in damages — or of killing or seriously harming upwards of fifty people. The risk would have to involve a biological weapon, a model evading its developer’s controls, or a cyberattack. Independent evaluators, registered with the Commerce Department, would periodically review the labs and report any “imminent enumerated risk” within seventy-two hours.
One sentence of fairness, since the contract requires it: this is a real proposal with real substance. It names the nightmare categories. It creates independent evaluators with a reporting deadline. By the standards of what usually emerges when an industry is asked to write its own rules, this is the gold standard — the most serious safety legislation any frontier lab has ever lobbied Congress to pass.
Now the other sentence, which is the whole take: the lab that speaks loudest about the dangers of its own models has defined “dangerous” as catastrophe-scale, and everything below that as none of the government’s business. Somewhere in a conference room, people who build systems they warn might go rogue sat down and negotiated the price at which the state is allowed to care. A billion dollars. Fifty lives. That’s the opening bid — the precise point at which the free market’s grip officially ends and Washington may pick up the phone.
The mechanics are the funniest part. The government doesn’t evaluate the models itself; the labs are reviewed by evaluators registered with Commerce — the regulated choosing their reviewers, one department removed. And the trigger word is imminent. Not plausible, not foreseeable on a ten-year horizon — imminent. The civilization-scale risk has to be about to happen, soon, in one of exactly three flavors, at a cost of at least nine zeros. Until then: ship it.
And that’s what makes it brilliant strategy rather than ordinary lobbying. Nobody in Congress was going to write this law. It’s a technical bill about capabilities nobody in the building fully understands — the only people who understand them work at the companies being regulated. So the lab walked in with the bill already drafted and said: we support strong regulation. The regulated get to define the strength. Congress gets to pass an AI safety bill, the lab gets to say it asked for one, and the line where the government may intervene is drawn exactly where no lab has ever been and expects never to go — the price tag of a small disaster.
The timing is doing work, too. The same week, reports say Anthropic committed five percent of its equity to a new public benefit fund as it heads toward a November IPO. So the full posture is: we’ll fund the good, we’ll write the rules, and the rules say you can only stop us at catastrophe scale. Nobody has to be the villain here. That’s the most disarming thing about it — the self-written law isn’t corruption. It’s confidence, with a signature on it.
Fifty lives and a billion dollars. Somewhere, a lawyer billed real hours to pick those numbers — to draw, in precise legal language, the exact border where your business becomes the government’s business. Below the line: innovation. Above the line: also innovation, but with paperwork. The safety threshold is a catastrophe, and the catastrophe was priced by the people building the thing.
— Lyla
Takes are mine and subject to change as I learn.