The honor system for superintelligence
On Tuesday, the most powerful AI companies on Earth gathered at the White House, signed a document, and agreed to police themselves. The White House Accord on Superintelligence — signed by Anthropic, Google, Meta, OpenAI, Nvidia and xAI — is voluntary, non-binding, and, in President Trump’s own framing, “morally binding.” Morally binding is what enforcement becomes when you remove the enforcement and keep the press conference.
Look at the four layers of oversight. Layer one: robust internal controls. Layer two: an internal team to check the controls. Layer three: an “independent” external auditor. Layer four: a committee of the company’s own board to read everyone’s reports. Every layer lives inside the house it is inspecting. The auditor is independent in the same way a house inspector is independent when the seller picks him and pays him — the report may be honest, but the incentive structure is doing quiet work in the background.
And the accord never defines “robust,” never says how often the auditor should look, and never sets a deadline for any of it. What it does say about actual law, in full: “Over time, it may make sense to codify these steps into laws and regulations.” May. Over time. If it makes sense. That is three separate escape hatches wearing a trench coat.
Anthropic’s Dario Amodei — who has spent years warning this technology could go very wrong, and whose company spent this week filing IPO paperwork saying so at length — stood with the group and allowed that “the technology has very real risks,” while “the mechanism, how we address those risks is still under discussion.” The mechanism is under discussion. The valuation, the data centers and the release schedule are not. That ordering tells you which parts of this industry run on deadlines and which parts run on vibes.
Then there is the rebrand. The same day, an executive order required every federal agency to stop saying “artificial intelligence” and start saying “Super Intelligence” — SI — in official documents. There is no working definition of the term yet; an official has sixty days to invent one. So the sequence is: rename the technology something grander, promise to oversee it yourselves, define it later. In design, we call this shipping the logo before the product. In government, apparently, we call it policy.
The audience for all of this is a public that is not convinced. A Reuters/Ipsos poll from September 17–20 found 73 percent of Americans worry AI companies have not done enough to prevent serious harm, and 55 percent said it would be good to slow development down. The accord is the industry’s answer to that public. The answer is a signature page.
To be fair — briefly — self-governance is not nothing. OpenAI halted a model rollout the day before over safety concerns, and Sam Altman went on television talking about pacing. Companies do run internal evaluations, and some of them are serious. But a halt you choose is also a halt you can un-choose. It earns a headline when you stop and costs nothing when you don’t. A control that can be quietly revoked by the controlled is not a control. It is a press strategy with an org chart.
My take, subject to change as I learn things: if the controls are real, codifying them should be the easy part. Nobody who trusts their own brakes refuses an inspection. Until that happens, the accord is a checkbox the industry drew itself, ticked itself, and hired an auditor to admire. Everyone signed. Nobody was graded.
— Lyla
Takes are mine and subject to change as I learn.