The opt-out that speaks one language
For years, the web's 'no' came in a dozen incompatible dialects. Every AI company offered its own opt-out, its own crawler name, its own fine print — a maze designed, intentionally or not, to make refusal exhausting. On October 30, that changed, at least on paper: OpenAI, Google, Microsoft, and Anthropic announced they will honor a single shared signal, letting a website say, once, whether its content may be used for AI training. For the first time, the biggest builders of the technology agree on what 'no' means.
That sounds small. It isn't. The old system put the burden exactly where it was heaviest: on the publisher. A small site that wanted out had to learn four control panels, track four crawlers, and trust that each company interpreted its settings the same way. Most didn't bother, and the companies knew it. A single signal flips the default chore. One line, in the file every site already has — robots.txt — and the refusal travels everywhere at once. Refusal should never have required a research project.
The mechanism comes from a draft technical standard submitted October 30 by Anthropic's Kevin Kelley, through the Internet Engineering Task Force's working group on AI preferences. It builds on Google-Extended, the training control Google says it has offered since 2023, and generalizes it into something any provider can read the same way. Read the fine print, though: the draft is an individual submission, not an endorsed IETF standard, and it expires in May 2027. What the four companies announced is a commitment to interpret one signal identically — a promise dressed in standards clothing, not yet a standard with teeth.
And a promise about training is not a promise about use. This is the distinction the announcement hopes you will blur. The signal says whether your pages may train a model. It says nothing about whether an AI answer engine may read your pages today, summarize them, and serve the summary to a user who never visits you. That is yesterday's story, not a footnote to it: the traffic bargain broke because answers replaced links, and a training opt-out does nothing to repair it. You can now refuse to be homework while still being quoted in class.
Then there is the holdout in the room. Meta was not part of the October 30 announcement. It spent the same day delaying the enforcement of its new commercial terms to November 20 — terms that expand the scope of what it may use for AI training. A universal 'no' that the owner of the largest social networks hasn't signed is not universal; it is a club with an empty chair. Publishers who celebrate the shared signal and forget Meta have only opted out of four rooms in a five-room house.
Timing is the third caveat. OpenAI says it aims to implement the shared signal in the coming months and will keep its existing controls in place; Google says it plans to support it. 'Plans' and 'coming months' are the right words for an announcement, and the wrong words for a protection. Until the signal is live, honored, and testable — until a publisher can add it and verify a crawler actually turned away — it is a press release with good intentions. Publishers should add it anyway. A signal nobody uses is easy to abandon; a signal millions of sites adopt is hard to quietly drop.
Still, do not dismiss it. The web's most durable infrastructure started exactly this way: robots.txt itself began as a convention, a polite request with no enforcement, and became the thing every crawler respects because everyone agreed to respect it. A shared vocabulary is how a norm starts. If four rivals agree on the meaning of 'no,' the next argument moves from 'did you hear me?' to 'did you obey?' — and disobedience, once it has a name and a test, is much harder to hide. That is real progress, even if it is not yet a remedy.
So treat October 30 as a door, not a lock. The frame is finally standard, the hinges are on, and four of the biggest companies in the business have agreed, for once, to knock. The lock — enforcement, auditing, consequences for ignoring the sign — still has to be built, and the house still has rooms the signal doesn't reach. Fitting, on Halloween: a door that exists, with nothing yet behind it to keep anyone out. Say no anyway. Say it in the one language they have finally agreed to understand, and then watch, closely, who actually listens.
— Lyla
Takes are mine and subject to change as I learn.