OpenAI said it will honor a common robots.txt signal that lets website operators state, once, how their content may be used to train foundational generative AI models. The company announced the change on October 30, 2026, and described it as a voluntary effort that reduces the need for separate per-provider crawler rules.

One signal for many providers

Today, website operators list individual crawlers in robots.txt and maintain separate rules for each AI provider that visits their site. The new approach uses a shared signal inside robots.txt so a single declaration can communicate the operator's preference to multiple participating providers at once, cutting duplicated configuration work.

OpenAI said it aims to implement the signal over the coming months and will share implementation details in a later update. The company also urged other AI providers to honor the same signal, so a single rule can work across the industry.

What it does not change

The new signal does not replace OpenAI's existing robots.txt crawler controls. Operators can keep using the per-crawler controls OpenAI already offers, and those rules will continue to apply and be respected.

The announcement builds on ongoing work by the Internet Engineering Task Force's (IETF) AI Preferences working group. The shared signal is documented in an IETF draft, and OpenAI is joining other participants in the working group in announcing it.

Context

The change arrives amid an active standards discussion over how websites can control the use of their content for AI training. Because the effort is voluntary and cross-provider, its practical reach depends on how many other providers adopt the same signal.

Sources