AI 基礎設施與網路標準
Cloudflare Separates Search, Training, and Agent Crawler Controls, but Bing’s Training Opt-Out Signal Is Not Yet Automatic
Cloudflare has introduced “Disallow AI Training,” combining `robots.txt` preferences with network-layer classification so websites can remain indexed by search engines while opting out of model training. Existing settings will be migrated automatically, but support varies across Applebot, Googlebot, and Bingbot—and selecting Block by mistake could still directly sacrifice search visibility.
On September 15, Cloudflare revamped AI Crawl Control, explicitly separating crawler activity into Search, Training, and Agent categories and adding “Disallow AI Training.” When enabled, Bot Preference Sync publishes vendor-specific training opt-out rules in `robots.txt`. For training crawlers without a trustworthy mechanism for distinguishing their purposes, Cloudflare can block them at the network layer instead of merely relying on voluntary compliance. Apple, Google, and Microsoft are classified as “Accountable” operators of dual-purpose crawlers. Qualification criteria include opt-outs for training and AI summaries, URL-level transparency, and assurance that opting out of training will not affect traditional search.
The key technical distinction is that preference signals and access controls are not the same thing. Google’s `Google-Extended` is only a control token; actual crawling still uses existing Google user agents. Google’s documentation confirms that disallowing it does not affect search indexing or ranking, but restricts content from being used for future Gemini training and certain grounding use cases. Apple separately provides `Applebot-Extended`. Bing currently relies primarily on `NOARCHIVE` and Bing Webmaster Tools. Cloudflare says it does not expect Bing to support site-level training opt-outs through `robots.txt` until early 2027, so enabling the new setting alone cannot yet automatically communicate the same preference to Bing.
The change also introduces migration risks. The previous “Block AI Bots” option will be retired, and existing Training rules set to Block will be converted to Disallow AI Training. The new Block option, however, will also block dual-purpose search crawlers. Operations teams should recheck the generated `robots.txt`, WAF events, search indexing, and crawler identity classification. The system still provides neither a legal nor cryptographic guarantee: crawlers that spoof their identities or ignore robots rules, as well as user-directed browsing agents, remain subject to no unified policy. IETF standards for expressing AI preferences are also still taking shape.