Cloudflare will block AI training and agent crawlers by default on ad pages from 15 September 2026

On 1 July 2026, in an announcement it calls Content Independence Day, Cloudflare said it will change what happens to AI crawlers by default. Cloudflare sits in front of a large share of the web, so its default settings decide a lot of bot traffic on their own. It sorts AI crawlers into three categories: Training, which fetches pages to train models; Agent, which fetches on a person’s behalf when an assistant is acting for them; and Search, which surfaces and cites pages inside an answer. From 15 September 2026, those three categories get new defaults.

What changes on 15 September 2026

On pages that display ads, Cloudflare will block the Training and Agent categories by default, while leaving Search allowed by default. In Cloudflare’s own words, Training and Agent will be blocked by default on the pages that display ads, while Search will remain allowed by default. The new defaults apply to sites onboarding to Cloudflare after that date; existing customers keep control and can set their own per-category preference in their security settings before the change takes effect.

Why the category, not the on-off switch, is the whole story

This is the same distinction our AI-visibility work keeps coming back to, now baked into a default at the network edge. The three categories do different jobs. Search crawlers are the ones that put you in an AI answer and cite you; Training crawlers feed model training; Agent crawlers fetch a page when someone asks their assistant about it. Allowing Search while blocking Training keeps you eligible to be quoted in answers without handing your pages to training. The trap is the blanket rule: block every AI bot and you also remove the Search crawler, which quietly takes you out of the answers you wanted to appear in.

The catch if you want assistants to reach you

The new default leaves Search allowed, so AI-search citations are not the thing at risk. The Agent category is. If you would rather AI assistants could still fetch your page when a user asks about it, and you run ad-supported pages on Cloudflare, the default from 15 September blocks that agent traffic unless you change it. The default is a starting point you can override per category, so the practical step is to decide, before the date, which of Training, Agent and Search you actually want to let in, rather than discovering the answer from a traffic dip.

The bigger shift: paid access instead of a free crawl

The default change sits inside a larger move. Cloudflare argues the old bargain, let a crawler in and it sends readers back, has broken down for AI, because AI crawlers fetch far more than they refer, and by Cloudflare’s figure more than half of crawl traffic re-fetches pages that have not changed. So it is shifting from charging AI companies per crawl toward paying publishers when their content actually appears in an answer. It launched this with two partners: Ceramic.ai, where an opted-in publisher is paid when their content shows up in Ceramic’s search results, and You.com, where an AI agent pays on demand for a specific piece of premium content at the moment it needs it. No pricing was disclosed for either.

Whether or not you sit behind Cloudflare, the lesson is the same: the crawlers you allow decide whether the answer engines can find, parse and quote you. Our free AI search visibility auditor checks whether your page is set up to be reached and cited, and flags the robots.txt rules that lock the retrieval crawlers out. Our guide on how to get cited by ChatGPT and Perplexity covers which crawlers to allow even when you block the training ones, and our piece on why Google’s AI Overviews cut your clicks explains why being the cited source now matters more than ranking a link.