Developing
The change applies to new sites and existing free-plan sites that had not set a preference.
In 60 seconds
- Cloudflare changed its AI-crawler defaults on September 15, 2026, replacing one switch with three categories: Search, Training and Agent.
- Training and Agent crawlers are blocked by default on pages that display ads; Search bots remain allowed.
- The defaults apply to new customers, new sites and existing free-plan customers with unchanged settings; paid customers with set preferences are not changed.
- Some coverage says multi-purpose crawlers such as Googlebot may be affected if Training is blocked; check this on your own site.
Impact Layer
- What changed
- A large share of the web now blocks AI training and agent fetches by default unless owners choose otherwise.
- Why it matters
- If agents are blocked, your pages may not be fetched when someone asks an assistant about your brand.
- Who is affected
- Site owners on Cloudflare, particularly free plans, and teams that want visibility in AI answers.
- Martech stack area
- AI, SEO & Search
- Stack impact
- Record your preference in Cloudflare's Security settings and test crawlers directly by status code instead of trusting dashboard counts.
- Business impact
- High We read one detailed secondary report; Cloudflare's own announcement was not found in the results. Claims about Googlebot come from other coverage and should be verified.
- Related technologies
- What to watch
- Cloudflare's own documentation, any change to how multi-purpose crawlers are classified, and the pay-per-use pilots.
On September 15, 2026, Cloudflare changed its default handling of AI crawlers, according to a detailed report by Remote Work Europe.
What changed
- One “block AI bots” switch is replaced by three categories: Search, Training and Agent.
- Training and Agent crawlers are blocked by default on pages that display ads. Search bots remain allowed.
- The defaults affect new customers, new sites added by existing customers and existing free-plan customers whose settings are unchanged. Paid customers who previously configured their preferences are not automatically changed.
The report stresses that “AI crawlers” are several things. OpenAI runs GPTBot for training data, OAI-SearchBot for ChatGPT search indexing and ChatGPT-User for live page fetching on a person’s behalf. Only the last produces visible referrals in analytics.
What site owners should do
- Record your preference in Security settings rather than relying on the default.
- Test crawlers directly using status codes. The report found that Cloudflare’s “unsuccessful requests” metric combines 429, 403 and 404 responses, which can misstate how much is blocked.
- Remember that blocking “AI bots” as one group can cut off referral sources.
Cloudflare is also piloting a “Pay Per Use” model with Ceramic.ai and You.com that pays by actual content use rather than crawl frequency.
Our read
Blocking training and answering are different decisions, and a default that blocks agent fetches on ad pages will cut some sites out of AI answers without anyone choosing that. Decide on purpose: allow search and user-triggered fetching if you want to be cited, and block training if you object to it. Other coverage claims multi-purpose crawlers like Googlebot may be caught by a Training block; we could not confirm that from Cloudflare, so test it on your own site.
Sources
- Remote Work Europe: Cloudflare changes its AI-crawler defaults on 15 September (opens in a new tab)
Source of the categories, who is affected, the checklist, the dashboard issue and the pay-per-use pilots.