Configuring Cloudflare AI Crawler Policies: Allow Google and AI Search, Block Model Training

On September 16, 2026, I received an email from Cloudflare with the subject:

Updates to managing AI crawlers on your account

Cloudflare is changing how site administrators control AI crawlers. The previously broad Block AI Bots is being gradually migrated to three more granular policy categories:

• Search
• Agent
• Training

Most notably, Cloudflare introduced the new Disallow AI Training setting on September 15, 2026.

It solves a very practical problem:

If I don’t want my website content used for AI model training, do I have to block normal search engines like Google and Bing at the same time?

The answer is now no.

Cloudflare’s new control method allows a website to remain discoverable by search engines while declaring that its content must not be used for AI Training. Cloudflare also explicitly states that the new Block and Block on pages with ads will override mixed-use crawlers that handle both search and training, such as Applebot, Bingbot, and Googlebot; if the goal is to “allow search indexing but disallow AI training,” you should use the new Disallow AI Training. (Cloudflare Blog)

This article documents my actual configuration process.

1. What changed in Cloudflare’s AI crawler control

Cloudflare now categorizes AI-related automated agents into three types based on their actual behavior:

Search

Used to crawl and index website content for later use in search results or answer results.

Agent

Represents real users accessing the website in real time, such as AI tools like ChatGPT or Claude opening a webpage based on a user request, or a Browser Agent performing web operations.

Training

Crawls website content for model training or fine-tuning.

Cloudflare’s official documentation specifically notes that a single bot can exhibit more than one behavior, so you cannot simply treat all “AI Bots” as having the same purpose. (Cloudflare Docs)

This is why the new control method is more reasonable than the old simple “Block AI Bots” approach.

2. AI Crawl Control can now show actual AI crawler visits

Cloudflare now provides a standalone:

AI Crawl Control

Located at:

Cloudflare → AI Crawl Control → Overview

You can see which AI services are currently accessing the website.

On my site, I could already observe crawlers from various sources, including:

• Microsoft / BingBot
• Google / Googlebot
• Apple / Applebot
• Perplexity
• Anthropic
• OpenAI
• Baidu
• DuckDuckGo
• Mistral
• Meta

among others.

Cloudflare’s AI Crawl Control isn’t just for blocking crawlers; it also serves as a highly intuitive traffic monitoring tool for AI crawlers.

The official documentation states that it can be used to view crawler activity, control individual AI crawlers, check robots.txt compliance, and more. (Cloudflare Docs)

3. The original settings allowed Search, Agent, and Training

Navigate to:

Security → Settings → Configure AI Bot Policy

You will see three independent options.

The initial state when I checked was:

Search: Allow (Not blocked)
Agent: Allow (Not blocked)
Training: Allow (Not blocked)

Search: Allow (Not blocked)
Agent: Allow (Not blocked)
Training: Allow (Not blocked)

Additionally:

Enable Bot Preference Sync: On

Enable Bot Preference Sync: On

This meant that all three uses—search, AI Agent, and model training—were currently allowed.

From the perspective of maximizing website content exposure, there is nothing wrong with this.

However, what I wanted to achieve was:

Search can index normally; AI search and AI Agents can discover and read the website; but there is no need to allow pure model training crawlers to scrape content at scale.

Therefore, I decided to only modify Training.

4. Training should not simply be set to “Block”

After opening the Training dropdown menu, you can currently see several different options:

Block
Block on ad pages
Disallow
Allow (Not blocked)

Block
Block on ad pages
Disallow
Allow (Not blocked)

This is very easy to misunderstand.

If the goal is simply to “disallow AI Training,” do not just select:

Block

Cloudflare changed this behavior starting September 15, 2026.

For mixed-use crawlers that handle both Search and Training functions, such as:

• Googlebot
• Bingbot
• Applebot

Selecting Block or Block on pages with ads might block their search access as well.

Cloudflare’s official documentation is very clear on this: if you still want the website to appear in search results while disallowing AI Training, you should use Disallow AI Training. (Cloudflare Blog)

5. My final configuration: Search Allow + Agent Allow + Training Disallow

Ultimately, I adopted:

Search: Allow (Not blocked)
Agent: Allow (Not blocked)
Training: Disallow
Bot Preference Sync: On

Search: Allow (Not blocked)
Agent: Allow (Not blocked)
Training: Disallow
Bot Preference Sync: On

Why this setup?

Search remains Allow

There is no controversy here.

My website still needs normal search portals like Google, Bing, and Apple, so there is no need to sacrifice traditional search indexing just to restrict AI Training.

Agent also remains Allow

This was a choice I made based on my own site’s situation.

Agent refers to AI automated agents that access the website in real time on behalf of users.

For example, if someone asks an AI a question, and the AI opens my webpage in real time to answer it.

For a content website, this type of access can potentially bring:

• AI search exposure;
• Source citations;
• Page links;
• Potential real user visits.

Therefore, at this stage, I do not plan to block AI Agents.

Training uses Disallow

This one is different.

Pure training crawlers scraping pages do not directly bring search visibility or user visits to the website.

Therefore, I chose:

Disallow AI Training

The Cloudflare API also explicitly distinguishes the several modes for Training as:

disabled
disallow
block
only_on_ad_pages

disabled
disallow
block
only_on_ad_pages

This shows that disallow is itself a configuration state independent of the technical block. (Cloudflare Docs)

6. Bot Preference Sync remains enabled

My Enable Bot Preference Sync was already in an enabled state, so I kept it that way.

It generates the corresponding crawler preference based on the current configuration and adds it to the Cloudflare-provided robots.txt content.

It is important to note that:

robots.txt itself only expresses the website owner’s preference and cannot enforce compliance from all crawlers at the network layer.

Cloudflare’s official documentation also explicitly states that robots.txt compliance is voluntary; if you truly need to forcibly block crawlers, you must use technical controls like AI Crawl Control. (Cloudflare Docs)

So:

Bot Preference Sync and AI Crawl Control are not exactly the same thing.

One is mostly for expressing policy, while the other can actually enforce access control.

7. After the changes, reviewing AI Crawl Control is quite interesting

After completing the setup, I went back into:

AI Crawl Control → Security

You can clearly see that the crawlers are divided into two groups.

Some search, search-augmentation, and real-time user access crawlers remain unblocked, such as:

BingBot
Googlebot
Applebot
PerplexityBot
Claude-SearchBot
OAI-SearchBot
ChatGPT-User
DuckAssistBot
MistralAI-User
Perplexity-User

BingBot
Googlebot
Applebot
PerplexityBot
Claude-SearchBot
OAI-SearchBot
ChatGPT-User
DuckAssistBot
MistralAI-User
Perplexity-User

Meanwhile, some AI Crawlers are now shown as blocked, such as:

ClaudeBot
GPTBot
Bytespider
Amazonbot
CCBot

ClaudeBot
GPTBot
Bytespider
Amazonbot
CCBot

This essentially achieved the result I wanted:

Search portals continue to exist.

Real-time user access portals like ChatGPT continue to exist.

AI search crawlers continue to exist.

Pure Training crawlers are restricted.

Compared to the old approach where “AI Bots were either all allowed or all blocked,” this is clearly much more reasonable.

8. An interesting exception: Claude-User

During my review, I also noticed a rather interesting phenomenon.

Claude-User is displayed in the current Cloudflare Dashboard as:

Claude-User
Anthropic
AI Crawler

Claude-User
Anthropic
AI Crawler

And it is in a blocked state.

I originally wanted to directly change this single crawler to allow, but Cloudflare does not permit direct modification.

When hovering over the corresponding toggle, a tooltip appears:

This crawler is being blocked by an existing AI bot policy.
Configure AI bot policies.

This means that this status stems from the global AI bot policy configured earlier, not from an individually set crawler rule in AI Crawl Control.

Interestingly, Cloudflare’s current documentation emphasizes “behavior” in its AI bot classification and explicitly states that a single bot can have multiple behaviors. (Cloudflare Docs)

Therefore, I did not change the entire Training setting back to Allow just to individually permit Claude-User.

Doing so would simultaneously re-allow many crawlers that are genuinely used for model training, which is clearly a higher price to pay.

Moreover, currently:

Claude-SearchBot

Claude-SearchBot

is still allowed normally.

Thus, Claude’s search discovery portal has not disappeared along with it.

Considering that this new AI crawler policy adjustment started on September 15, 2026, and the Dashboard and policies are still in the rollout phase, this detail will need continued observation going forward.

9. AI Crawl Control no longer requires manual configuration one by one

After completing this round of settings, I did not continue creating individual rules for dozens of crawlers in AI Crawl Control.

The reason is that Cloudflare’s global AI Bot Policy itself continuously updates the corresponding bot classifications and handling methods.

The official documentation also recommends prioritizing built-in bot settings for standard scenarios; compared to manual WAF rules, these managed settings will update as Cloudflare identifies new crawlers. (Cloudflare Docs)

So my current approach is:

Global Policy:

Search   → Allow
Agent    → Allow
Training → Disallow

Global Policy:

Search   → Allow
Agent    → Allow
Training → Disallow

Then:

AI Crawl Control → primarily for observation

AI Crawl Control → primarily for observation

Only if I later find that a specific crawler is clearly causing issues will I consider blocking it individually.

This makes configuration simpler and avoids unexpected conflicts from accumulating too many rules.

10. The problem genuinely solved by this Cloudflare update

I think the most valuable aspect of this Cloudflare change isn’t simply that “a few more AI Bot toggles were added.”

What it truly solves is:

Site administrators can finally separate search discovery, AI Agents, and AI model training.

In the past, simply blocking AI Bots easily led to a dilemma:

Not blocked:
Content might be used for AI Training.

Blocked:
Discovery capabilities for Google, Bing, or AI Search might be lost.

Not blocked:
Content might be used for AI Training.

Blocked:
Discovery capabilities for Google, Bing, or AI Search might be lost.

Now you can configure more granularly:

Search: Allow
AI Agent: Allow
Model Training: Disallow

Search: Allow
AI Agent: Allow
Model Training: Disallow

For websites that rely on public content, SEO, and search discovery, I believe this control method is clearly more practical.

At least for my own site, my current choice is:

Keep content as discoverable as possible, but there is no need to unconditionally allow content to be used for model training.

This will likely serve as my default policy when observing AI crawler traffic for the foreseeable future.

References

On September 15, 2026, Cloudflare officially released an explanation regarding mixed-use AI crawlers and Disallow AI Training, explicitly introducing the three control types—Search, Agent, and Training—as well as the migration path for old settings. (Cloudflare Blog)

The official Cloudflare AI Crawl Control documentation covers crawler monitoring, individual Allow/Block controls, robots.txt compliance, and its relationship with WAF/Bot Management. (Cloudflare Docs)