Cloudflare AI Crawl Control vs robots.txt: Check Both Layers

If your site sits behind Cloudflare, your robots.txt may have changed in the last month without you touching it. Cloudflare's Bot Preference Sync now writes AI rules into robots.txt from your dashboard settings, and on September 15 the old managed opt-out block disappeared from a lot of files. Meanwhile, Cloudflare AI crawl control can still block or challenge bots at the edge, no matter what robots.txt says.

That gap is the real story. A HasData study of about 10,900 sites found that what sites declare and what their servers do often don't match. If you want AI search and AI agents to reach your pages, you need to check both layers, not just the text file.

TL;DR

Cloudflare now syncs robots.txt from its Search, Agent and Training settings, and the legacy managed block was removed from many files on September 15.
Robots.txt and edge enforcement are separate: HasData found 39.5% of sites that disallowed GPTBot still served it, while others blocked it without saying so.
Test with a real request, not just a file read: compare a bot user agent against a browser from the same machine, then fix the layer that's wrong.

What changed with Cloudflare AI crawl control

Cloudflare has two controls that look similar but behave very differently.

AI Crawl Control (formerly AI Audit) works at the edge. Per crawler you can allow, block or charge (pay per crawl is still a closed beta). Blocking creates or updates a WAF custom rule on your zone and returns a 403, or a 402 Payment Required if you prefer. That's enforcement: the bot gets an error page, not your content. Cloudflare's docs also note that on the free plan crawlers are identified by user agent string only.

Bot Preference Sync, announced on August 21, is about declaration. You pick a policy for Search, Agent and Training, and Cloudflare prepends a block to your robots.txt between # BEGIN Cloudflare Bot Preference Sync and # END Cloudflare Bot Preference Sync markers. It's on by default for new customers. Customers on the legacy managed robots.txt feature are supposed to review and confirm before moving over.

According to HasData's follow-up analysis reported by PPC Land, 118 archived Cloudflare sites carried the managed block (with a Content-Signal line and a GPTBot ban) before September 15. On September 16, none of them did. Bot Preference Sync showed up in none of the 6,451 robots.txt files they collected.

Why robots.txt and edge blocking disagree

Robots.txt is a request. Your firewall is a wall. HasData's numbers show how often the two disagree:

Of 592 sites disallowing GPTBot, 234 (39.5%) still served it a live page.
Sites blocking GPTBot without declaring it rose from 115 in July to 178 in September.
From one datacenter IP, publishers served a Chrome control 83.8% of the time but GPTBot only 54.2%, which suggests enforcement keys on the user agent string.

Cloudflare Radar data cited in a TechnologyChecker.io analysis points the same way: sites on Cloudflare denied 13.47% of AI bot requests in Q3 2026, up from 5.98% a year earlier. Those figures are preliminary, but the direction is clear.

A robots.txt that says "allow" means nothing if a WAF rule quietly returns 403. And a "disallow" protects nothing if the edge serves the page anyway.

The technical breakdown: three layers to line up

Think of AI access as three layers stacked on top of each other.

Layer 1: robots.txt (the declaration)

After sync, a Cloudflare-managed file can look like this, with your own rules below the marked block:

# BEGIN Cloudflare Bot Preference Sync
User-agent: ExampleTrainingBot
Disallow: /
# END Cloudflare Bot Preference Sync

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /admin/

Cloudflare maintains the bot list inside the block. If you need per-bot exceptions with complex logic, its own guidance is to turn sync off and maintain the file by hand.

Layer 2: edge rules (the enforcement)

AI Crawl Control blocks, WAF custom rules, bot fight settings and rate limits all live here. This layer decides the actual HTTP status a crawler receives.

Layer 3: what the page offers once a bot gets in

Even with access, an agent needs clean HTML, structured data and clear headings to use your content.

Setup	robots.txt	Edge response	Result for AI search
Aligned allow	Allow	200	Crawled and citable
Silent block	Allow	403 or challenge	Invisible, and you won't know why
Leaky opt-out	Disallow	200	Your "no" is only a request
Aligned block	Disallow	403	Blocked by choice

What it means for site owners

Most site owners want AI search and user-triggered agents in and training bots out. Cloudflare's Search, Agent and Training categories map to that split. The problem is drift between what you picked and the rules that actually fire.

If you used the legacy managed robots.txt, check whether your training opt-out is still in the file. HasData's data suggests many weren't re-added. Rules you wrote yourself (Patreon is their example) survived.
If you sell online or run a SaaS, a silent block on search or agent crawlers hurts more than a training opt-out ever helped. AI agents that compare products on a user's behalf can't recommend a page they got a 403 from.
Google is its own case. Per HasData's summary, Google-Extended doesn't control AI Overviews or AI Mode, so blocking it won't remove you from those.
Blocking doesn't guarantee you won't be cited, and llms.txt has no enforcement at all. Treat it as a helpful map, not a gate.

How to check and fix your setup

Here's a 20-minute audit you can run today.

Read your live robots.txt. Open https://yoursite.com/robots.txt and look for the Bot Preference Sync markers, Content-Signal lines or rules you didn't write.
Open the Cloudflare dashboard. Review your Search, Agent and Training policies, and confirm the Bot Preference Sync prompt if you were on the legacy feature.
Check AI Crawl Control. Look at the per-crawler actions and the robots.txt violations column. Note anything set to block that you actually want in.
Test the real response. From the same machine, compare a browser user agent with a bot user agent:

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0" https://yoursite.com/pricing
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot/1.0" https://yoursite.com/pricing
curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot/1.1" https://yoursite.com/pricing

A 200 for the browser and a 403 for a search bot is a silent block. Keep in mind your test isn't coming from a verified crawler IP, so treat it as a strong hint rather than proof. 5. Fix the layer that's wrong. Change the WAF or AI Crawl Control action if enforcement is off, or edit robots.txt (or turn sync off) if the declaration is off. 6. Check what the page gives agents once they're in. A free AgentReady scan checks the Browser Readiness signals that matter after access: structured data like JSON-LD and Open Graph, ARIA landmarks and heading structure. 7. Re-test after every Cloudflare settings change, because the dashboard and the file can drift apart again.

Check your own site

Crawler access is the gate; your Agent Readiness Score measures what happens after it. The Browser Readiness pillar (40% of the score) looks at structured data, landmarks and headings, and a low result there usually means agents that do get through can't parse what they find. Our guide to robots.txt for AI crawlers covers user agent names, and the llms.txt Generator gives agents a clean map. A single-page scan is free and takes about a minute. Scan your site free.

Key takeaways

Cloudflare AI crawl control enforces at the edge; Bot Preference Sync only writes robots.txt. They can disagree.
Many legacy managed opt-outs vanished on September 15, so re-check your file if you relied on them.
Silent blocks are rising, and they hide you from AI search without any warning.
Test with real requests using different user agents, then fix the specific layer that's off.
Access is step one. Structured data and clean markup decide whether agents can use what they reach.

FAQ

Does Cloudflare block AI crawlers by default?

Not for everyone. Cloudflare's own post says new customers start with no blocks unless they choose the option saying they monetize pages with ads, in which case Training defaults to disallow. Existing zones keep whatever AI Crawl Control and WAF rules were already set, so check your dashboard rather than assuming.

What is Bot Preference Sync in Cloudflare?

It's a feature that writes your Search, Agent and Training choices into robots.txt automatically. Cloudflare prepends a marked block listing crawlers it maintains, with Disallow: / lines where you've opted out. It's on by default for new customers, and you can switch it off if you need custom per-bot rules.

How do I know if GPTBot can access my site?

Read your robots.txt for GPTBot rules, then request a page with a GPTBot user agent and compare the status code with a normal browser request from the same machine. If the browser gets 200 and the bot gets 403 or a challenge, something at the edge is blocking it, regardless of what robots.txt says.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended governs use of your content for Gemini models, not Search features. Per HasData's summary of Google's controls, AI Overviews and AI Mode are handled through Search itself, so blocking Google-Extended in robots.txt won't take you out of those results.

Open your robots.txt and your Cloudflare dashboard side by side today, and run the three curl commands above against your most important page. Once crawlers can get in, make sure they find something worth citing. Run a free AgentReady scan.

Cloudflare AI Crawl Control: Is It Silently Blocking AI Search?