robots.txt for AI Crawlers: How to Allow or Block GPTBot, ClaudeBot and PerplexityBot

Your robots.txt is the first thing an AI crawler reads, and it decides whether your pages can appear in AI answers at all. The complication is that AI companies run several user agents each, for different purposes. Blocking the wrong one can make you invisible; allowing the wrong one can feed a training set you did not intend. This guide sorts them out.

The three jobs AI user agents do

1. Training crawlers collect content to train or improve models.
2. Search crawlers index content so an AI search product can retrieve and cite it.
3. User agents fetch a page on demand because a person asked an assistant to look at it.

These have separate tokens, so you can treat them differently.

Common AI user agents

Operator	User agent	Purpose
OpenAI	GPTBot	Training data collection
OpenAI	OAI-SearchBot	Search indexing for ChatGPT search
OpenAI	ChatGPT-User	User-initiated page fetches
Anthropic	ClaudeBot	Training data collection
Anthropic	Claude-SearchBot	Search quality and retrieval
Anthropic	Claude-User	User-initiated page fetches
Perplexity	PerplexityBot	Search indexing
Perplexity	Perplexity-User	User-initiated page fetches
Google	Google-Extended	Control token for use in Gemini models and grounding (not a separate crawler)
Apple	Applebot-Extended	Control token for Apple AI training use
Common Crawl	CCBot	Open web corpus used by many models

Vendors change these names and behaviors. Verify against each operator's current documentation before you ship rules, and note that some user-initiated agents may not follow robots.txt the way crawlers do.

Recommended setup for most businesses

If you want to be cited in AI answers and found by agents, allow the search and user agents. Decide training access separately based on your content strategy:

# Allow AI search and user-initiated fetching
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

# Optional: opt out of model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

# Everyone else
User-agent: *
Allow: /
Disallow: /admin
Disallow: /dashboard
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

Mistakes we see all the time

• Blanket Disallow: / for unknown bots. Many security plugins and hosting templates ship this. It silently removes you from AI search.
• Blocking /_next/, /static/ or asset folders. Rendering crawlers need them to see your content.
• Confusing robots.txt with noindex. Disallow stops crawling, but a blocked URL can still be listed if linked elsewhere. Use noindex on a crawlable page to remove it.
• Forgetting the CDN. A WAF or bot-protection rule can return 403 to a user agent that robots.txt allows. Test with a real request.
• No Sitemap line. It costs nothing and helps every crawler.

Test it

1. Fetch https://yourdomain.com/robots.txt and confirm it returns 200 as text/plain.
2. For each user agent you care about, check which group applies. A specific group overrides the * group.
3. Request a key page with that user agent string and confirm your CDN does not block it.
4. Run an AgentReady scan. Crawler access is one of the first things it checks, and a blocked crawler caps the rest of your score.

Where robots.txt fits in the bigger picture

Access is the foundation, but it is only step one of an AI readiness assessment. Once crawlers can reach you, accurate structured data and an llms.txt help them understand you, and WebMCP lets agents act.