Skip to content

How to allow AI crawlers in robots.txt (GPTBot, ClaudeBot and more)

AI companies run separate bots for training, search and live fetches. Here is how to let the ones that cite you through robots.txt, block training if you want, and find the CDN and plugin rules that quietly override you.

The seoscore team 10 min read

On this page
  1. Why robots.txt decides whether AI engines can quote you
  2. Training bots, search bots and user-triggered bots
  3. AI crawler user agents to know
  4. How to check what your robots.txt allows right now
  5. robots.txt examples to copy
  6. When robots.txt says yes but bots are still blocked
  7. Is llms.txt a replacement for robots.txt?
  8. How to confirm AI crawlers can reach your pages
  9. After access: checking whether you're actually being cited
  10. Check your site with seoscore
  11. Next steps

To be quoted by AI engines, allow their search and user-triggered crawlers in robots.txt. That means OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot and Claude-User for Claude, PerplexityBot for Perplexity, and Googlebot and Bingbot for Google AI Overviews and Copilot. Training crawlers such as GPTBot, ClaudeBot and Google-Extended are separate, so you can block them and still be cited. Then check that your CDN, firewall or a plugin isn't blocking the bots your robots.txt lets in.

Why robots.txt decides whether AI engines can quote you

AEO (answer engine optimization) is making your pages easy for AI engines to read, trust and quote. The first condition is basic: an answer engine can't quote a page its crawler was never allowed to fetch. robots.txt, the plain text file at the root of your domain, is the first thing every well-behaved crawler checks.

The same applies to GEO (generative engine optimization), which is whether AI assistants name and cite your brand in their answers. Crawler access is the page-side prerequisite for both. If the bots can't get in, nothing else you do on the page counts. The GEO guide covers the wider picture, and our article on answer engine optimization covers the rest of the page work.

Training bots, search bots and user-triggered bots

Most AI companies now run several crawlers, each with its own user agent (the name a bot gives when it requests a page). They fall into three kinds:

  • Training crawlers collect pages that may be used to train future models. Examples: GPTBot, ClaudeBot, CCBot. Blocking them limits what goes into future training data. It doesn't remove you from AI search answers.
  • Search and index crawlers build the index that cited AI answers draw on. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot, plus Googlebot and Bingbot. Block these and you largely drop out of cited answers. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they may still appear as navigational links.
  • User-triggered fetchers visit a page when someone asks an assistant something that needs it. Examples: ChatGPT-User, Claude-User, Perplexity-User. Blocking them stops live retrieval, but not all of them honour robots.txt.

This split is what makes a middle path possible: allow search and user fetches, block training.

A shared training opt-out is also coming. On 30 October 2026 Google said it plans to support a robots.txt policy token, __usage_ai-training, from early 2027, treating it as a Google-Extended opt-out when Google-Extended isn't named; Microsoft and OpenAI said they would honour it too (summary). Anthropic says it already recognises a machine-readable no-training signal, but a rule that names ClaudeBot directly takes priority. The token covers training only, so if you want to be cited you still need the per-bot rules below.

AI crawler user agents to know

Company User agent token Purpose Respects robots.txt Effect of blocking on AI answers
OpenAI GPTBot Model training Yes None on ChatGPT search; excludes content from training
OpenAI OAI-SearchBot ChatGPT search results Yes Not shown in ChatGPT search answers
OpenAI ChatGPT-User Fetches for user actions in ChatGPT May not apply Little; use OAI-SearchBot to opt out of search
Anthropic ClaudeBot Model training Yes None on search; excludes future content from training
Anthropic Claude-SearchBot Claude search quality Yes May reduce visibility in Claude's search results
Anthropic Claude-User Fetches when a Claude user asks Yes Claude can't retrieve your page for users
Perplexity PerplexityBot Perplexity search index Yes Not surfaced or linked in Perplexity answers
Perplexity Perplexity-User User-requested fetches Generally not Little; it may fetch anyway
Google Googlebot Search, including AI Overviews and AI Mode Yes Removes you from Google Search entirely
Google Google-Extended Control token for Gemini training and grounding Yes (no separate crawler) None on Search or AI Overviews
Microsoft Bingbot Bing and Copilot Yes Removes you from Bing and Copilot answers
Apple Applebot Siri, Spotlight, Safari Yes Removes you from Apple's search features
Apple Applebot-Extended Control token for Apple model training Yes (no separate crawler) None on Siri or Spotlight
Meta meta-externalagent AI training and products Yes, per Meta Limits use in Meta AI
Common Crawl CCBot Open web dataset, widely used for training Yes None on search answers
ByteDance Bytespider Training Widely reported not to Little; robots.txt alone may not stop it

Sources: OpenAI's bots page, Anthropic's crawler page, Perplexity's bots guide, Google's common crawlers page and Search Engine Journal's report on Claude's bots.

Three points people get wrong:

  • Google-Extended does not control AI Overviews. AI Overviews and AI Mode use Googlebot and the normal Search index. Google-Extended covers Gemini model training and grounding in Gemini Apps, so blocking it doesn't keep you out of AI Overviews. Google's page on AI features and your website explains the controls that do.
  • Google-Extended and Applebot-Extended never appear in your logs. They are robots.txt tokens that govern how data from Googlebot's and Applebot's crawls is used.
  • Newer tokens keep appearing, such as Amzn-SearchBot, MistralAI-Index, DuckAssistBot and meta-webindexer, and Anthropic's agent browsers add Claude-Agent to a normal browser user agent. Recheck the list from time to time.

How to check what your robots.txt allows right now

Open the file

Go to https://yourdomain.com/robots.txt in a browser. Each subdomain has its own file, so check shop. or blog. subdomains separately.

Read the groups

A group starts with one or more User-agent: lines followed by Allow: and Disallow: rules. Under Google's robots.txt rules, which follow the RFC 9309 standard:

  • A crawler obeys only the single most specific group that matches its name. If a User-agent: GPTBot group exists, GPTBot ignores the User-agent: * group completely.
  • If no group names it, the bot falls back to *. So User-agent: * with Disallow: / blocks every AI bot you haven't named.
  • Within a group, the longest matching path wins. If an Allow and a Disallow tie, the Allow wins.
  • User agent names are case-insensitive. Paths are case-sensitive.

Look for lines you didn't write

CMS settings, SEO plugins and CDNs can add or generate robots.txt rules. WordPress, for example, serves a virtual robots.txt that plugins can edit, so the file on your server may not be what visitors get. Always judge the live file at the URL, not your copy. If you find AI bots blocked that you never chose, trace which plugin or service added them before you edit.

robots.txt examples to copy

Replace the example paths and sitemap URL with your own.

1. Allow all AI crawlers

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

You don't need a group per bot. Every crawler without its own group follows *. Adding User-agent: GPTBot with Allow: / would actually remove your /admin/ rule for GPTBot.

2. Allow AI search and user fetches, block training

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
Disallow: /

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

Search and user bots (OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, Bingbot) fall through to * and are allowed. Note the trade-off: blocking Google-Extended also stops grounding in Gemini Apps, which is a place you might want to be cited.

3. Allow AI search crawlers into some sections only

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /blog/
Allow: /guides/
Disallow: /

The longer Allow paths beat Disallow: /, so these bots can reach /blog/ and /guides/ only. That is how RFC 9309 and Google resolve it. If you rely on it, confirm in your logs that each bot behaves that way.

4. Keep private paths blocked for every bot

Because a bot only reads its own group, every named group must repeat your private rules:

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /account/
Disallow: /checkout/

User-agent: *
Disallow: /account/
Disallow: /checkout/

robots.txt is a request, not security. Put anything genuinely private behind a login.

Where to put it and mistakes to avoid

  • The file must sit at the root (/robots.txt) as plain UTF-8 text, on every subdomain.
  • One directive per line, and paths start with /.
  • Disallow: with nothing after it means allow everything; Disallow: / means block everything.
  • Google ignores anything past 500 KiB, so keep the file lean.
  • A robots.txt that returns a server error (5xx) can be treated as a full block. A 404 means no restrictions.
  • Changes aren't instant. Google generally caches robots.txt for up to 24 hours, and OpenAI says a change takes about 24 hours to affect its search.

When robots.txt says yes but bots are still blocked

A correct robots.txt is often overridden further up the chain.

CDN and WAF bot rules

CDNs and web application firewalls (WAFs, which filter traffic before it reaches your server) can block bots outright. Since July 2025 Cloudflare has asked new domains about AI crawlers at signup and blocks them by default. It also offers a managed robots.txt that adds AI-bot rules to your file, a "Block AI bots" setting and Bot Fight Mode, any of which can stop crawlers your file allows. Per-bot controls live in AI Crawl Control. OpenAI's ChatGPT search help article tells site owners to allow OAI-SearchBot and OpenAI's IP addresses at the host and CDN.

Firewalls, plugins, rate limits and geo-blocking

  • Hosting firewalls may block data-centre traffic, which is where all crawlers come from.
  • Security plugins with bot blocking or aggressive rate limiting can challenge or reject AI user agents.
  • Rate limits that return 429 errors slow or stop crawling.
  • Geo-blocking can catch crawlers. Check their published IP ranges against any country rules.

Pages that need JavaScript

Many AI crawlers fetch the raw HTML and don't reliably run JavaScript. If your text only appears after scripts run, the bot may see an empty page. Use server-side rendering or static HTML for the content you want quoted.

Snippet rules

Crawling allowed doesn't mean quoting allowed. Google applies nosnippet, data-nosnippet, max-snippet and noindex to its AI features, so a nosnippet meta tag can keep a page out of AI Overviews even when Googlebot can fetch it. Check templates for these tags on pages you want cited.

Is llms.txt a replacement for robots.txt?

No. llms.txt is a markdown file at /llms.txt that points AI assistants to your most useful pages, with a one-line description of each. It controls nothing: a bot blocked in robots.txt won't read your pages because llms.txt lists them. Use robots.txt for access and llms.txt as a guide, listing your key product, pricing, documentation and help pages. The llms.txt generator creates the file and checks an existing one.

How to confirm AI crawlers can reach your pages

Search your server logs

Search access logs for the user agent tokens:

grep -iE "OAI-SearchBot|ChatGPT-User|Claude-SearchBot|Claude-User|PerplexityBot|GPTBot|ClaudeBot" access.log

Look at the status codes. 200 means the bot got the page. 403, 429 or 503 points to a firewall, rate limit or CDN rule. No hits at all after a few weeks can also mean a block upstream, where the request never reaches your server.

Check the IP ranges

Anyone can fake a user agent. Compare the requesting IP with the lists the companies publish: OpenAI's openai.com/gptbot.json, searchbot.json and chatgpt-user.json; Anthropic's claude.com/crawling/bots.json; Perplexity's perplexitybot.json; Bing's bingbot.json; and Google's googlebot.json. Google and Apple crawlers can also be checked with a reverse DNS lookup. If you allow-list IPs in a firewall, pull these files automatically, as they change. Don't block AI bots by IP as your opt-out: Anthropic warns this may not work reliably, because a blocked bot can't read your robots.txt either.

Test the page

Logs tell you what happened; a test tells you what will happen. Run your page through the free AEO and GEO checker. It checks AI crawler access in robots.txt, whether the content loads without JavaScript and whether snippet rules limit quoting, along with llms.txt, entity markup and answer blocks.

After access: checking whether you're actually being cited

Allowing crawlers is necessary but not enough. To see whether it pays off, ask ChatGPT, Perplexity and Google the questions your customers ask and note whether your pages appear as sources. That works for a handful of questions but doesn't scale.

If you are crawled but not cited, the cause is usually the page: no direct answer near the top, thin content, no clear authorship or dates, or a competitor that answers better. Our GEO article covers what to fix next.

Check your site with seoscore

  • Run key pages through the AEO and GEO checker after any robots.txt, CDN or plugin change.
  • Run an SEO audit to see the whole site. AEO is 15 of the 100 points, and each fix is ranked by the points it adds. The methodology page lists every check.
  • Use the AI mentions report to see where Google AI Overviews and ChatGPT cite your site, which pages, for which questions and which other sites appear beside you. It is on Starter and up, with a free trial.

Next steps

  1. Open your live robots.txt and list every group that matches an AI bot, including *.
  2. Decide your policy: allow everything, or allow search and user bots and block training.
  3. Paste the matching example, keeping your private paths in every group.
  4. Check your CDN, firewall and security plugins for AI-bot blocking.
  5. Remove stray nosnippet rules and make sure key content renders without JavaScript.
  6. After a day or two, check your logs and test the page again.

Questions people ask

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects data for model training. ChatGPT's search answers rely on OAI-SearchBot, so you can block GPTBot and still appear in ChatGPT search as long as OAI-SearchBot is allowed.

Does blocking Google-Extended stop my pages appearing in AI Overviews?

No. AI Overviews and AI Mode are part of Google Search and use Googlebot. To limit how your content appears there, Google points to the normal Search controls such as nosnippet, max-snippet and noindex.

How long does a robots.txt change take to work?

Usually up to a day. Google generally caches robots.txt for up to 24 hours, and OpenAI says changes take about 24 hours to affect its search. Other crawlers may take longer to revisit.

Should I block AI crawlers by IP address instead of robots.txt?

It is rarely the best tool for opting out. Anthropic warns that IP blocks may not reliably work as an opt-out, because a blocked bot can't read your robots.txt rules either. Use robots.txt for policy and keep IP rules for verifying real bots or stopping abusive traffic.

Is it safe to allow all AI crawlers?

For most public marketing, product and help pages, yes: it gives AI engines the best chance to quote you. The trade-off is that training crawlers may use your content in future models. If that matters to you, block the training bots and keep the search and user bots allowed.

Sources

  1. One robots.txt line to opt out of AI training: Google, Bing and the IETF's new policy token : fireup.pro fireup.pro
  2. Does Anthropic crawl data from the web, and how can site owners block the crawler? support.claude.com
  3. Overview of OpenAI Crawlers developers.openai.com
  4. developers.google.com/search/docs/crawling-indexing/google-common-crawlers developers.google.com
  5. Anthropic’s Claude Bots Make Robots.txt Decisions More Granular www.searchenginejournal.com
  6. developers.google.com/search/docs/appearance/ai-features developers.google.com
  7. developers.google.com/search/docs/crawling-indexing/robots/robots_txt developers.google.com

seoscore

Will AI answer engines quote you?

Run the free AEO checks on any page and see what to fix first.

Check my AEO
More on AEO