AI Crawlers and robots.txt: How to Control Access
Treat AI crawlers by job: allow the search bots that let assistants cite you, decide deliberately on training bots, and make your CDN enforce the same rules.
To control how AI crawlers use your site, sort them by job. Allow the search and user-triggered crawlers that let ChatGPT, Claude, and Perplexity retrieve and cite your pages. Make a deliberate call on training crawlers like GPTBot and ClaudeBot. Then confirm that your CDN and firewall enforce the same policy your robots.txt states.
By the end of this guide you will have three things. The first is a robots.txt policy that handles training, search, and user-triggered AI crawlers separately. The second is a content delivery network (CDN) and firewall setup that matches that policy. The third is a way to confirm in your logs that the crawlers you want are getting through. Before you start, you need:
Edit access to robots.txt on every hostname you serve
At least a week of raw server or CDN logs
Admin access to your CDN or bot management dashboard
A short list of site sections that hold material you consider proprietary
Which AI crawlers should you know about?
The crawlers that matter most come from OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, Meta, and Common Crawl. Most AI vendors now run separate agents for training, for search indexing, and for fetching a page when a user asks about it. Each agent needs its own decision.
User agent or token | Operator | What it does | What blocking it changes |
|---|---|---|---|
GPTBot | OpenAI | Collects pages for model training | Your future content stays out of OpenAI training data |
OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search | ChatGPT is less able to surface and cite you when it searches |
ChatGPT-User | OpenAI | Fetches a page because a user's request needs it | Limits on-demand reads of specific pages |
ClaudeBot | Anthropic | Collects pages for model training | Your future content stays out of Anthropic training data |
Claude-SearchBot | Anthropic | Indexes pages for Claude's search results | Claude is less able to cite you when it searches |
Claude-User | Anthropic | Fetches a page on a user's behalf | Limits on-demand reads |
PerplexityBot | Perplexity | Indexes pages for Perplexity answers | Perplexity is less able to cite you |
Perplexity-User | Perplexity | Fetches a page on a user's behalf | Limits on-demand reads |
Google-Extended | A robots.txt token, not a crawler | Content Googlebot fetched is withheld from Gemini training and Gemini grounding. AI Overviews and Search are unaffected | |
Applebot-Extended | Apple | A robots.txt token, not a crawler | Content Applebot fetched is withheld from Apple's model training |
CCBot | Common Crawl | Builds an open web archive that many training datasets draw on | Your future pages stay out of new Common Crawl snapshots |
Meta-ExternalAgent | Meta | Collects pages for Meta's AI models | Your future content stays out of Meta training data |
Googlebot | Builds the Search index | You leave Google Search, AI Overviews, and AI Mode | |
Bingbot | Microsoft | Builds the Bing index | You leave Bing and lose the index Copilot grounds its answers in |
Google and Microsoft break the separate-bot pattern. AI Overviews and AI Mode draw on the same index Googlebot builds for regular Search, and Copilot grounds its answers in Bing's index. There is no way to stay in those search engines while blocking their AI answers at the robots.txt level.
Vendors rename and add agents, so treat this table as a starting map. Check each company's crawler documentation before you publish changes. Many vendors also publish the IP ranges their crawlers use.
How do training crawlers differ from search crawlers?
Training crawlers feed what a model learns. Search crawlers feed what an assistant can find and cite when it looks something up. The two map onto the two halves of AI Optimization (AIO).
Training is the territory of Generative Engine Optimization (GEO). Pages a training crawler collects may end up in the data a future model learns from. That shapes what the model says about your category when it answers without searching. The effect shows up only after the next model is trained, and blocking a crawler today does not remove anything it already collected.
Retrieval is the territory of Answer Engine Optimization (AEO). When ChatGPT, Claude, or Perplexity searches the web to answer a question, it pulls from an index its search crawler built, or from a search engine index like Bing's or Google's. If you block that crawler, the assistant has nothing of yours to retrieve, so it cites a competitor or a review site instead.
User-triggered fetchers sit in a third category. A person pastes your URL into a chat or asks about a specific page, and the assistant fetches it on the spot. Vendors treat these requests differently from automated crawling. Some state that robots.txt may not apply because a person initiated the request. If you need a hard block on these, it has to happen at your CDN or firewall.
For a brand that wants to show up in AI answers, blocking search crawlers works against you. The real decision is about training crawlers.
How to configure AI crawler access step by step
Inventory every robots.txt: Fetch robots.txt from each hostname you serve, including www, the bare domain, docs, blog, and help subdomains. Each host is governed only by its own file, so a strict file on a docs subdomain can quietly hide your best answer content.
Pull your logs: Filter a week of server or CDN logs for the user agents in the table above. Note which ones visit, which paths they request, and what status codes they get back.
Classify your content: Split the site into marketing and support content you want quoted, and proprietary content you would rather keep out of models. Product pages, documentation, comparisons, and pricing usually belong in the first group.
Decide policy per crawler type: Allow search crawlers on everything public. Decide whether training crawlers get the whole site, everything except the proprietary sections, or nothing.
Write explicit user-agent groups: Give each AI crawler category its own group in robots.txt, and repeat any shared disallow rules inside every group.
Check your CDN and firewall: Open your bot management settings in Cloudflare, Akamai, Fastly, or whatever sits in front of your site. Confirm that no AI-bot blocking rule, challenge page, or rate limit contradicts what robots.txt allows.
Confirm content renders without JavaScript: Request key pages with JavaScript disabled or with a plain HTTP client. Many AI crawlers read the raw HTML and do not execute scripts, so client-rendered content can come back as an empty shell.
Test and monitor: Verify the rules with a robots.txt parser and request logs, then recheck monthly as vendors add new agents.
How should you decide what to block?
Base the decision on whether a piece of content is your marketing or your product. Marketing content exists to be repeated, and having models learn it is the point. Product content, such as a paid research library, a proprietary dataset, or premium course material, loses value when a model can paraphrase it for free.
Take a B2B analytics company with product pages, public documentation, a blog, and a paid annual benchmark report. Product pages, docs, and blog posts should be open to every crawler, training included. Those pages are how a model learns what the company does and which problems it solves. The full report should sit behind a login, where no crawler reaches it anyway. A free executive summary on a public page can stay open to search crawlers, so assistants can cite it and send readers to the gated version. Training crawlers could be disallowed from that summary path if the company would rather models not absorb the findings.
Two cautions apply to competitive protection. First, robots.txt is a public file. Listing a path like /internal-pricing/ advertises that it exists to anyone who reads the file, so anything confidential belongs behind authentication. Second, robots.txt is a request, not a lock. The major declared crawlers say they honor it, but nothing forces them to, and a competitor reading your page in a browser ignores it entirely.
Blocking all training crawlers across a whole marketing site rarely helps. You give up influence over what future models know about your category and gain very little protection, because the same content is already summarized on review sites and forums that models do learn from.
Writing the robots.txt groups correctly
A crawler follows only the single most specific group that names it and ignores the general User-agent: * group. This is the rule that trips up most teams. Add a GPTBot group with one line in it, and GPTBot stops reading every disallow you wrote under the asterisk.
Here is a policy that allows search crawlers everywhere public, keeps training crawlers out of a research section, and keeps every bot out of admin and checkout paths:
` User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Disallow: /admin/ Disallow: /checkout/
User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Meta-ExternalAgent User-agent: Google-Extended User-agent: Applebot-Extended Disallow: /admin/ Disallow: /checkout/ Disallow: /research/
User-agent: * Disallow: /admin/ Disallow: /checkout/
Sitemap: [full address of your sitemap] `
The first group covers search and answer crawlers, the second covers training crawlers and tokens, and the asterisk group covers everyone else, including Googlebot and Bingbot. On the Sitemap line, use the complete address of your sitemap file, such as the sitemap.xml file at the root of example.com.
Stacking several User-agent lines above one set of rules is valid under the Robots Exclusion Protocol standard and keeps the file readable. Notice that the admin and checkout disallows appear in every group. Leave them out of a named group and that crawler is free to request those paths.
The Sitemap line helps search crawlers find pages you want indexed. It does not change training behavior, and no robots.txt directive will get a page cited. It only decides whether a crawler is allowed to read the page in the first place.
How do you know it worked?
Your logs are the first proof. Within a few days of publishing, allowed crawlers should fetch robots.txt and then request your pages with 200 status codes. Disallowed crawlers should fetch robots.txt and then stop requesting the blocked paths. A run of 403 responses or challenge pages served to an allowed crawler means your CDN is overriding your file.
Next, check identity. Anyone can send a request labeled GPTBot. Match the requesting IP addresses against the vendor's published ranges, or use a reverse DNS lookup for Googlebot and Bingbot, before you trust what the logs say about who visited.
Then test retrieval directly. Publish or update a page, wait for the search crawlers to visit it, and ask Perplexity or ChatGPT a question that page answers well. Check whether the answer lists your page as a source. Run the prompt several times, because AI answers vary between runs and a single result tells you little.
Finally, watch referral traffic in your analytics. Visitors who click a cited link arrive from domains like chatgpt.com, perplexity.ai, claude.ai, gemini.google.com, and copilot.microsoft.com. That traffic only appears when people click, so it undercounts how often you are cited, but a jump after you open access is a good sign.
nonBot AI checks whether AI crawlers can read your site and tracks which of your pages show up as sources across ChatGPT, Gemini, Perplexity, Claude, and Google AI Overviews, which covers the retrieval side without manual prompting.
What are the most common mistakes?
Most broken setups come from one of four problems, and each one is easy to miss because nothing visibly breaks on the site.
Named groups that drop the general rules: A team adds an Allow group for GPTBot or PerplexityBot and forgets that those bots now ignore the asterisk group. The bot gets into admin, staging, or faceted search paths it was never meant to crawl. Repeat every shared disallow inside every named group.
A CDN that disagrees with robots.txt: Cloudflare and other providers offer one-click AI bot blocking, and some apply it by default on new zones. Your robots.txt says welcome while the edge returns a challenge page, and the search crawler leaves with nothing. Whoever owns robots.txt and whoever owns the CDN need to agree on one policy.
Blocking the wrong Google or Microsoft agent: Blocking Google-Extended does not remove you from AI Overviews, and teams that block Googlebot or Bingbot to stop AI use disappear from regular search as well. If you want to limit how Google quotes you in AI features, look at the nosnippet and max-snippet controls, and remember that they also shorten your regular search snippets.
Content that only exists after JavaScript runs: A pricing table or feature list rendered client-side can be invisible to crawlers that read raw HTML. Server-side rendering or static generation for important pages fixes this.
A broken robots.txt can also block everything. Google treats a robots.txt that returns a server error as a temporary instruction to crawl nothing, and other crawlers may behave the same way. Make sure the file returns a 200 status as plain text.
Start today by requesting robots.txt from every hostname you own, then compare it line by line with your CDN's bot settings. Fix any place where the two disagree before you change anything else.
FAQ
Does blocking GPTBot remove my content from ChatGPT?
No. Blocking GPTBot keeps future pages out of OpenAI's training data, but anything already collected stays in models trained on it. ChatGPT search also relies on OAI-SearchBot and other sources, so a page blocked from GPTBot can still be retrieved and cited when ChatGPT searches.
Does llms.txt replace robots.txt for AI crawlers?
No. llms.txt is a proposed file that points AI systems to your most useful content in a clean format, and it carries no access rules. robots.txt remains the file crawlers check for permission, so you need it regardless of whether you publish llms.txt.
Can I keep my content out of AI Overviews but stay in Google Search?
Only partially. AI Overviews use the same index as regular Search, so robots.txt cannot separate them. Google's snippet controls, such as nosnippet and max-snippet, limit how much of a page can be quoted in AI features, but they apply to regular search snippets too.
What should I do about crawlers that ignore robots.txt?
Enforce the policy at your CDN or firewall. Block or rate-limit by verified IP range and user agent there, because those rules apply whether or not the crawler cooperates. Keep watching your logs, since undeclared crawlers often show up as unusual traffic patterns from cloud hosting IP ranges.
How often should I review my AI crawler rules?
Monthly is a sensible default. AI vendors add and rename agents, and a new search crawler you have not allowed can leave you out of one assistant's answers without any error on your side. Recheck after any CDN change or site migration too, because those are the moments when bot rules get reset.
Measure Your Brand's AI Visibility
See how often AI assistants like ChatGPT and Perplexity recommend your business.
Free analysis • No credit card required
About nonBot AI: We help brands optimize their visibility across AI platforms—both retrieval-based and training-based. Our AI Visibility tool tracks your presence across ChatGPT, Perplexity, Claude, and more. If you're ready to build a real AIO strategy, talk to an expert.
