Robots.txt Checker

Most robots.txt testers answer one question: can Googlebot crawl this URL. That was the only question worth asking when Google was the only crawler that mattered. It is not anymore. Every AI assistant that cites live sources has its own crawler, each obeys robots.txt separately, and each can be blocked without touching your Google traffic at all.

This checker fetches your robots.txt and resolves eight AI crawlers against it: GPTBot and ChatGPT-User (OpenAI), ClaudeBot and anthropic-ai (Anthropic), PerplexityBot, Google-Extended, Bytespider (ByteDance), and CCBot (Common Crawl, which feeds training data to most of the field). For each one it reports allowed, blocked, or not specified. It reads the crawler's own User-agent block first and falls back to the wildcard block, which is how a compliant crawler reads the file.

It also checks whether you have an llms.txt at the root of your site. robots.txt says who may read you; llms.txt says what you would like them to read.

Most sites come back with almost everything marked not specified, and that is fine. Robots.txt is a deny list. A crawler that is not mentioned is allowed by default, so an empty file and a file that welcomes everyone behave identically.

What matters is the wildcard block. A User-agent: * followed by Disallow: / blocks every crawler that has no rule of its own, which on most sites means every AI crawler at once. That line usually arrives from a staging config that was never removed, and it is the single most common way a site disappears from AI answers without anyone noticing.

It depends on which crawler, and the distinction people miss is between training and retrieval. CCBot and Google-Extended feed model training. Blocking them keeps your content out of the next model, and costs you nothing in citations today. GPTBot, ClaudeBot, and PerplexityBot fetch pages to answer a question someone is asking right now. Blocking those removes you from the answer.

Google-Extended is worth checking specifically. A lot of teams added it during the 2023 scraping arguments, when the trade looked like giving content away for nothing. Google now uses it for AI Overviews, which sit above the organic results those same teams are trying to rank in. The block is still in place on a surprising number of sites, and nobody has revisited the decision.

If your business depends on being recommended, blocking retrieval crawlers is a decision to be invisible. If your content is your product and you are protecting it from being trained on, blocking the training crawlers is reasonable. Either way it should be a decision, not a leftover.

Permission is not traffic. robots.txt tells you an AI crawler is allowed to visit; it says nothing about whether one ever has. Sites that pass this check cleanly and still never appear in AI answers are common, and the reason is usually further down the stack: pages that render entirely in JavaScript, so a crawler that does not run scripts sees an empty shell, or content nobody else has ever linked to or written about.

Fix the block first, because nothing else works while it is there. Then check whether the pages behind it are readable, and whether the models actually mention you.