Bot Sniffer
Bot Sniffer tracks which AI bots are actually visiting your website. While Crawler Detection checks whether your robots.txt allows AI crawlers, Bot Sniffer tells you which ones are actually showing up, giving you concrete evidence of AI system interest in your content.
Why Bot Detection Matters
Knowing which AI bots crawl your site helps you confirm that AI systems are indexing your content, identify which AI providers are actively interested in your pages, correlate bot visits with changes in your AI visibility, validate that robots.txt changes are having the intended effect, and prioritize optimization for the AI models that are actually crawling you.
How It Works
Bot Sniffer uses a lightweight tracking pixel installed on your website. When any visitor, human or bot, loads a page with the pixel, the visit is logged and analyzed. AI bots are identified by their user agent strings, and the data is aggregated in your nonBot dashboard.
Installation
Navigate to Playbook > Bot Sniffer and choose one of two installation methods.
Direct HTML
Copy the provided HTML snippet and paste it into the <head> section of your website. This method catches all bots, including those that don't execute JavaScript. It's the recommended approach for maximum coverage.
Google Tag Manager
If you use GTM, copy the provided snippet and add it as a Custom HTML tag in your GTM container. This method is easier to install if you already use GTM, but it only catches bots that execute JavaScript; some AI crawlers don't.
Each installation method uses a unique tracker ID tied to your account. After installing, click Verify Installation to confirm the pixel is working correctly.
Reading Your Data
Once installed, the Bot Sniffer dashboard shows the status indicator: green means data is being received, yellow means the pixel is installed but no bot visits have been detected yet. For each bot, you can see how frequently it visits and when it was last detected.
Tracked AI Bots
Bot Sniffer detects 19 AI crawlers across all major providers:
| Bot Name | Provider | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data collection for GPT models |
| ChatGPT-User | OpenAI | Real-time web browsing for ChatGPT responses |
| OAI-SearchBot | OpenAI | SearchGPT and AI search results |
| Googlebot | Google Search indexing and AI Overviews | |
| Google-Extended | Gemini and Vertex AI training data | |
| Google-CloudVertexBot | Google Cloud Vertex AI data collection | |
| ClaudeBot | Anthropic | Training data collection for Claude models |
| anthropic-ai | Anthropic | Anthropic research and evaluation |
| PerplexityBot | Perplexity | Perplexity AI search answers |
| xAI-Grok | xAI | Grok model training and X/Twitter AI features |
| meta-externalagent | Meta | Meta AI and Llama model training |
| Applebot-Extended | Apple | Apple Intelligence and Siri AI features |
| Amazonbot | Amazon | Alexa answers and Amazon AI services |
| Bytespider | ByteDance | TikTok and ByteDance AI training |
| DeepSeekBot | DeepSeek | DeepSeek model training data |
| YouBot | You.com | You.com AI search engine |
| Timesbot | Various | News and media AI aggregation |
| cohere-ai | Cohere | Cohere enterprise AI model training |
| CCBot | Common Crawl | Open dataset used by many AI models |
This list is continuously updated as new AI crawlers emerge.
What to Do With This Data
If a specific AI model isn't crawling your site, check that your robots.txt isn't blocking its user agent (use Crawler Detection to verify). If AI bots are visiting but your AI visibility isn't improving, the issue is likely with your content authority or entity signals rather than access. If you see increased crawl activity after making content changes, that's a positive signal that AI systems are re-indexing your updated content.
Limitations
Bot Sniffer can only detect bots that actually visit your website. It cannot detect AI models that reference your brand from third-party sources without visiting your site directly. Some AI systems use residential IP addresses or generic user agents that may not be identifiable as bots. The tracking pixel requires page loads to register visits, so API-only crawls won't be detected.
Access Requirements
Bot Sniffer is available on Pro, Elite, and Agency plans.
Next Steps
Review which bots have access to your site with Crawler Detection, or run a Site Audit to check your overall AI readiness.