More and more people find answers through ChatGPT, Claude, Perplexity, Gemini and other AI assistants instead of a search engine. Those assistants learn about your product by reading your docs. The AI crawlers report on the PinkDocs dashboard shows which AI crawlers visit your documentation, why they come and which articles they read.
Open the report
Go to PinkDocs → Dashboard and scroll to AI crawlers. The report uses the same knowledge base and date range as the rest of the dashboard, so switch either at the top of the page.
What the report shows
| Part | What it tells you |
|---|---|
| Crawler visits | How many times AI crawlers loaded your docs pages in the period. |
| AI crawlers seen | How many different crawlers visited. |
| Training | Visits from crawlers that collect pages to train AI models. |
| AI search | Visits from crawlers that index pages so AI assistants can find and cite them in answers. |
| On request | Visits made because someone asked an AI assistant about a page, and it read the page right then. |
| Crawlers | Each crawler with its company, purpose, how many docs it read and when it was last seen. |
| Most read by AI | The articles AI crawlers read most, and how many different crawlers read each one. |
Which crawlers are recognised
PinkDocs recognises the main AI crawlers by the name they send with each request (their user agent):
| Company | Training | AI search | On request |
|---|---|---|---|
| OpenAI (ChatGPT) | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic (Claude) | ClaudeBot, anthropic-ai | Claude-SearchBot | Claude-User |
| Perplexity | PerplexityBot | Perplexity-User | |
| Google (Gemini) | GoogleOther | Google-CloudVertexBot, Gemini Deep Research | |
| Apple | Applebot | ||
| Meta | Meta-ExternalAgent | Meta-ExternalFetcher | |
| Amazon | Amazonbot | ||
| DuckDuckGo, You.com | DuckAssistBot, YouBot | ||
| Mistral | MistralAI-User | ||
| Others | Bytespider (ByteDance), CCBot (Common Crawl), AI2Bot, Diffbot, Timpibot, Webz.io, Cohere |
Regular search engine crawlers like Googlebot and Bingbot aren't counted here: they show up in your search engine's own tools.
What is counted
- Documentation pages only: articles, categories, tags and your docs homepage. Other pages of your site aren't counted.
- Each page load, grouped by day, crawler and article. Repeat visits to the same article add up.
- Only while PinkDocs is active. If your licence isn't active, readers and crawlers get an error page instead of your docs, so those visits aren't counted either.
No personal data is stored: only the crawler's name, the page and the time. The data is kept for as long as your other reader statistics (PinkDocs → Settings → General → Privacy).
Why the report might be empty
Your docs are new or small
Crawlers find new sites gradually. It can take a few days to a few weeks before the first visits appear. Make sure your docs are linked from your site and listed in your sitemap.
Pages come from a cache or CDN
If a page cache plugin (WP Rocket, LiteSpeed Cache, W3 Total Cache…) or a CDN like Cloudflare serves the page, WordPress never sees the visit, so PinkDocs can't count it. The visit still happened: only the count is missing. Crawlers that read pages the cache hasn't stored yet are still counted.
Your robots.txt blocks AI crawlers
Many sites block AI crawlers without knowing it, through a security plugin, a hosting setting or Cloudflare's Block AI bots option. See Optimise your docs for AI crawlers to check.
Your docs are restricted
Docs in a knowledge base that needs a login aren't visible to crawlers, so they never visit them. This is on purpose.
Limits
- Crawlers are recognised by the name they send. A crawler that pretends to be a normal browser isn't recognised.
- Someone could send a fake crawler name. That's rare, and it only affects these counts, nothing else.
- Counts show what crawlers read, not whether an AI assistant later used your page in an answer.
Common questions
No. Recognising a crawler is a quick check of the request, and each visit adds or updates one small database row.
No. It only counts visits. To allow or block crawlers, use your robots.txt; see Optimise your docs for AI crawlers.
Yes. Add this to your theme's functions.php or a code snippets plugin: add_filter('pinkdocs_track_ai_crawlers', '__return_false');