When someone asks ChatGPT, Claude, Perplexity or Gemini how to do something with your product, the answer is only as good as what the assistant could read about it. Clear, open documentation is the best way to make sure AI assistants give correct answers, and link to your docs while they do.
This guide covers three things: letting the right crawlers in, structuring your docs so AI can understand them, and checking the results with AI crawler analytics.
1. Let AI crawlers read your docs
AI companies use different crawlers for different jobs. You can allow or block each one in your site's robots.txt file:
| Purpose | What it means | Crawlers |
|---|---|---|
| AI search | Indexes your pages so assistants can find and cite them in answers. | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, Amazonbot, DuckAssistBot |
| On request | Reads a page when a person asks an assistant about it. | ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User |
| Training | Collects pages to train future AI models. | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider |
Check your robots.txt
Open https://your-site.com/robots.txt in your browser. If you see lines like these, AI crawlers are blocked:
A rule for User-agent: * with Disallow: / blocks every crawler, including search engines.
Allow AI search, block training
WordPress creates a virtual robots.txt. Edit it with your SEO plugin (Yoast: Yoast SEO → Tools → File editor; Rank Math: Rank Math → General Settings → Edit robots.txt), or upload a robots.txt file to your site's root folder. This example lets AI assistants find and cite your docs while opting out of training:
To allow everything, you don't need any AI rules at all: crawlers are allowed unless something blocks them.
Google-Extended and Applebot-Extended aren't separate crawlers. They tell Google and Apple whether they may use your pages for their AI models. Blocking them doesn't affect your normal Google or Apple search results. On request crawlers act for a person, so some of them don't follow robots.txt rules.2. Make your docs easy for AI to understand
AI assistants read your pages much like a new reader skimming for an answer. What helps people helps them too.
Keep docs public and fast
- Public knowledge bases only. Docs that need a login can't be read by any crawler. Keep internal docs restricted, and everything customers need public.
- Text, not pictures of text. Write steps, settings and error messages as text. Use screenshots to show things, with an alt text that says what's in them.
- Code as code. Put commands, settings and examples in code blocks, never in images.
One topic per article
- Give each article one clear job, such as Connect Stripe or Reset a password, so an assistant can match it to a question.
- Use titles that say what the article does, in the words your customers use.
- Write a short excerpt for each article. It's used in search results and in the page description that crawlers read.
Structure every article
- Start with one or two sentences that answer the main question. Assistants often quote the opening.
- Use headings (H2, H3) that describe the section, not "Overview" or "More".
- Use the PinkDocs blocks: Steps for procedures, FAQ for common questions, Alert for warnings and Tabs for alternatives. Their structure is easy for AI to follow.
- For APIs, use the API reference blocks. Methods, parameters and example requests in a fixed format are much easier to read correctly than prose.
- Name things exactly as they appear in your product: the menu path, the button label, the setting name.
Keep it current
- Update articles when your product changes, and remove ones that no longer apply. Outdated docs lead to wrong AI answers.
- Keep your changelog up to date. Assistants use it to answer "Does it support…?" and "What changed in…?" questions.
- Link related articles to each other, so crawlers find all of them and see how they fit together.
3. Let search engines and AI find every page
PinkDocs already does most of this for you:
| What | Where |
|---|---|
| Page titles, descriptions and social previews | PinkDocs → Settings → General → Search engines |
| Structured data (schema.org) that describes each article | Search engines → Structured data (on by default) |
| Every public article in your WordPress sitemap | Automatic, at /wp-sitemap.xml (or your SEO plugin's sitemap) |
| Restricted docs kept out of sitemaps, search and feeds | Automatic |
Submit your sitemap in Google Search Console and Bing Webmaster Tools. Bing's index is also used by several AI assistants.
llms.txt is a proposed file that lists a site's most important pages for AI. It isn't an official standard yet, and the major assistants don't rely on it. PinkDocs doesn't create one. If you want one, upload a plain text file named llms.txt to your site's root folder with links to your main docs.4. Give AI tools direct access (optional)
Turn on the public docs MCP in PinkDocs → MCP. Your readers can then connect their own AI tools (Claude, ChatGPT, Cursor and others) to your published docs, and get answers straight from the latest version instead of from what a crawler read weeks ago. Drafts and restricted knowledge bases are never included. See Connecting PinkDocs to AI tools.
5. Check the results
Open PinkDocs → Dashboard → AI crawlers to see which crawlers read your docs and which articles they read most. Read more in AI crawler analytics.
Check that crawlers visit
After changing your robots.txt, give it a week or two. AI search crawlers should start to appear.
Look at the most-read articles
These are the pages most likely to shape AI answers about your product. Make sure they're accurate and up to date.
Ask the assistants yourself
Ask ChatGPT, Claude and Perplexity a few questions your customers ask. Check that the answers are right and link to your docs. If not, improve the article that should have answered.
Common questions
Rarely. AI crawlers visit far less often than search engines. If one becomes a problem, you can slow or block it in robots.txt or Cloudflare.
No. Crawlers can't log in, so docs that need a login are never read or used.
No. Answers with links come from the AI search and On request crawlers. Blocking training only stops your pages being used to train future models.
AI search crawlers usually pick up changes within days to weeks. On request crawlers read the live page, so they see changes right away.