Let each engine's search crawler in, be in the index it reads, and write pages it can quote in one sentence. ChatGPT, Perplexity, Claude and Google publish different rules for each step; this page puts them side by side, from their own docs.
Our pick
Would you rather not track six engines' rules yourself? supereo ours is our pick. It asks your buyers' questions on ChatGPT, Gemini, Perplexity, Claude, Google AI Mode and Google AI Overviews, reports how often each one recommends you, with its range, and tells you what to change on each page ($149 a month for Copilot). The free audit shows every answer from all six in about 2 minutes.
| Engine | Search crawler | Live fetcher | Training crawler | Where results come from, in its own words |
|---|---|---|---|---|
| ChatGPT | OAI-SearchBot; blocked sites "will not be shown in ChatGPT search answers" | ChatGPT-User; "robots.txt rules may not apply" | GPTBot | OpenAI's index, and "ChatGPT search sometimes partners with other search providers" (Microsoft and Shopify named) |
| Perplexity | PerplexityBot; "not used to crawl content for AI foundation models" | Perplexity-User; "generally ignores robots.txt rules" | None named | Its own index, "covering hundreds of billions of webpages" |
| Claude | Claude-SearchBot; blocking it "may reduce your site's visibility" | Claude-User | ClaudeBot | Web search subprocessors Brave Search, TurboPuffer and Parallel Web Systems; image results "powered by Bing" |
| Google AI Overviews and AI Mode | Googlebot; "There are no additional technical requirements" | None separate | Google-Extended, for Google's other AI systems | Google's Search index; a page "must be indexed and eligible to be shown in Google Search with a snippet" |
Sources, read November 8 and 9, 2026: OpenAI crawlers, ChatGPT search help, Perplexity crawlers, Perplexity Search API post, Anthropic crawlers, Anthropic subprocessors, Claude web search help, Google AI features.
Two rows have a catch. Claude's main search provider, Brave, runs a crawler that "does not advertise a differentiated user agent" and won't crawl a page "not crawlable by Googlebot", so your Googlebot rules decide Brave too. And Google's AI features add one switch outside robots.txt: Google's guide says "a site must be included in Search generative AI features in Search Console to be eligible for display in generative AI features on Google Search."
A robots.txt group can name several crawlers, so one file covers them all. Search crawlers and live fetchers get in. Training crawlers are your call: OpenAI and Anthropic run them separately from search, and Google uses Google-Extended for its other AI systems.
# AI search crawlers: they build the indexes the engines answer from
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
# Live fetches when a user asks about one of your pages
User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: Claude-User
Allow: /
# Google's AI features use Googlebot, and Brave follows Googlebot's rules
User-agent: Googlebot
Allow: /
# Training: Allow, or "Disallow: /" to opt out of training only
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Allow: /
Sitemap: https://www.example.com/sitemap.xmlrobots.txt is only half of access. Each company also publishes its crawlers' IP ranges (OpenAI, Perplexity, Anthropic) so your CDN or firewall can let the real ones through. Perplexity's docs give a Cloudflare rule that skips other security rules for its bots and add that "Bot Fight Mode can't be skipped with WAF custom rules". The full Cloudflare checklist is in step 1 of the main guide.
The engines don't agree on sources, so the page that wins one may not win another. From supereo's own runs:
| Engine | Answers | Cited a domain under 6 months old | Run-to-run flips |
|---|---|---|---|
| ChatGPT | 28 | 15 or more | 23% |
| Perplexity | 28 | 8 or more | 15% |
| Claude | 28 | 0 | 35% |
| Gemini | 28 | 0 | 37% |
| Google AI Mode | 28 | 0 | 48% |
| Google AI Overviews | 80 | 0 | 40% |
Young domains: supereo's November 7, 2026 run, six buyer questions; registration dates from the public domain registries, so the counts are lower bounds. Flips: the share of question-and-brand pairs that changed between recommended and not from one run to the next, supereo's October 16, 2026 variance run (20 questions, 10 runs each).
“Perplexity leans on Reddit/forums, ChatGPT repeats the same 2-3 listicle sites, Gemini's all over the place.”u/RealisticPass2388 on r/AISEOforBeginners · Aug 20, 2026 · Read the post
What helpsTrack each engine separately and read its sources: a page cited by one engine tells you little about the others.
What every engine needs, in one checklist:
Google's guide to its generative AI features has a list headed "Mythbusting generative AI search: what you don't need to do". It speaks for Google Search only, but it's the one engine that has said it:
The OpenAI, Perplexity and Anthropic pages we read ask for none of these either. All of them ask for crawler access.
| Engine | The company's own report | What it counts |
|---|---|---|
| Google AI Overviews and AI Mode | Search Console, Generative AI performance report | "how many times links to your site were shown", on all sites since August 31, 2026 |
| Microsoft Copilot and Bing | Bing Webmaster Tools, AI Performance | Total citations, average cited pages and grounding queries |
| ChatGPT | None in the OpenAI pages we read | Visits tagged utm_source=chatgpt.com, per OpenAI's archived publishers FAQ |
| Perplexity and Claude | None in the pages we read | Their answers show sources, so you read them yourself |
| All crawlers | Cloudflare AI Crawl Control | Requests from each AI crawler, allowed and unsuccessful |
| All six engines ours | supereo | How often answers to your buyers' questions recommend you, with its range |
Read November 8, 2026: Google, Bing, OpenAI FAQ, archived October 4, 2026, Cloudflare.
Logs need care. On our own Cloudflare zones in October 2026, none of 4,169 user-agent hits came from Perplexity-User's published IP ranges, so edge logs can't tell you when Perplexity reads a page, and most hits claiming to be ChatGPT-User came from outside OpenAI's list. Verify every crawler hit against its published IPs before you count it.
Allow each engine's search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot) in robots.txt and at your CDN, make sure each page answers its question in the first sentence in plain HTML, and get onto the roundups the engines cite. Then measure each engine separately.
Be in the index each engine reads: OpenAI's for ChatGPT, Perplexity's own, Brave and the others for Claude, Google's for AI Overviews and AI Mode. A page none of them can crawl can't be cited by any of them.
There's no shared ranking. ChatGPT cites vendor pages and docs most, Claude, Gemini and Perplexity cite roundups, and Google's AI surfaces lean on video and forums. Pick the engines your buyers use and build the page types each one cites.
The foundations are the same: crawlable, indexable pages that answer the question. Google says its AI features are "rooted in our core Search ranking and quality systems". The differences are extra crawlers to allow, the first 200 characters that ChatGPT's index keeps, and lists that matter more than links.
ChatGPT, if your buyers use it: it cited new domains in at least 15 of 28 answers in our November 7 run, and its recommendations were the second steadiest in our variance run. Perplexity second. Both reward the same first step: crawler access.
Each engine's crawler names, quotes and controls come from its own documentation, read November 8 and 9, 2026, linked under each table. Citation patterns come from supereo's October 2026 page panel and November 7, 2026 run, flips from its October 16, 2026 variance run, and log figures from our own Cloudflare zones. Full method.