Claude recommends from what it already knows and from what its web search finds. Let Claude-SearchBot and Claude-User in, be in Brave Search, whose crawler follows your Googlebot rules, and get onto the roundups Claude cites.
Our pick
Want to know whether Claude recommends you? supereo ours is our pick: it tracks Claude on every plan alongside ChatGPT, Perplexity, Gemini and Google's AI Mode and AI Overviews, with a range on every rate, and tells you what to change when Claude names someone else. Claude is in the free audit too: 20 of your buyers' questions, every answer shown, no card. Run the free audit.
Claude answers some questions from memory and searches the web for others. Anthropic's help page says that when you ask about topics that benefit from current information, "Claude invokes a search tool", and that in the new Claude experience "there's no web search toggle. Claude searches the web when it helps." So two things decide whether Claude names you: what it learned in training, and what its search turns up on the day.
“I have lost the ability to disable web search in the app.”u/SpiritFederation on r/claude · Sep 18, 2026 · Read the post
Worth knowingAnthropic's help page confirms it: in the new Claude experience there's no toggle, and Claude searches when it decides it helps. For your pages, that means more answers that can cite you.
| Crawler | Anthropic's description | What blocking it does |
|---|---|---|
Claude-SearchBot | "navigates the web to improve search result quality for users" | "prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results" |
Claude-User | "When individuals ask questions to Claude, it may access websites using a Claude-User agent." | "prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search" |
ClaudeBot | Collects web content "that could potentially contribute to" model training | "signals that the site's future materials should be excluded from our AI model training datasets" |
Source: Anthropic, Does Anthropic crawl data from the web, read November 8, 2026. Anthropic's crawler IP list is at claude.com/crawling/bots.json, dated October 20, 2026.
Three details from the same page matter for access. Anthropic says opting out "requires modifying the robots.txt file" and that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out". Its bots "will not attempt to bypass CAPTCHAs", so a bot challenge such as Cloudflare's Bot Fight Mode keeps them out. And since October 30, 2026, Anthropic has begun recognizing a separate machine-readable "no AI training" preference; rules that name ClaudeBot take priority over it.
# Claude's search index and live fetches
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /
# Training: Allow, or "Disallow: /" to opt out of training only.
# What Claude knows without searching comes from training.
User-agent: ClaudeBot
Allow: /
# Brave Search's crawler has no user agent of its own and
# won't crawl what Googlebot can't, so keep Googlebot open
User-agent: Googlebot
Allow: /Anthropic's subprocessor list, read November 8, 2026, names three companies for web search: Brave Search (all products), TurboPuffer and Parallel Web Systems (all products except Claude for Government). Image results are "powered by Bing", per Anthropic's help page. Brave is the one you can act on directly. When Brave Search was added to that list in March 2025, Simon Willison checked a Claude search against Brave and found "the ten results it showed as citations were an exact match for that search on Brave".
Brave's crawler has a rule worth knowing: it "does not advertise a differentiated user agent because we must avoid discrimination from websites that allow only Google to crawl them. However, if a domain or page is not crawlable by Googlebot, then Brave Search's bot will not crawl it either." So:
Anthropic has written about the bias it designed against. In its post on building Claude's research system: "human testers noticed that our early agents consistently chose SEO-optimized content farms over authoritative but less highly-ranked sources like academic PDFs or personal blogs. Adding source quality heuristics to our prompts helped resolve this issue." A page that reads like a primary source, with a method, real numbers and dates, fits what Claude was tuned to pick.
Claude's citations are short. Anthropic's API docs say each citation carries "Up to 150 characters of the cited content", and each search result carries a page_age, "When the site was last updated". Write the claim you want quoted as one plain sentence near the top of its section, and keep your updated dates accurate.
For answers Claude gives without searching, it relies on training. Anthropic's ClaudeBot collects content that "could potentially contribute to" that training, so blocking it keeps your future pages out. Beyond that, being known is slow work: the roundups, directories and articles in your category that name you are what a future model learns your name from. One founder describes a faster route for a product that works as a tool:
“The listing is what did it. When someone asks Claude about flights or hotels, it recommends us right next to Expedia and Booking.com.”u/ElevatorFriendly648 on r/mcp · Nov 4, 2026 · Read the post · The poster says he founded SlickTrip, the product in the post
Let Claude-SearchBot and Claude-User crawl your site, keep Googlebot open so Brave Search can index you, get onto the roundups Claude cites for your category, and publish pages that read as primary sources with the answer first. Then check how often Claude recommends you over many runs.
Anthropic's subprocessor list names Brave Search, TurboPuffer and Parallel Web Systems for web search, and its help page says image results are powered by Bing. It doesn't name Google.
Anthropic says its bots honor robots.txt and the Crawl-delay extension, and that robots.txt is the reliable way to opt out; blocking IPs "may not work correctly". Brave's crawler follows your Googlebot rules.
Not from search. ClaudeBot is the training crawler; Claude-SearchBot and Claude-User are the ones that affect search. Blocking ClaudeBot does keep your future pages out of training, which is where Claude's no-search answers come from.
Ask it your buyers' questions with web search on, several times, and read the citations. Claude skips search on some answers, so note which answers came from memory. supereo's free audit runs Claude with the other engines.
Anthropic's statements come from its crawler and web search help pages, its API docs, its subprocessor list and its engineering blog; Brave's from its crawler help page; all read November 8, 2026. Claude's behavior figures come from supereo's November 7, 2026 run, October 2026 panel and October 16 variance run, which asked Claude through its API with web search on. Quoted posts are word for word. Full method.