Allow Claude-SearchBot but Block ClaudeBot in robots.txt
Want to know if AI search engines can actually reach and read your site? Check it free. Run the AI visibility check.
Two Anthropic bots, two very different jobs
Plenty of sites blocked ClaudeBot to opt out of AI training, then noticed they had also vanished from Claude's live search answers. The cause is almost always one robots.txt mistake, and it takes two minutes to fix.
Anthropic runs separate crawlers. ClaudeBot is the training crawler. Blocking it is an opt-out from model training, and it does not affect whether Claude can cite you. Claude-SearchBot is the crawler behind Claude's live search. Disallow it and Claude stops surfacing your pages in answers. The same split exists elsewhere: GPTBot, CCBot, Google-Extended, and Applebot-Extended are training or opt-out controls, while OAI-SearchBot, PerplexityBot, Googlebot, and Applebot decide whether you show up in ChatGPT search, Perplexity, Google AI Overviews, and Apple Intelligence. There is no separate opt-out crawler for AI Overviews, they ride the normal Google Search index.
The robots.txt rules that do what you actually want
robots.txt works in groups. Each bot obeys the most specific group that names it, and falls back to the User-agent: * group if nothing does. That fallback is where people get burned.
- Add a group for the search bot:
User-agent: Claude-SearchBotthenAllow: / - Add a separate group for the training bot:
User-agent: ClaudeBotthenDisallow: / - Check your
User-agent: *group. If it has a broadDisallow, Claude-SearchBot needs its own explicit group or it inherits the block. - Do not merge the two bots into one group. A group only applies to the user agents named in it, so a rule written for ClaudeBot says nothing about Claude-SearchBot.
- Watch out for Google-Extended and Applebot-Extended. They are robots-only control tokens with no crawler of their own, so listing them changes training use, not crawling.
One honest caveat: robots.txt is your stated policy, not proof of behavior. Well-run crawlers follow it, but some agents, like Perplexity-User and Bytespider, are reported to ignore it. Your server logs tell you what actually fetched pages. Anthropic publishes its crawler user agents and IP ranges in its official docs, and that is the source to trust over any blog post, including this one.
After the edit, confirm the bots can actually read you
A correct robots.txt is step one. AI crawlers still need to reach and parse your pages.
Only Googlebot documents JavaScript rendering. If your content only appears after client-side JS runs, treat it as at risk of being invisible to the other AI crawlers, since none of them document rendering. Server-rendered HTML is the safe path. You can run our free AI visibility checker to see how your robots.txt treats each major AI crawler and whether your key pages are reachable. And since being cited is only useful if your outreach lands, the free domain scorecard checks your SPF, DKIM, and DMARC at the same time.
FAQ
Does blocking ClaudeBot remove me from Claude's search answers?
No. ClaudeBot is the training crawler. Claude's live answers rely on Claude-SearchBot, so you only disappear from answers if you disallow Claude-SearchBot itself, or if it falls into a broad wildcard block.
Can I allow training but block Claude's search instead?
Yes, just flip the groups: allow ClaudeBot and disallow Claude-SearchBot. Most sites want the opposite, since search visibility drives traffic while training crawl mostly does not.
How do I verify my rules are working?
Fetch yoursite.com/robots.txt and read the groups by user agent, then check server logs for what actually crawled. Remember robots.txt states policy, it does not prove behavior. For a quick automated read, the AI visibility checker flags conflicts between your wildcard rules and each named AI bot.