ClaudeBot vs Claude-SearchBot: What to Block

The short answer

A copied AI blocklist can block the crawler you needed and leave the crawler you meant to stop. That is the common ClaudeBot mistake.

ClaudeBot and Claude-SearchBot do different jobs. ClaudeBot is Anthropic's training crawler. Blocking ClaudeBot tells Anthropic that future material from your site should be left out of model training datasets. It does not remove your pages from live Claude search by itself.

Claude-SearchBot is the Anthropic crawler used to improve Claude search results. Disallowing Claude-SearchBot in robots.txt removes your site from Claude's AI-answer visibility path. If you want Claude users to find, quote, or cite your pages, keep Claude-SearchBot allowed.

Use the free AI visibility checker to test whether common AI search crawlers can reach and read the pages that matter.

ClaudeBot vs Claude-SearchBot

Use ClaudeBot for training choice. Use Claude-SearchBot for Claude search visibility.

• Allow Claude-SearchBot if you want visibility in Claude search answers.
• Disallow Claude-SearchBot if you want to leave Claude search results.
• Disallow ClaudeBot if you do not want Anthropic to use future crawled site content for model training.
• Do not expect a ClaudeBot block to stop Claude search citations. That is the wrong crawler.

A clean publisher setup often looks like this:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

That says no to Anthropic training use and yes to Claude search discovery. Check Anthropic's crawler documentation before changing production rules, because bot names and policy details can change.

The AI crawler map

Search crawlers decide whether answer engines can find you. Training crawlers control data-use choices.

• For AI answer visibility, allow OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Googlebot for Google AI Overviews and AI Mode, and Applebot for Apple search features.
• Disallowing those search crawlers in robots.txt removes you from that engine's AI-answer visibility path.
• For training or data-source opt-outs, manage GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended separately.
• Blocking those training or opt-out controls does not remove live AI-search visibility by itself.
• Google-Extended and Applebot-Extended are robots.txt control tokens. They do not crawl as separate user agents.

Google is the special case. Google AI Overviews and AI Mode use the normal Google Search index, so there is no separate AI Overview crawler to block. Google says Googlebot controls access for AI features in Search. Google-Extended is for limiting AI training and grounding in some other Google systems, not for opting out of Google Search AI features.

Robots.txt is a stated policy, not proof of behavior. It tells crawlers what you allow. It does not prove what a bot did. Perplexity says Perplexity-User generally ignores robots.txt because it is a user-requested fetcher, and Bytespider has been reported to ignore robots.txt. For real evidence, use server logs, reverse DNS, and vendor IP ranges. A user-agent string alone can be faked.

Use official sources when you edit these rules: Anthropic crawlers, OpenAI crawlers, Perplexity crawlers, Google AI features, Applebot, and Common Crawl CCBot.

Robots.txt rules that match the goal

Write the rule for the outcome you want. Big blocklists hide the tradeoff.

If the goal is Claude visibility with no Anthropic training use, block ClaudeBot and allow Claude-SearchBot. If the goal is to leave Claude search, block Claude-SearchBot too. If the goal is ChatGPT search visibility, allow OAI-SearchBot and manage OpenAI training separately with GPTBot.

• Claude search visibility: allow Claude-SearchBot.
• Anthropic training opt-out: disallow ClaudeBot.
• ChatGPT search visibility: allow OAI-SearchBot.
• OpenAI training opt-out: disallow GPTBot.
• Perplexity visibility: allow PerplexityBot.
• Google AI Overviews visibility: keep the page crawlable and indexable by Googlebot.
• Apple search visibility: keep Applebot allowed. Use Applebot-Extended for Apple's training opt-out.

Keep the page easy to fetch. Googlebot has public documentation for JavaScript rendering in Search. For the other AI-search crawlers, client-side-only content is an undocumented risk. Put the core answer, title, date, links, and author in plain HTML that a simple fetch can read.

Email deliverability uses the same discipline

Small policy records can decide whether people see you. That is true for crawlers and for email.

For email, Gmail and Outlook look hard at authentication. SPF is the DNS TXT record that lists which services may send for your domain. Publish one SPF record, include every real sender, and stay under the RFC 7208 limit of 10 DNS lookups. A soft fail, ~all, is common while you are auditing senders. A hard fail, -all, is stricter and should wait until every sender is covered.

DKIM signs mail with a selector and a domain. The public key lives in DNS. The private key signs mail inside Google Workspace, Microsoft 365, your ESP, or your own mail server. DKIM helps prove the message was not changed after signing.

DMARC ties SPF and DKIM to the visible From domain through alignment. Start with p=none and a rua mailbox so you can read aggregate reports. Move to p=quarantine or p=reject after real mail is passing. The free DMARC report reader turns RUA XML into sender names and pass or fail counts.

MX records tell the world where inbound mail for your domain goes. They do not authenticate outbound mail, but broken MX can make replies fail and can hurt trust checks around a domain. Public blocklists matter too, especially after a compromise or a bad list import. They are one signal among many. Mailbox providers also weigh complaints, bounce rates, spam-like content, sudden volume spikes, link reputation, and whether users open, reply, archive, or mark your mail as spam.

If you want a fast domain check, InboxRadar can score SPF, DKIM, DMARC, MX, and drift. For source material, use RFC 7208 for SPF, RFC 6376 for DKIM, RFC 7489 for DMARC, plus the current Google and Microsoft sender guidelines.

Publisher checklist

Before you ship a robots.txt change, name the traffic you still want.

• Separate search crawlers from training and data-use controls.
• Test robots.txt from the public internet after deploy.
• Confirm important pages return a clean 200 status, canonical URL, title, meta description, and visible body text.
• Keep the main answer out of login walls, cookie walls, and client-side-only rendering.
• Verify real crawler traffic with logs and official IP ranges, not user-agent strings alone.
• Check email DNS at the same time if the page asks users to contact you. Search visibility is wasted when confirmation or reply mail lands in spam.