Allow AI Search Crawlers, Block Training Bots

The line that can erase you from AI answers

One broad Disallow: / can keep your pages out of ChatGPT search, Claude search, Perplexity, Google AI Overviews, and Apple search surfaces. The fix is to allow the bots that power live answers and block the bots or control tokens used for model training.

Use this split. Allow OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude search, PerplexityBot for Perplexity, Googlebot for Google Search and AI Overviews, and Applebot for Apple search and Apple Intelligence features. If you disallow these crawlers in robots.txt, you remove yourself from that engine's live search or answer system.

Block training and opt-out controls separately: GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended. Blocking those does not remove you from live AI-search visibility. Google-Extended and Applebot-Extended are robots-only control tokens. They do not have their own separate crawl user agent.

• Keep search and answer crawlers allowed.
• Block training crawlers and opt-out tokens by name.
• Do not block Googlebot if you want Google Search or AI Overviews.
• Do not block Applebot if you want Apple search visibility.
• Check the live file with the AI visibility checker after you publish it.

A safe robots.txt pattern

Put the allow rules above the training blocks so a human can read the policy in one pass. Crawlers match their own user-agent group, so the main job is naming the right agents.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Applebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

This is a policy statement, not proof of behavior. A real crawler can be spoofed, a log line can lie, and some agents have been reported to ignore robots rules. Perplexity says Perplexity-User is for user-requested fetches and generally ignores robots.txt. Bytespider has also been reported to ignore robots.txt. Treat robots.txt as the public rule, then use server logs, reverse DNS, published IP lists, and WAF data when you need evidence.

If a firewall blocks these crawlers, the allow rule will not help. Check 403s, bot-fight modes, country blocks, JavaScript challenges, and rate limits. Also make sure your main text is present in the first HTML response. Google documents JavaScript rendering for Googlebot, but the other AI-search crawlers do not document the same rendering path. Client-side-only content is an undocumented risk.

Do not fix AI visibility and break email reach

The same kind of small policy error can hurt email. Mailbox providers read DNS records and make fast trust calls before a person ever sees your message.

SPF is a TXT record that says which servers may send mail for your domain. Publish one SPF record. Use your provider's includes, stay under the 10 DNS-lookup limit from RFC 7208, and avoid +all. Most senders use ~all while testing and move to -all only when every sender is known.

DKIM signs each message with a private key. The public key lives in DNS under a selector, such as selector1._domainkey.example.com. If the selector is missing, the key is wrong, or the platform is not signing, DKIM fails even when SPF passes.

DMARC ties SPF and DKIM to the visible From domain through alignment. Start with p=none and a rua address so you can read aggregate reports, then move to p=quarantine or p=reject after all real senders pass. If a report is hard to read, use the free DMARC report reader.

• Confirm MX records point to the service that receives your mail.
• Check public blocklists if mail was compromised or volume changed fast.
• Watch Gmail and Outlook errors for authentication, spam rate, and reputation signals.
• Run a free InboxRadar domain scorecard when DNS changes, then watch for drift.

Source check before you ship

Vendor names change. Keep the policy short, then compare it with the official docs during each crawl-policy review.

• OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot for training control.
• Anthropic documents controls for ClaudeBot, Claude-User, and Claude-SearchBot.
• Perplexity documents PerplexityBot as the search crawler.
• Google documents Googlebot and Google-Extended. Google-Extended is a control token for Gemini and Vertex AI training and grounding, not a separate crawler.
• Apple documents Applebot and Applebot-Extended. Applebot-Extended does not crawl pages on its own.
• Common Crawl documents CCBot and its robots.txt opt-out.
• For email authentication, use RFC 7208 for SPF, RFC 6376 for DKIM, RFC 7489 for DMARC, Google's email sender guidelines, and Microsoft's Outlook sender requirements.