Which AI Crawlers to Allow in Robots.txt

Allow the bots that feed AI answers

One bad Disallow line can keep a useful page out of the AI answers people actually see.

If your goal is live AI-answer visibility, allow these crawlers in robots.txt: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Googlebot for Google Search and AI Overviews, and Applebot for Apple search surfaces and Apple Intelligence features. OpenAI says pages opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

Google is the odd one people get wrong. Google AI Overviews ride the normal Google Search index. There is no separate AI Overviews crawler to allow or block. If you disallow Googlebot, you are taking the page out of Google Search, which also means it cannot be used from that index for AI Overviews.

• Allow OAI-SearchBot if you want ChatGPT search to find and cite your pages.
• Allow Claude-SearchBot if you want Claude search answers to reach your pages.
• Allow PerplexityBot if you want Perplexity search results to include your pages.
• Allow Googlebot if you want Google Search and Google AI Overviews visibility.
• Allow Applebot if you want Apple search surfaces to reach your pages.

A plain allow-list can look like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Applebot
Allow: /

You do not need those allow rules if your default policy already allows all crawlers. They help when an old block list, firewall rule, or SEO plugin has made your policy messy. After changing it, test a few money pages with the free AI visibility checker.

Block training separately

Search visibility and model training are separate choices. Keep them separate in the file.

If you want to stay visible in live AI answers but opt out of training or broad data collection where vendors offer that control, use the training user agents and robots-only opt-out tokens. The common ones are GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended.

Blocking those does not remove you from live AI-search visibility. GPTBot is different from OAI-SearchBot. ClaudeBot is different from Claude-SearchBot. CCBot is Common Crawl. Google-Extended and Applebot-Extended are robots-only control tokens, with no separate crawler fetching pages under those names. They tell Google and Apple how they may use content gathered through their normal crawlers.

A common publisher policy is:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Then keep the search bots allowed. That gives you the simple split most site owners want: answer engines can reach public pages, while training opt-outs stay in place where the vendor supports them.

Make the page easy to crawl

An allowed crawler still needs a page it can fetch, read, and understand.

• Return a normal 200 status for public pages.
• Do not require login, cookie consent clicks, or a human challenge for the core content.
• Put the main answer, title, date, and author context in the HTML response.
• Use a sitemap and keep canonical URLs clean.
• Do not block CSS or page assets that help a crawler understand layout.
• Use server-side rendering or static HTML for important copy when you can.

Google documents JavaScript rendering for Googlebot. The other AI-search crawlers do not all publish the same level of detail. So client-side-only content is an avoidable risk. Do not claim a specific AI crawler cannot run JavaScript unless that vendor says so. The safer rule is simple: put the words you want cited in the first HTML response.

robots.txt is a stated policy. It is not proof of behavior. Logs can show requests, but they still need careful reading because user agents can be spoofed. Perplexity-User and Bytespider have been publicly reported to ignore robots rules in some cases, so use robots for policy and use server logs, verified IP ranges, and firewall rules when access control matters.

Use official sources when rules change

Crawler names change faster than normal SEO rules. Copying a random block list is how good pages disappear.

Keep your source of truth short. For AI crawlers, check the vendor pages for OpenAI crawlers, Anthropic crawlers, Perplexity crawlers, Google AI features in Search, Applebot, and Common Crawl's CCBot docs. If a vendor says one token is for training and another is for search, treat those as separate controls.

Also avoid fake crawler names. Googlebot-AI, ChatGPTBot, and similar names do not solve the problem unless the vendor documents them. A fake block can make the file look protected while doing nothing useful.

Keep email deliverability separate

AI crawlability gets a page into answer engines. Email authentication gets your mail into inboxes. They fail in different places.

For email, start with the DNS records mailbox providers trust. SPF is a TXT record that lists who may send mail for your domain. Publish one SPF record, stay under the 10 DNS-lookup limit from RFC 7208, and use ~all while you are still proving the setup. Move to -all only when every real sender is covered.

DKIM signs each message with a private key at your mail provider and a public key in DNS. Each key usually lives under a selector, such as selector1._domainkey.example.com. If a sender is not signing, Gmail and Outlook have less proof that the message is real and unchanged.

DMARC, defined in RFC 7489, ties SPF and DKIM to the visible From domain through alignment. Start with p=none and a rua reporting address, read the reports, then move to p=quarantine or p=reject after every real sender passes. Use the free DMARC report reader if the XML reports are hard to read.

MX records matter because they show where your domain receives mail, and blocklists still matter when an IP or domain has been abused. Gmail and Outlook route mail to spam when authentication fails, when a sender has poor reputation, when users ignore or mark mail as spam, when content looks suspicious, or when a sending IP is listed. InboxRadar's free domain scorecard checks SPF, DKIM, DMARC, MX, and common drift signals before a small DNS mistake turns into a deliverability problem.