Block GPTBot, Allow OAI-SearchBot: Safe robots.txt

The safe OpenAI rule

One wrong line in robots.txt can make your site vanish from an AI answer engine while you think you only opted out of training.

If your goal is to block OpenAI training use and keep ChatGPT search visibility, split the two OpenAI agents. Allow OAI-SearchBot. Block GPTBot. OpenAI documents these as separate controls. OAI-SearchBot is for ChatGPT search. GPTBot is the training crawler control.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Do not put a broad OpenAI rule above this unless you have tested how your parser and the crawler will read it. Keep the groups simple. Check the live file at https://yourdomain.com/robots.txt, then test the page you care about with the free AI visibility checker.

Which bots affect AI answers

Treat search crawlers and training controls as different jobs. Blocking the wrong one costs visibility.

• Allow OAI-SearchBot if you want pages eligible for ChatGPT search.
• Allow Claude-SearchBot if you want Claude to find and cite your pages.
• Allow PerplexityBot if you want Perplexity to crawl your pages for its answer index.
• Allow Googlebot if you want Google Search and Google AI Overviews visibility. Google AI Overviews ride the normal Google Search index. There is no separate AI Overviews crawler to allow.
• Allow Applebot if you want Apple search surfaces, including Apple Intelligence related search features, to reach your pages.

By contrast, GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended are training or opt-out controls. Blocking them does not remove you from live AI-search visibility by itself. Google-Extended and Applebot-Extended are robots-only control tokens, not separate crawlers you will see as their own user agent in normal logs.

Robots.txt is a stated policy. It is not proof of what a bot did. Some agents, including Perplexity-User and Bytespider, have been reported by others to ignore robots.txt or use confusing fetch behavior. Do not make log claims from the policy file alone.

Make the page readable

AI search engines can only cite what they can reach and understand.

Return a normal 200 page to the allowed crawlers. Avoid blocking them in your WAF, CDN, geo rules, bot rules, or login wall. Keep your canonical tag clean. Make sure the answer-worthy text is in the HTML response when possible.

Google documents JavaScript rendering for Googlebot. The other AI crawler docs do not all promise the same rendering path. That does not prove a specific bot cannot run JavaScript. It means client-side-only content is a risk. If the page matters, put the core text, title, date, author, and links in server-rendered HTML.

• Do not disallow the article path, assets needed to read it, or the whole site for search crawlers.
• Do not serve thin HTML that needs a large app bundle before any useful text appears.
• Do not use noindex on pages you want cited.
• Use plain internal links so crawlers can discover related pages, such as your hub at guides.

Email visibility is a different checklist

Robots rules help crawlers. They do nothing for inbox placement.

Mailbox providers such as Gmail and Outlook route mail using authentication, reputation, user complaints, content signals, and sending patterns. Start with DNS. SPF is a TXT record that authorizes servers for the SPF domain, usually the Return-Path domain. It helps DMARC only when that domain aligns with the visible From domain. Use one SPF record. Keep it under the 10 DNS-lookup limit from RFC 7208. End with ~all while you are still testing, then use -all when you are sure every real sender is listed.

DKIM signs a message with a key tied to a selector, such as selector1._domainkey.example.com. Turn signing on in your mail platform and publish the public key it gives you. DKIM can keep mail trusted through forwarding when the signed parts stay unchanged. Message rewrites can break the signature.

DMARC checks whether SPF or DKIM passes and aligns with the visible From domain. Start with p=none and a rua address so you can read aggregate reports. Move to p=quarantine or p=reject after real senders pass. If those XML reports are hard to read, use the free DMARC report reader before you enforce.

• Check MX records so receivers know where inbound mail for the domain should go.
• Check blocklists if spam placement started after a compromise, list upload, or sudden volume spike.
• Watch drift. DNS changes by a vendor or teammate can break SPF, DKIM, DMARC, or MX without warning.

InboxRadar can grade SPF, DKIM, DMARC, MX, and common blocklist issues from the free domain scorecard. Use it after you change robots.txt too, because a site can be visible to AI search and still fail the inbox.