OAI-SearchBot robots.txt: allow search, block training

Start with the right OpenAI rule

A single broad AI bot block can keep a useful page out of ChatGPT search. The fix is to separate search crawlers from training controls.

OAI-SearchBot is the OpenAI crawler for ChatGPT search. If you want a page to appear in ChatGPT search answers, allow this bot in robots.txt and make sure your firewall allows OpenAI's published searchbot IP ranges. GPTBot is different. It is a training crawler. Blocking GPTBot tells OpenAI not to use the crawled content for training its foundation models, but that block does not remove you from ChatGPT search by itself.

A clear starting point is:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Keep those groups separate. Do not place OAI-SearchBot under a blanket Disallow: / rule unless your goal is to stay out of ChatGPT search answers. After you edit robots.txt, fetch the exact file from your live domain and check the HTTP status. The policy only helps if crawlers can read it.

• Allow OAI-SearchBot for ChatGPT search visibility.
• Block GPTBot only when your policy is to opt out of training crawl.
• Do not copy AI-bot block lists without checking what each user agent does.

Know which bots affect AI answers

AI search has two buckets: bots that feed live answers, and controls for training or reuse. Mixing them up costs visibility.

The crawlers that decide whether you appear in AI answers are OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Googlebot for Google Search and AI Overviews, and Applebot for Apple Intelligence. Disallowing these in robots.txt removes you from that engine.

The training and opt-out controls are GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended. Blocking these does not affect live AI-search visibility. Google-Extended and Applebot-Extended are robots-only control tokens. They are not separate crawl user agents.

Robots.txt states your site policy. It is not proof of what a bot did. Perplexity says Perplexity-User generally ignores robots.txt because a user requested the fetch, and Bytespider has been reported to ignore robots.txt. Do not write that a bot obeyed or broke a rule unless your server logs and verified IP checks show the request. The free AI visibility checker can catch obvious robots.txt blocks before they cost you coverage.

Make the page readable to crawlers

Robots access is the first gate. The crawler still needs a useful page that it can fetch and read.

Put the title, answer, and main facts in server-rendered HTML when you can. Google documents JavaScript rendering for Googlebot. Other AI search crawlers do not all document the same rendering behavior, so client-side-only content is an avoidable risk. Treat plain HTML as the safer default, and do not claim that a specific bot cannot run JavaScript unless that vendor says so.

Use normal href links between related pages. If you maintain a guide hub, link to related guides from the article body. Keep robots.txt, noindex, canonical tags, and firewall rules pointed in the same direction. A page that returns 200 but hides the real answer behind a login wall is still a bad candidate for AI search.

• Serve the main article text in HTML before client scripts run.
• Return a 200 status for the canonical URL.
• Keep CSS, article routes, and canonical URLs crawlable.
• Use vendor docs and published IP ranges as the source of truth for bot checks.

Check email DNS at the same time

Robots.txt affects discovery. SPF, DKIM, DMARC, MX, and sender reputation affect whether your follow-up reaches the inbox.

SPF is a TXT record that lists which hosts may send mail for your domain. Publish one SPF record per domain, include every real sender, and stay within the RFC 7208 limit of 10 DNS lookups. ~all is a soft fail, often used while you are still finding senders. -all is a hard fail, better after the record is complete.

DKIM signs mail with a private key. The public key sits in DNS under a selector, such as selector._domainkey.example.com. If a mail platform rotates selectors or you remove an old DNS record too soon, messages can lose DKIM authentication.

DMARC checks whether SPF or DKIM aligns with the visible From domain. p=none monitors, p=quarantine asks receivers to treat failures as suspicious, and p=reject asks receivers to reject failures. Add a rua address so aggregate reports show who is sending as your domain. If the XML is hard to read, use the free DMARC report reader.

MX records route inbound mail. They do not authenticate outbound mail, but broken MX can stop replies and make a domain look neglected. Gmail and Outlook look at authentication, alignment, DNS health, complaint rates, spam-trap risk, list hygiene, volume spikes, blocklists, and user engagement. After DNS changes, run a quick domain scorecard because drift is common.