Google AI Overviews and robots.txt: What to Allow

Google AI Overviews use Googlebot

One bad robots.txt rule can take a page out of Google Search. That can also take it out of Google AI Overviews.

There is no separate Google AI Overviews crawler to allow or block. Google AI Overviews ride the normal Search index, and Googlebot is the crawler that matters. If you disallow Googlebot, Google loses the normal crawl path for that page. That means the page can lose its path into Search and AI Overviews.

Google-Extended is a different control. It is a robots.txt token for model training and related AI data use. It has no separate crawl user-agent. Blocking Google-Extended does not remove a page from Google Search or Google AI Overviews. If your goal is AI Overviews visibility, keep useful public pages open to Googlebot.

The bots that affect AI answers

Live AI answers and model training use different controls. Keep those two decisions separate.

• Allow Googlebot for Google Search and Google AI Overviews.
• Allow OAI-SearchBot for ChatGPT search answers.
• Allow Claude-SearchBot for Claude search answers.
• Allow PerplexityBot for Perplexity answers and search results.
• Allow Applebot for Apple search surfaces and Apple Intelligence answers that use web context.

Disallowing one of those search crawlers removes the normal crawl path for that engine. That is the robots.txt choice that can affect whether your public page is found and used in live AI answers.

The training and opt-out controls are separate: GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended. Blocking those does not block live AI-search visibility by itself. Google-Extended and Applebot-Extended are robots-only control tokens with no separate crawl user-agent. For source-of-truth details, check the vendor docs from OpenAI, Anthropic, Perplexity, Google, Apple, and Common Crawl. To check the policy your own site exposes, use the free AI visibility checker.

A practical robots.txt pattern

Open the search crawlers you want. Block training controls only if that matches your policy.

A common policy looks like this:

User-agent: Googlebot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

This is an example, not a rule for every site. Some publishers allow training crawlers. Some block them. The key point is simple: Googlebot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Applebot are the crawlers tied to live discovery and answers.

Robots.txt is a stated policy, not proof of behavior. Do not use it as your only evidence of what a crawler did. Perplexity-User and Bytespider have been reported to ignore robots.txt in some cases, so be careful with claims about actual bot behavior. If you need proof, use server logs, verified IP ranges, reverse DNS, and the vendor's own crawler docs.

Make the page readable to crawlers

An allowed crawler still needs a page it can fetch, parse, and trust.

• Return a real 200 page for public content.
• Put the main text in the first HTML response when you can.
• Do not put the only useful copy behind login walls, cookie gates, or blocked scripts.
• Use normal links with href attributes so crawlers can find related pages.
• Check that your firewall does not challenge the crawlers you mean to allow.

Google documents JavaScript rendering for Googlebot. The other AI-search crawlers do not all document the same rendering behavior. That makes client-side-only content a risk, not a known failure for every bot. For important pages, serve the article, product copy, pricing, and support text as plain HTML or server-rendered content. You do not need a special AI page. You need the same clear page a person would want.

Email domains have their own trust layer

Crawlability helps people and AI systems find your site. Email authentication helps your mail avoid spam.

For InboxRadar users, this often hits the same domain. Gmail and Outlook look at authentication, reputation, complaints, blocklists, message quality, and sending patterns when they decide inbox or spam. MX records route inbound mail. SPF, DKIM, and DMARC help receivers decide whether outbound mail is really tied to your domain.

SPF lists the servers allowed to send for your domain. Publish one SPF TXT record for a sending domain, avoid +all, use ~all while testing, move to -all only when every real sender passes, and stay within the SPF 10-DNS-lookup limit in RFC 7208.

DKIM signs mail with a private key and lets receivers find the public key in DNS using the selector. RFC 6376 is the base DKIM source. DMARC, defined in RFC 7489, ties SPF and DKIM to the visible From domain through alignment. Start with p=none and a rua address so you can read aggregate reports. Move to p=quarantine or p=reject after the real senders pass. For provider rules, check the Gmail email sender guidelines and Microsoft's Outlook sender requirements. To inspect your own domain, run the free InboxRadar domain check. To read a DMARC aggregate file, use the free DMARC report reader.