Applebot, robots.txt, and Apple Intelligence

The short answer

One bad robots.txt rule can keep a useful page out of Apple Intelligence answers. The line to check first is Applebot, not Applebot-Extended.

Apple says Applebot powers search across Spotlight, Siri, Safari, and other Apple surfaces. Apple also says Applebot-crawled data can be used as fresh context when AI models generate output in Apple products and services. If you want a public page eligible for Apple Intelligence answer context, Applebot needs to be allowed to fetch it.

Applebot-Extended is different. Apple documents it as a robots.txt control for whether content already crawled by Applebot may be used to train Apple foundation models. Applebot-Extended does not crawl webpages. Pages that disallow Applebot-Extended can still appear in Apple search results when Applebot itself is allowed.

The robots.txt rule to check

Read the rule group that applies to Applebot first. Then check any wider wildcard rule that may catch Applebot by accident.

• Allow Applebot on public pages that you want eligible for Apple search and Apple Intelligence answer context.
• Use Applebot-Extended only when you want to opt out of Apple foundation model training use.
• Use Applebot nosnippet controls if you want a page discoverable by Apple but do not want its text used as answer context.
• Do not treat robots.txt as proof of crawler behavior. It is a public policy file. Logs still need bot verification before you draw conclusions.
• Keep important copy in HTML text. Only Googlebot documents JavaScript rendering for this AI search visibility check. For the other AI crawlers, client-side-only text is an undocumented risk, but do not claim a specific bot cannot run JavaScript unless its docs say so.

A common setup is to allow Applebot and disallow Applebot-Extended. That keeps the page eligible for Apple discovery while saying no to training use. If you block Applebot with Disallow: /, you are telling Apple not to crawl those URLs for the search surfaces that feed Apple visibility.

You can test this kind of issue with the free AI visibility checker. It checks whether live AI search crawlers can reach and read the page, which is the part that affects visibility.

Live search crawlers are not training controls

The name on the robots.txt line matters because each vendor split these jobs in a different way.

The live AI-search visibility crawlers to know are OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude search, PerplexityBot for Perplexity, Googlebot for Google AI Overviews through the normal Search index, and Applebot for Apple Intelligence. Blocking these removes your pages from that engine's answer source set.

Training and opt-out controls are a separate bucket: GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended. Blocking those controls tells the vendor or dataset operator not to use content for training or related model uses as documented. It does not, by itself, block live AI search visibility. Google-Extended and Applebot-Extended are robots.txt control tokens with no separate page-fetching user agent.

Perplexity documents Perplexity-User for user-triggered fetches and says those fetches generally ignore robots.txt. Reports also exist about Bytespider ignoring robots.txt. Treat robots.txt as your stated rule, then verify real requests with server logs, IP ranges, and reverse DNS where the vendor provides them.

Why this belongs in the same audit as email deliverability

Search bots and mailbox providers both judge your domain by small DNS and access details. A typo can cost traffic or inbox placement.

For email, SPF is the TXT record that lists who may send for your domain. Keep one SPF record, stay under the 10 DNS lookup limit in RFC 7208, and use ~all while you are still finding senders. Move toward -all only when every real sender is covered.

DKIM signs each message with a private key and a DNS public key under a selector, as defined in RFC 6376. If a sender uses your domain but does not sign with your domain or a domain that aligns, DMARC can still fail even when SPF passes.

DMARC, from RFC 7489, checks alignment with the visible From domain. Start with p=none and a rua mailbox so you can see who is sending. Then move to quarantine or reject when the reports are clean. If a RUA file is hard to read, use the free DMARC report reader before changing policy.

MX records tell the world where to deliver replies and inbound mail. Blocklists, complaint rate, bounce rate, bad content, missing unsubscribe headers, and weak engagement still affect Gmail and Outlook placement. Google and Microsoft both publish sender rules that point back to SPF, DKIM, DMARC, alignment, and low abuse signals. InboxRadar's free domain scorecard is useful here because it checks the mail side and flags drift before it turns into spam placement.

A practical audit order

Start with the controls that can fully block access, then look at the weaker signals.

• Fetch /robots.txt and confirm Applebot is not blocked by its own group or by User-agent: *.
• Confirm Applebot-Extended matches your training policy, not your live visibility policy.
• Check OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Googlebot while you are already in robots.txt.
• Fetch the URL without logged-in state and make sure the main answer text is present in the HTML response.
• Check SPF, DKIM, DMARC, MX, and blocklist status for the same domain. Bad mail trust will not block Applebot, but it can hurt sales replies after AI search sends a lead.
• Read the official docs before making broad blocks. Start with Applebot, OpenAI crawlers, Anthropic crawlers, Perplexity crawlers, Google AI features, Common Crawl CCBot, RFC 7208, RFC 6376, RFC 7489, Google sender guidelines, and Microsoft sender guidance.

For related crawlability and sender setup guides, see the InboxRadar article library.