Applebot vs Applebot-Extended: What to Allow

The short answer

One robots.txt rule can hide you from Apple search. The other only opts your content out of Apple model training.

Applebot is Apple's web crawler. Apple says Applebot supports search technology across Apple products, including Spotlight, Siri, and Safari. Apple also says crawled data can provide current context when AI models generate output for display in Apple products and services. If you disallow Applebot, you are telling Apple not to crawl the site. That can remove the page from Apple search and Apple Intelligence answer paths.

Applebot-Extended is the training control. Apple calls it a secondary user agent, but its own Applebot page says Applebot-Extended does not crawl webpages. It only tells Apple how content crawled by Applebot may be used. Blocking Applebot-Extended does not block Applebot, and pages that disallow Applebot-Extended can still appear in Apple search results.

• Allow Applebot if you want Apple search and Apple Intelligence visibility.
• Disallow Applebot-Extended if you want to opt out of Apple foundation-model training.
• Do not look for Applebot-Extended page crawls in server logs. Check your robots.txt policy.

The safe robots.txt pattern

For most publishers, the clean setup is to allow search crawlers and block training controls.

Use this if you want Apple to find your pages, but do not want Apple to use the content for training its foundation models:

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Disallow: /

That policy allows Applebot to crawl for search and answer context. It also sets the Applebot-Extended opt-out for training use. If you block both Applebot and Applebot-Extended, you are blocking Apple crawl access. If you allow both, you are allowing Applebot crawl access and you are not setting the Apple training opt-out.

The same split matters across AI search. The crawlers that decide whether a site can appear in live AI answers are OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Googlebot for Google AI Overviews, and Applebot for Apple Intelligence. Disallowing those crawlers in robots.txt removes you from that engine.

Training and opt-out controls are separate: GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended. Blocking them does not remove you from live AI-search visibility. Google AI Overviews use the normal Google Search index, so there is no separate Google AI Overviews crawler to block. Google-Extended and Applebot-Extended are robots.txt control tokens, not separate page-crawling user agents.

If you want to check whether the right AI search crawlers can reach a page, use the free AI visibility checker. It catches the common mistake: blocking the search crawler when you only meant to block training use.

What robots.txt proves

Robots.txt is a policy file. It is not a security wall, and it is not proof that a bot behaved.

Good crawlers read robots.txt and follow the rules they document. Other bots may parse the file differently, use a spoofed user-agent string, or ignore it. Treat robots.txt as your stated policy. Treat access logs as evidence of requests. Do not use either one alone to claim what a bot actually did.

Perplexity says Perplexity-User generally ignores robots.txt because it is used for user-requested fetches. Bytespider has also been reported to ignore robots.txt. Say that these are reported behaviors. Do not turn those reports into a claim that every request from those names broke your policy.

JavaScript is a crawlability risk. Google documents Googlebot rendering JavaScript. The other AI search crawlers do not all publish the same rendering path. That does not prove they cannot run JavaScript, but it means client-side-only content may be missed. Put important copy, pricing, author names, product proof, and links in the initial HTML when AI visibility matters.

For policy decisions, use the official sources: Apple's Applebot page, OpenAI's crawler docs, Anthropic's crawler docs, Perplexity's crawler docs, Google Search Central for Googlebot, Google-Extended, AI Overviews, and JavaScript rendering, plus Common Crawl for CCBot.

How this connects to email deliverability

Crawler rules and email DNS records have the same failure mode: one small text record can change who trusts you.

For receiving mail, MX records point to the servers that accept mail for the domain. For sending mail, SPF authorizes the servers allowed to send for the SPF domain. Publish one SPF record at each SPF domain. Multiple SPF records at the same name can break evaluation. SPF also has a hard DNS lookup limit in RFC 7208: no more than 10 lookup-causing mechanisms or modifiers during evaluation. Use ~all while you are still finding real senders. Use -all after every real sender is covered.

DKIM signs mail with a private key at the sender. DNS publishes the matching public key under a selector, such as selector1._domainkey.example.com. Receivers use that selector to find the right public key and verify the signature. Good senders sign normal mail with DKIM because forwarding often breaks SPF, while DKIM can survive if the message is not changed in transit.

DMARC checks whether SPF or DKIM passes and aligns with the visible From domain. Start with p=none so you can read reports without asking receivers to quarantine or reject mail. Move to p=quarantine or p=reject after your real senders pass and align. Add a rua address for aggregate reports. If the XML is hard to read, the free DMARC report reader makes it easier to spot failures.

Gmail and Outlook do not send mail to the inbox just because SPF, DKIM, and DMARC pass. They also look at sender reputation, complaint rate, sending history, volume spikes, content, links, blocklists, and user behavior. Broken authentication makes spam placement or rejection more likely, but clean authentication is only the floor. You can check the basics with the free domain scorecard, then review related guides at InboxRadar articles.