PerplexityBot Robots.txt Example
Want to know if AI search engines can actually reach and read your site? Check it free. Run the AI visibility check.
PerplexityBot robots.txt example
A PerplexityBot block is a visibility choice. It can take your pages out of Perplexity answers, but it cannot prove what any crawler did.
To block PerplexityBot from the whole site, put this in the root robots.txt file:
User-agent: PerplexityBot
Disallow: /To allow PerplexityBot, leave it unblocked. An empty Disallow: line is valid robots.txt, but if you have no policy to state, no PerplexityBot group is easier to audit.
Robots.txt is stated policy. It tells compliant crawlers what you ask them not to fetch. It is not access control, a log, or proof of crawler behavior. If you need to know whether PerplexityBot fetched a URL, check server logs, the documented user-agent, vendor IP data when it exists, and the request time. Perplexity says Perplexity-User generally ignores robots.txt for user-requested fetches. Bytespider has also been reported to ignore robots.txt, so do not use robots.txt alone to claim what either bot did.
Which AI crawlers affect answers?
The user-agent name matters. Training controls and live answer controls do different jobs.
The crawlers that decide whether you appear in AI answers are OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Googlebot for Google, and Applebot for Apple Intelligence. Google AI Overviews use the normal Google Search index. There is no separate opt-out crawler for them. Disallowing these live-search crawlers in robots.txt removes you from that engine.
GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended are training or opt-out controls. Blocking them does not affect live AI-search visibility. Google-Extended and Applebot-Extended are robots-only control tokens with no separate crawl user-agent.
Only Googlebot documents JavaScript rendering. If key copy, prices, docs, or product pages appear only after client-side JavaScript runs, treat that as an undocumented risk for other AI crawlers. Server-render the parts you need crawlers to read. You can check basic AI-search reachability with the free AI visibility checker.
Use vendor docs as the record when names change: OpenAI crawlers, Anthropic crawlers, Perplexity crawlers, Google crawlers, Applebot, and CCBot.
Email DNS needs the same proof habit
A DNS record states a policy too. Real mail still needs testing against real receivers.
SPF says which services may send for your domain. Keep one SPF record, watch every include and redirect, and stay under the 10 DNS lookup limit from RFC 7208. A soft fail, ~all, tells receivers the source is probably not allowed. A hard fail, -all, tells them it is not allowed. Move to hard fail only after you know every real sender is covered.
DKIM signs mail with a selector, such as selector1._domainkey.example.com. Receivers use the public key in DNS to verify that the message was signed and was not changed after signing. The base DKIM spec is RFC 6376.
DMARC checks alignment. SPF or DKIM has to pass and align with the visible From domain. DMARC policy can be p=none, p=quarantine, or p=reject. Start with reports, fix sources, then tighten. RFC 7489 is the older DMARC spec. The current core protocol is RFC 9989, with aggregate reporting in RFC 9990 and failure reporting in RFC 9991. A rua address receives aggregate reports. The free DMARC report reader helps turn those XML files into sources and failure patterns.
MX records decide where inbound mail for your domain is delivered. They do not make outbound mail trusted. Gmail, Outlook, and other mailbox providers look at authentication, alignment, complaint rate, bounce rate, sending history, volume spikes, link reputation, blocklists, and user engagement. A domain can pass SPF, DKIM, and DMARC and still land in spam if reputation is poor. The free domain scorecard is a quick way to catch broken records before you blame content. For receiver rules, compare Gmail sender guidelines and Microsoft email authentication docs.
Checklist before you publish
Use robots.txt for clear policy. Use logs and tests for proof.
- Decide whether you want Perplexity visibility before adding
Disallow: /for PerplexityBot. - Block live AI-search crawlers only when you accept removal from that engine's answers.
- Use GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended for training opt-out choices, not live search control.
- Keep important page content readable in server HTML where possible.
- Check server logs before saying a bot ignored or obeyed robots.txt.
- Audit SPF, DKIM, DMARC, MX, and blocklist status before diagnosing email placement.
FAQ
What is the exact PerplexityBot robots.txt block?
Use User-agent: PerplexityBot followed by Disallow: /. Put it in https://yourdomain.com/robots.txt. That states a sitewide block policy for PerplexityBot.
Does blocking PerplexityBot prove Perplexity did not crawl my site?
No. Robots.txt states your policy. It is not proof of behavior. Use server logs and vendor-documented crawler details when you need evidence.
Will blocking GPTBot remove my pages from ChatGPT search?
No. GPTBot is a training control. For ChatGPT search visibility, the live crawler is OAI-SearchBot. Blocking OAI-SearchBot removes you from that engine.
Why include email deliverability in a crawler guide?
The same mistake happens in both places. A public record is a policy signal. Real results still depend on receiver behavior, logs, and tests. For mail, that means SPF, DKIM, DMARC alignment, reports, reputation, and blocklist checks.