PerplexityBot vs Perplexity-User: SEO Guide

PerplexityBot is the crawler that affects search visibility

One robots.txt line can decide whether Perplexity can crawl a page for answers, so the bot name on that line matters.

Perplexity documents PerplexityBot as its crawler for indexing web content. If a site disallows PerplexityBot, it is telling Perplexity not to crawl those URLs for its search and answer system. That can remove the blocked pages from Perplexity visibility.

Perplexity-User is a different user agent. Perplexity says it is tied to user actions inside Perplexity, where a user's request may cause a page fetch. It is not described as the indexing crawler or a training crawler. Some publishers have reported Perplexity-User ignoring robots.txt, and Bytespider has also been reported to ignore it. Treat those reports as a reason to check server logs and edge rules. A robots.txt file states your policy. It does not prove what any bot did.

• Use PerplexityBot when you are deciding whether Perplexity can crawl your site for search answers.
• Use Perplexity-User when you are deciding whether user-triggered Perplexity requests may fetch your pages.
• Do not read a robots.txt rule as evidence of access. Use logs, IP checks, and edge controls for that.
• Keep important public pages reachable over normal HTML with clean status codes, canonical URLs, and no login wall.

AI search bots are different from training bots

If your goal is to appear in AI answers, allow the crawlers that power live search. Training opt-outs are a separate control.

The crawlers that decide whether you appear in AI answers are OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Googlebot for Google, and Applebot for Apple Intelligence. Google AI Overviews use the normal Google Search index, and Google does not provide a separate AI Overviews opt-out crawler. Disallowing one of these live search crawlers in robots.txt removes you from that engine.

Training and opt-out controls are different. GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended are training or opt-out controls. Blocking them does not affect live AI search visibility. Google-Extended and Applebot-Extended are robots-only control tokens with no separate crawl user-agent.

Only Googlebot documents JavaScript rendering. That does not prove other AI crawlers fail on JavaScript. It does mean client-side-only content is an undocumented risk for AI search. If the main answer, price, address, or guide text appears only after JavaScript runs, make a server-rendered or static version available. For a quick crawlability check, use the free AI visibility checker.

How to set robots.txt for this query

Start by deciding whether the page should be found in Perplexity answers. Then write the smallest rule that matches that choice.

If you want Perplexity visibility, do not disallow PerplexityBot. If you want to keep Perplexity's search crawler away, disallow PerplexityBot and expect those pages to drop out of that surface. For Perplexity-User, treat robots.txt as your stated policy for user-triggered fetches, not as proof of enforcement. If that access matters, pair the policy with WAF rules, IP verification, and log review.

• Allow live AI search crawlers on public pages you want cited: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Applebot.
• Block training or data corpus crawlers only if that is your policy: GPTBot, ClaudeBot, CCBot, Google-Extended, and Applebot-Extended.
• Check that key URLs return 200, are not blocked by robots.txt, and do not depend on a session cookie.
• Put the useful answer in the HTML. Do not hide it behind a form, a script-only app shell, or an image with no text.
• Use the official vendor docs for bot names. Bot lists change, and third-party lists can lag.

Email DNS is a separate trust layer

Crawler access affects whether a site can be read. Email authentication affects whether mail from the same domain is trusted.

SPF is a DNS TXT record that lists who may send mail for a domain. Publish exactly one SPF TXT record on each domain used in MAIL FROM, Return-Path, or HELO checks. Keep it under the RFC 7208 limit of 10 DNS-querying mechanisms and modifiers, including nested include, a, mx, ptr, exists, and redirect. ~all means soft fail for unmatched senders. -all means hard fail. Use -all only after the sender list is complete.

DKIM signs mail with a private key and publishes the matching public key in DNS under a selector, such as selector1._domainkey.example.com. For DMARC, the DKIM d= domain must align with the visible From domain, using the domain's relaxed or strict alignment policy.

DMARC ties SPF and DKIM to the visible From domain. p=none monitors. p=quarantine asks receivers to treat failures as suspicious, often by putting them in spam. p=reject asks receivers to reject failures. A rua address receives aggregate reports that show which sources pass, fail, and align. The free DMARC report reader can help read those XML reports.

MX records say where inbound mail for the domain should go. They do not authorize outbound mail unless your SPF record uses the mx mechanism. Blocklists are separate too. A clean SPF record will not erase a listed IP, a damaged domain reputation, spam complaints, suspicious links, or unwanted mail.

Why Gmail and Outlook still send mail to spam

Passing SPF, DKIM, and DMARC is the floor for serious sending. Inbox placement also depends on behavior.

Gmail and Outlook look at authentication, alignment, domain and IP reputation, complaint rates, list quality, message content, links, malware signals, forwarding behavior, volume spikes, and local user actions. A domain can pass DMARC and still land in spam if recipients often ignore it, mark it as spam, or receive mail they did not ask for.

Fix the simple DNS failures first because they are clear and testable. Then watch reputation and list quality. Check MX so replies and bounces work. Check blocklists when delivery changes suddenly. Read DMARC aggregate reports before moving from p=none to p=quarantine or p=reject. For a fast outside view of SPF, DKIM, DMARC, MX, and blocklists, run the free domain scorecard.

• Keep SPF valid and below 10 DNS lookups.
• Enable DKIM signing for every mail platform that sends as your domain.
• Use DMARC aggregate reports to prove alignment before enforcement.
• Keep MX records correct so replies, bounces, and verification mail work.
• Watch blocklists, complaints, and sudden volume changes together.

Official sources to keep handy

Use primary docs when a bot name or sender rule matters. These details change over time.

For email authentication, use RFC 7208 for SPF, RFC 6376 for DKIM, and RFC 7489 for DMARC, plus the current Google sender guidelines and Microsoft email authentication guidance. For AI crawlers, use the official docs from OpenAI, Anthropic, Perplexity, Google, Apple, and Common Crawl. For related deliverability guides, see InboxRadar articles.