Server Side Rendering for AI Crawlers

AI answers start with plain HTML

A page can look perfect in your browser and still be thin to a crawler. That risk gets bigger when the real copy appears only after JavaScript runs.

Server side rendering means the first HTML response already contains the main text, links, title, headings and canonical signals. For AI search, that matters because the systems that choose sources must first fetch and read the page. If they receive an empty app shell, a loading state, or content hidden behind a blocked script, you have made discovery harder.

The crawlers that decide whether you can appear in AI answers are OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Googlebot for Google Search and AI Overviews, and Applebot for Apple Intelligence and Apple search features. If you disallow these in robots.txt, you remove that engine's normal path to your content.

Google documents JavaScript rendering for Googlebot. It says Google can crawl, render and index JavaScript pages, and that server side rendering or pre-rendering is still a good idea because it helps users and crawlers. The other AI search crawlers do not publish the same full JavaScript rendering contract. So the honest rule is simple: client-side-only content is an undocumented risk for AI crawlers. Do not claim a specific bot cannot run JavaScript unless that vendor says so.

Which bots to allow

The names in robots.txt matter. Search bots and training bots do different jobs.

• Allow OAI-SearchBot if you want ChatGPT search to surface and cite your pages. OpenAI says blocking it removes pages from ChatGPT search answers, though a page may still show as a navigational link.
• Allow Claude-SearchBot if you want Claude's search systems to index your content for search quality. Anthropic says disabling it may reduce site visibility and accuracy in user search results.
• Allow PerplexityBot if you want Perplexity search results to surface and link your site. Perplexity says this bot is for search results and is not used for foundation model training.
• Allow Googlebot if you want Google Search, AI Overviews and AI Mode eligibility. Google says AI features use the normal Search controls. There is no separate opt-out crawler for AI Overviews.
• Allow Applebot if you want Apple search features, Siri, Spotlight, Safari and Apple Intelligence surfaces to find your public pages.

Training and opt-out controls are separate. GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended are about training or broader model data controls. Blocking them does not block live AI search visibility by itself. Google-Extended and Applebot-Extended are robots-only control tokens, not separate crawl user agents. Check the current OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl docs before changing these rules.

Treat robots.txt as a stated site policy, not proof of what happened. Major search crawlers publish rules for how they use it, but the file cannot enforce behavior by itself. Perplexity-User and Bytespider have been reported to ignore robots.txt, so use server logs, verified IP ranges, WAF logs and vendor docs before saying what a bot did.

You can test the public side with the free AI visibility checker. It is a quick way to catch blocked bots, empty HTML and crawl barriers before they cost you citations.

How to make a page readable on first fetch

The safest page for AI crawlers is boring at the transport layer: a clean 200 response with useful HTML already in it.

• Return a 200 status for the canonical URL. Do not serve the article behind a soft 404, a login gate, a cookie wall, or a heavy bot challenge.
• Put the core article text, headings, links and FAQs in the initial HTML. Hydration can add interaction later, but the answer should not depend on it.
• Keep the same visible content for users and crawlers. Do not use cloaking. If you pre-render, pre-render the real page.
• Make internal links plain <a href> links. Crawlers should be able to discover related guides at the guide index without clicking buttons.
• Let key CSS and script assets be fetched. Google says blocked resources can stop rendering. Other crawlers may be stricter, less patient, or less documented.
• Check your CDN and WAF. A page allowed in robots.txt can still fail if the firewall blocks the bot, returns a challenge page, or serves different HTML by country.

Do the dumb test before the fancy one. Fetch the URL with JavaScript off, then view source. If the main answer is missing, server render it or pre-render it. After that, test the real user agents and known IP ranges where vendors publish them.

Email deliverability has the same lesson

Crawlers and mailbox providers both punish guesswork. They read records, headers and reputation signals, then make a fast decision.

For email, the first layer is DNS. SPF lists which servers may send mail for your domain. Publish one SPF TXT record, keep it within the 10 DNS-lookup limit in RFC 7208, and avoid +all. Use ~all while you are still testing. Use -all only when you are sure every real sender is covered.

DKIM, defined in RFC 6376, signs mail with a private key and lets receivers check the public key at a selector such as selector._domainkey.example.com. If your email platform can sign mail, turn it on. A message that passes DKIM with an aligned domain gives Gmail, Outlook and other providers stronger proof that the mail is really yours.

DMARC, defined in RFC 7489, ties SPF and DKIM to the visible From domain. Start with p=none and a rua address so you can read aggregate reports. Move to p=quarantine or p=reject after your real senders pass and align. If the XML reports are hard to read, use the free DMARC report reader.

MX records tell the world where to deliver inbound mail. Missing or broken MX breaks replies and can be one more weak signal in a sender review. Blocklists matter when a domain or IP has sent abuse or been compromised, but they are rarely the only issue. Google and Microsoft sender guidelines also focus on authentication, domain reputation, IP reputation, spam complaints, volume spikes, bad lists, suspicious links and user engagement. A clean SPF, DKIM and DMARC setup will not save a bought list, but a broken setup can sink good mail.

InboxRadar's free domain scorecard checks SPF, DKIM, DMARC and MX, then watches for drift. Use it after every DNS change and after adding any new sender.

A practical crawl and email checklist

Use this when a page needs to be cited by AI search and the domain also sends mail.

• Fetch the page source. Confirm the main body, headings, FAQ text and internal links are present before JavaScript runs.
• Check robots.txt for OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Applebot. Do not confuse them with training controls.
• Verify the server returns the same useful content to normal browsers and verified crawlers.
• Review CDN, WAF and geo rules for challenge pages, blocked countries, bot scores and rate limits.
• Confirm SPF has one record, valid senders, no +all, and no more than 10 DNS lookups.
• Confirm DKIM signing is active for every mail stream, including marketing, billing, support and product mail.
• Confirm DMARC exists at _dmarc, aligns with SPF or DKIM, sends rua reports, and has a policy that matches your readiness.
• Check MX and blocklists, then watch complaint rate and bounce patterns after each campaign.