Googlebot runs JavaScript. Most AI crawlers do not. That single difference explains why a website can rank well on Google and still be invisible to ChatGPT, Perplexity, and Claude: if your pages are assembled in the browser by JavaScript — a pattern called client-side rendering — crawlers like GPTBot, ClaudeBot, and PerplexityBot may fetch them and find little more than an empty shell and a loading spinner. This post explains why that happens, how to test your own site in under a minute, and the practical fixes: server-side rendering, static generation, prerendering, and a habit called progressive enhancement.
Why would an AI crawler see a blank page?
Because most AI crawlers read the raw HTML your server sends and do not execute the JavaScript that builds the visible page. On a client-side-rendered site — a single-page application built with plain React, Vue, or Angular — the server responds with a nearly empty HTML document plus a JavaScript bundle. A human's browser downloads the bundle, runs it, fetches your data, and paints the page. A crawler that skips that step sees the “before” state: a root element with nothing inside it.
Googlebot invests in a full Chromium-based rendering pipeline because Google has two decades of search infrastructure behind it. AI crawlers are newer, crawl at enormous scale, and rendering every page in a headless browser is expensive — so the sound working assumption is to treat every AI crawler as a non-rendering crawler. OpenAI, Anthropic, and Perplexity all document their bots' names, but none of them promises Googlebot-style JavaScript execution.
The consequence is direct: answer engines quote text they can retrieve. If your pricing, your FAQ answers, or your product descriptions exist only after JavaScript runs, they are not in the text an AI system pulls back — and content that cannot be retrieved cannot be cited. (For the full retrieval pipeline, see how AI answer engines work.)
How do I test what a non-JS crawler sees?
Fetch the raw HTML and search it for a sentence that matters. Three ways, from fastest to most thorough:
- View source. Open an important page, press Ctrl+U (Cmd+Option+U on Mac), and search the source for your phone number, a price, or a sentence from your main heading. If it is not there, a non-rendering crawler cannot see it. Careful: right-clicking and choosing “Inspect” shows the rendered page after JavaScript ran — that is exactly the wrong test. Use “View page source.”
- curl. From a terminal, run curl -s followed by your page URL and read what comes back. That response is precisely what a non-rendering bot receives — no JavaScript, no rendering, just what your server sent.
- Disable JavaScript. In Chrome DevTools settings, tick “Disable JavaScript,” then reload the page. If you see a blank screen or an endless spinner, so does most of the AI ecosystem.
Run the test on your money pages, not just the homepage: service pages, product pages, pricing, FAQs, and contact. Two minutes of checking beats months of wondering why AI never mentions you.
What are the fixes: SSR, SSG, or prerendering?
All three deliver the same outcome — complete HTML in the server's first response — and differ only in when that HTML gets built.
- Static site generation (SSG) builds every page into a plain HTML file at deploy time. It is the safest and fastest option for content that changes occasionally: marketing pages, blog posts, service descriptions. There is nothing left to render because the finished page already exists.
- Server-side rendering (SSR) builds the HTML on each request. Use it when content is personalized or changes constantly. It needs slightly more infrastructure, but crawlers get the same complete HTML.
- Prerendering is the retrofit: a service or middleware detects bot user agents and serves them a pre-built HTML snapshot, while human visitors still get the JavaScript app. It is the least invasive fix for an existing single-page application you cannot rebuild soon — but it adds a moving part that can break silently, so check the snapshots regularly.
Many modern frameworks also offer hybrid modes, such as incremental static regeneration, which rebuild static pages on a schedule — a good match for product catalogs and blogs.
Which frameworks are safe by default?
Traditional server-rendered platforms are safe out of the box; JavaScript meta-frameworks are safe by default but easy to break; plain single-page applications are unsafe until you add one of the fixes above.
- Safe by default: WordPress, Shopify, Squarespace, Webflow, and Wix all send complete HTML from the server, as do classic server stacks like PHP, Rails, and Django. Whatever their other trade-offs, their content is visible to non-rendering crawlers.
- Safe if used as designed: Next.js, Nuxt, SvelteKit, Astro, and Remix render on the server or at build time by default. The trap is fetching data in the browser after load — content loaded in a React useEffect hook or a client-only component never appears in the initial HTML, even on a “safe” framework.
- Unsafe until fixed: a bare React, Vue, or Angular single-page app without a server-rendering layer ships an empty container element and builds everything client-side. These need SSR, SSG, or prerendering before AI crawlers can read them.
Treat the framework label as a starting point, not a guarantee — the view-source test is the truth. Even on WordPress, a page-builder widget or third-party plugin can inject key content with JavaScript.
Progressive enhancement: keep critical facts in the HTML
Progressive enhancement means the plain-HTML version of your page carries the essential information, and JavaScript only adds polish on top. For answer engine optimization, that translates into a short list of facts that must survive with JavaScript turned off:
- Who you are and what you do, in one plain sentence
- Prices, or at least price ranges
- Location, hours, phone, and email
- Answers to your most common customer questions
- Product or service names and their key specs
Common leak points: review widgets that inject testimonials with JavaScript, pricing tables pulled from a billing API in the browser, tab and accordion content injected on click, and contact details that live only in a chat widget. If a third-party widget owns a critical fact, duplicate that fact as ordinary HTML text nearby.
The same rule applies to structured data. JSON-LD placed in the page's initial HTML is readable by any crawler; JSON-LD injected through a tag manager exists only after JavaScript runs, which defeats the point for non-rendering bots. Our guide to schema markup for AEO covers what to include and where.
Frequently asked questions
Does ChatGPT's crawler run JavaScript?
OpenAI does not document GPTBot as a JavaScript-rendering crawler, and the safe operating assumption is that it — like most AI bots — reads raw HTML only. Build for the non-rendering case and every crawler can read you, including Googlebot.
My site ranks well on Google. Doesn't that mean AI can read it?
No. Googlebot renders JavaScript, so a Google ranking proves nothing about what a non-rendering crawler sees. A client-side-rendered site can hold a top Google position while being effectively blank to AI crawlers — that gap is part of what separates AEO from SEO.
Is serving prerendered pages to bots considered cloaking?
Not if the prerendered snapshot contains the same content humans see. Cloaking means showing bots materially different content to manipulate rankings; dynamic rendering that serves equivalent content has long been treated as a legitimate workaround. Keep the two versions identical in substance and you are fine.
How can I tell whether AI crawlers visit my site at all?
Check your server logs for user agents like GPTBot, ClaudeBot, and PerplexityBot. If they never appear, check your robots.txt and firewall rules first — our guide to AI crawlers and robots.txt explains who they are and how to admit the ones you want.
Not sure what AI crawlers see on your site?
We can tell you in plain English. The free AEO Fail audit fetches your pages exactly the way non-rendering AI crawlers do and flags every critical fact missing from your raw HTML — request your free audit. If something is broken, we fix it at $99/hour — usually a targeted rendering change, not a rebuild — and $19/month monitoring re-checks your pages so a framework update never silently blanks you out again.