Do AI Crawlers Render JavaScript?

A page that looks perfect in Chrome can arrive at an AI engine as forty lines of boilerplate and one empty container. Not a slow page. Not a blocked page. An empty one.

This keeps turning up in teardowns. A company ranks well, has real content, has a technically clean site, and gets almost nothing in AI answers. Then you fetch the page the way a bot fetches it, without a browser, and the pricing table is gone. The feature list is gone. The FAQ is gone. What is left is a title tag and a script tag. The cause is architectural, and it is not a secret: industry benchmarks put the share of AI crawlers that cannot execute JavaScript at roughly 69%.

This piece is the rendering layer specifically. It does not repeat the crawler roster or the robots.txt rules, which are covered in how AI crawlers index your site. That is about access. This is about whether, once a bot is through the door, there is anything on the page for it to read.

01Crawling, fetching and executing: what's the difference?

Crawling is the network transaction: a GET returns status, headers and a body, one round trip. Asset fetching is the bot parsing that body, finding references to stylesheets, scripts and images, and issuing secondary GETs. Fetching a file is not the same as using it. Script execution is a different category of work: initialising a JS runtime, building the DOM and CSSOM, firing lifecycle events, waiting on async calls to your API, and hydrating the interface. It needs a browser.

The distinction matters because several AI bots do fetch your JavaScript files, which makes their logs look like rendering. They are not rendering. Those files are collected as raw text for training corpora, not executed to paint components. The reason is economics, not capability: a search engine monetises the index and can pay for a fleet of headless Chromium instances; an AI operator optimising for corpus volume and a two-second answer cuts the browser out of the hot path.

02Which AI crawlers render JavaScript, and which don't?

The pattern is clean. Every crawler that passes is operated by a company that already ran a search engine before it ran a model. Google-Extended is not really a crawler at all, it is an access token in robots.txt that governs whether Googlebot's Web Rendering Service output can feed Gemini training. Applebot inherits Apple's browser-based crawling stack. Everyone else built new infrastructure in the last three years, and nobody building new infrastructure chose to put Chromium in the hot path.

03Why do the user agents look like browsers?

ClaudeBot fetches JavaScript on roughly 23.84% of its requests, more than double OpenAI's rate, and spends another 35.17% of its fetches on images. It runs none of the scripts it downloads. GPTBot sits at 11.50%. ChatGPT-User, the bot that fires when a person asks a live question in a chat window, fetches no secondary assets at all, zero CSS, zero JS, one request and done. If you are diagnosing this from server logs, that is the trap: you will see a bot that identifies as Chrome, requesting your bundle files, hitting your CDN, generating traffic that looks like a browser session. It never renders a pixel.

04What does each AI fleet actually do?

OpenAI

GPTBot handles training-corpus collection, OAI-SearchBot builds the search index, and ChatGPT-User fetches URLs live when a conversation needs them. Telemetry across more than 500 million GPTBot fetches recorded zero instances of client-side script execution. ChatGPT-User is the strictest of the three, because a person is waiting: it runs a roughly five-second ceiling, and if your page does not deliver text inside that window it issues an HTTP 499, closes the connection, and falls back to querying Google with a site: operator to scrape whatever the cached index holds. Your server sees a request; your page never gets read; the answer gets built from someone else's summary of you.

Anthropic

ClaudeBot and Claude-SearchBot behave the same way structurally. Server-log monitoring across Next.js endpoints shows heavy asset fetching and no runtime. Content painted by client-side scripts comes back as an empty shell, and the download volume is misleading, which is exactly why so many teams conclude they are fine.

Perplexity, Google and Apple

PerplexityBot is built for low-latency live retrieval and skips asset downloading almost entirely, processing raw markup. In benchmarks against a client-rendered React app it says so out loud, telling the user the page depends on dynamic JavaScript and it cannot summarise it, the honest failure case; the dangerous one is silence. Gemini is the exception, and an inherited one: it sits on Google's search infrastructure and benefits from the Web Rendering Service, so client-rendered content stays visible as long as Googlebot's Chromium instances do not time out first. Applebot similarly runs browser-based crawling. Neither is evidence that AI crawlers render, both are evidence that search crawlers render and two AI products happen to be downstream of one.

05What does a non-rendering crawler actually receive?

This is what a non-rendering crawler receives, the raw server response that GPTBot, ClaudeBot and PerplexityBot ingest:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Enterprise Analytics Platform | Pricing & Features</title>
  <script src="/static/js/main.7c89b2e1.js" defer></script>
  <link href="/static/css/main.css" rel="stylesheet">
</head>
<body>
  <div id="root"></div>
  <!-- Parsing ends here. Extractable content: 0 bytes. -->
</body>
</html>

And this is what a browser, Googlebot or Applebot ends up with once the runtime has run:

<body>
  <div id="root">
    <main class="dashboard-container">
      <h1>Enterprise Analytics Engine</h1>
      <section class="pricing-matrix">
        <h2>Plan pricing</h2>
        <div class="tier">
          <h3>Pro Plan</h3>
          <p class="price">$299 / month</p>
          <ul class="features">
            <li>Real-time event streaming</li>
            <li>Automated LLM citation tracking</li>
            <li>Unlimited seat licenses</li>
          </ul>
        </div>
      </section>
    </main>
  </div>
</body>

Same URL, two payloads. The bot gets the first box; your buyer's question gets answered from the second, on somebody else's site. The useful way to quantify this is a single ratio: the extractable body text in the raw response divided by the extractable body text in the hydrated DOM.

CVR      =  T_raw / T_rendered           # Content Visibility Ratio
CVR_site =  Σ (CVR_i x V_i) / Σ V_i       # weight by commercial value

  T  = word count of visible body text (excl. nav, scripts, boilerplate)
  V  = commercial weight of page i (use pipeline influence, not sessions)
  1.0 = fully server-rendered   ~0.0 = pure client-rendered

A fully server-rendered page sits near 1.0, a pure client-rendered page near 0.0, and most real sites are hybrids: a static marketing shell with a dynamically loaded pricing widget, or a server-rendered article with client-loaded comparison tables. Compute the site-level number too, but weight it by commercial value, not page count, an average across ten thousand blog URLs will tell you everything is fine while the six pages that matter score near zero.

06Why does the five-second budget break client-rendered pages?

T_visible = TTFB + T_transfer + T_bundle + T_exec + T_api  < 5000 ms

  server-rendered page:  only TTFB + T_transfer exist
  client-rendered page:  all five terms, in series, before any text
  target: TTFB < 500 ms, full response well inside the 5s ceiling

The practical target is a time to first byte under 500 milliseconds with the full response well inside five seconds, achievable on almost any stack when the HTML is complete on arrival, and close to impossible when the content depends on a round trip your API has not started yet. Geography makes it worse, and it is the part most teams outside the United States miss: the AI fleets do not crawl from distributed regions. ChatGPT fetches primarily out of Des Moines and Phoenix, and Claude crawls exclusively out of Columbus. If your origin sits in Frankfurt, Mumbai or Singapore, every request pays a trans-continental round trip out of the same five-second budget, and no CDN edge helps you if the HTML itself is assembled at the origin.

07Why do your logs look busier than your visibility?

Vercel's telemetry across Next.js infrastructure puts ChatGPT at 34.82% of fetches hitting 404s and Claude at 34.16%, largely from repeated requests for stale files under paths like /static/js/. ChatGPT burns a further 14.36% in redirect chains, against Googlebot's 1.49%. Two consequences: raw bot hit counts are a terrible proxy for how much of your content was actually read, and crawl budget is finite even for well-funded crawlers, so the waste comes out of the same allocation your real pages need.

08Why does client-side rendering fail twice?

ChatGPT Search does not fetch most URLs live, roughly 92% of its live web queries resolve through Bing's index rather than a fresh GPTBot request. Bingbot's rendering is limited and selective compared with Googlebot's. So the client-rendered page misses on the direct path because OpenAI's bots run no runtime, and misses on the indexed path because the index it is reading from never held the rendered DOM either.

P(visible) = 1 - (1 - r_direct)(1 - r_index)

  r_direct = P(a live bot fetch returns extractable content)
  r_index  = P(the index it falls back to holds rendered text)
  client-side rendering drives BOTH terms toward zero at once.

The page types this hits hardest are the commercial ones. Product and specification pages account for around 20.1% of AI citations and 14.2% of post-referral conversions; comparison and alternatives pages account for another 9.3% and carry the highest purchase intent on the site. Those are exactly the pages built as dynamic tables fed by an API. Domain authority does not rescue you: organic traffic volume is one of the strongest predictors of AI-citation frequency, but volume only amplifies content the engine can read, so a high-authority domain serving app shells converts its advantage into nothing. The page-level counterpart is the anatomy of a high-citation page, which covers what goes inside the HTML once the HTML actually exists.

09Why does ranking on Google prove nothing?

Standalone AI engines have no deferred rendering stage. There is no queue, no second visit, no eventual consistency. Whatever was in the initial response body is the complete record of your page, permanently, until the next crawl returns the same empty shell. A page at position one in Google can be a blank document to the model a buyer is asking for a recommendation. This is the rendering-layer version of ranking is not visibility.

10How do you test your own site in ten minutes?

1. The single-URL curl test

Fetch the page with a crawler user agent and grep for something that only exists in your real content, a price, a plan name, a specific feature string.

curl -A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" \
     -s https://example.com/pricing | grep -i "per month"

# no output means the string is not in the raw HTML,
# which means it is not in the model's view of your page.

2. The multi-bot sweep

Different bots occasionally get different treatment, usually because of a CDN bot rule somebody added years ago. Test the fleet, not one agent. The word-count column is the one to watch, a page that returns 40 words to every agent and 1,400 in a browser has told you everything you need to know.

#!/usr/bin/env bash
URL="https://example.com/pricing"
NEEDLE="per month"
declare -A AGENTS=(
  [GPTBot]="Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)"
  [OAI-SearchBot]="Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)"
  [ChatGPT-User]="Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)"
  [ClaudeBot]="Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
  [PerplexityBot]="Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/bot)"
)
for name in "${!AGENTS[@]}"; do
  code=$(curl -A "${AGENTS[$name]}" -s -o /tmp/body -w "%{http_code}" --max-time 5 "$URL")
  words=$(sed 's/<[^>]*>/ /g' /tmp/body | wc -w)
  grep -qi "$NEEDLE" /tmp/body && hit="FOUND" || hit="MISSING"
  printf "%-16s http=%s words=%-6s %s\n" "$name" "$code" "$words" "$hit"
done
# the word-count column is the tell: 40 words to every bot,
# 1,400 in a browser, and you have found your ceiling.

3. Scoring the Content Visibility Ratio

For a real number across a set of URLs, compare the raw fetch against a rendered fetch and diff the text. This is the version worth wiring into CI.

import requests, subprocess, json
from bs4 import BeautifulSoup

BOT  = "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)"
DROP = ["script", "style", "noscript", "nav", "footer", "svg"]

def visible_words(html):
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(DROP):
        tag.decompose()
    return len(soup.get_text(" ", strip=True).split())

def raw(url):
    return requests.get(url, headers={"User-Agent": BOT}, timeout=5).text

def rendered(url):                       # any headless runner works here
    return subprocess.check_output(["node", "render.js", url], text=True)

def cvr(url):
    t_raw, t_rendered = visible_words(raw(url)), visible_words(rendered(url))
    return round(t_raw / max(t_rendered, 1), 3)

URLS = ["https://example.com/pricing",
        "https://example.com/product/analytics",
        "https://example.com/compare/competitor-x"]
report = {u: cvr(u) for u in URLS}
print(json.dumps(report, indent=2))
assert all(v >= 0.85 for v in report.values()), "CVR floor breached"  # fail the build

A fourth check takes ten seconds and needs no tooling: open Chrome DevTools, hit the command palette, type Disable JavaScript, and reload. What remains on screen is roughly the dataset available to a non-rendering bot. Missing favicons are a useful tell, since SPAs that inject icon references through script leave AI interfaces showing a generic placeholder.

11What are the four fixes, and what does each cost?

In the Next.js App Router, React Server Components run data fetching on the server by default. The async work completes before the response is sent, so core content is already in the initial HTML, the browser still hydrates and stays interactive, and the crawler gets a populated document. Nobody has to choose.

// app/pricing/page.tsx
// Server Component by default. No "use client" directive.
// The fetch resolves before a single byte is sent to the client.
export const revalidate = 3600;          // ISR: regenerate hourly

export default async function PricingPage() {
  const tiers = await fetch("https://api.example.com/pricing", {
    next: { revalidate: 3600 },
  }).then((r) => r.json());

  return (
    <main>
      <h1>Enterprise Analytics Engine pricing</h1>
      {tiers.map((tier) => (
        <section key={tier.id}>
          <h2>{tier.name}</h2>
          <p>{tier.price} per month</p>
          <ul>{tier.features.map((f) => <li key={f}>{f}</li>)}</ul>
        </section>
      ))}
    </main>
  );
}

One detail changes how you test. Next.js streams progressive HTML chunks wrapped in Suspense boundaries to human browsers, but when incoming headers match a known crawler user agent it pauses streaming until every server promise resolves and delivers one complete document. So a page can look chunked in a browser and arrive whole to a bot. Test with the bot user agent, not with your own.

12How does edge prerendering work for a codebase you can't rewrite?

A CDN worker intercepts the request before it reaches your origin and branches on user agent. Human traffic passes straight through to the normal SPA bundle. Known crawler traffic is checked against a key-value store of rendered snapshots: a cache hit returns complete HTML in single-digit milliseconds, a miss dispatches a render to a managed headless browser, returns the result, and writes it back to the store.

// Cloudflare Worker: bot interception with a KV snapshot layer
const BOTS = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|Bytespider|CCBot/i;

export default {
  async fetch(request, env, ctx) {
    const ua  = request.headers.get("user-agent") || "";
    const url = new URL(request.url);

    // humans and unknown agents: straight to origin, untouched
    if (!BOTS.test(ua)) return fetch(request);

    const key    = `snap:${url.pathname}`;
    const cached = await env.SNAPSHOTS.get(key);
    if (cached) {
      return new Response(cached, {
        headers: { "content-type": "text/html", "x-prerender": "hit" },
      });
    }
    // miss: render in a managed headless browser, then persist
    const html = await renderWithBrowser(env, url.toString());
    ctx.waitUntil(env.SNAPSHOTS.put(key, html, { expirationTtl: 86400 }));
    return new Response(html, {
      headers: { "content-type": "text/html", "x-prerender": "miss" },
    });
  },
};

Cloudflare documents this pattern in its Browser Rendering prerender guide, and if you would rather not run the headless layer yourself, Prerender.io publishes a Worker integration that does the same job as a managed service.

13What can you fix for free with progressive enhancement?

<!-- invisible to non-rendering crawlers: content mounts on click -->
<div class="accordion" data-faq-widget></div>

<!-- visible to everything: collapsed in the browser, present in the HTML -->
<details>
  <summary>Does the Pro plan include SSO?</summary>
  <p>Yes. SAML and OIDC single sign-on are included on Pro and above at
     no additional cost, with SCIM provisioning available on Enterprise.</p>
</details>

The second wound is treating structured data as content delivery. Schema is valuable and you should ship it, as covered in schema markup and AI citations, but empirical tests on ChatGPT-User show it frequently bypasses JSON-LD parsing and works from plain text in the body. If your price only exists inside a JSON-LD block, treat it as unpublished, mirror every critical fact, pricing, availability, specifications, in ordinary semantic HTML. Third, check what your framework does to navigation: a client-rendered nav means the crawler finds no internal links, so discovery of your other pages depends entirely on the sitemap, which is a fragile way to run a site.

14Which pages should you fix first?

Risk_i = C_i x (1 - CVR_i)

  C   = the page type's share of AI citations
  CVR = how much of it a non-rendering bot can read
  sort descending -> your remediation queue, top to bottom.

Run the ratio across your commercial URLs, sort descending, and stop where the score falls below your smallest meaningful number. In almost every audit the answer comes back the same: pricing first, product and specification pages second, comparison and alternatives pages third.

15What's the 30-day sequence?

The last row matters more than it looks. Once the HTML is readable, the constraint moves from infrastructure to content, and the question changes from whether the engine can read the page to whether it wants to cite it. That is a different problem with a different playbook, and prompt-to-citation tracking covers how to measure the second half. The retrieval pipeline your rendered text then has to survive is in how RAG actually works.

16What should you do this week?

• Run the curl test on your pricing page. One command, one grep. If the price is not in the output, you have found your ceiling.
• Score five URLs, not five hundred. Pricing, top product page, main comparison page, one docs page, one blog post. The spread tells you the shape of the problem.
• Stop citing Search Console as evidence. It measures a rendering queue that only one of these engines has.
• Pick the cheapest fix that works. Content changes daily, SSR or ISR. Monthly, SSG. Codebase untouchable, edge prerendering.
• Put the check in CI before you ship the fix, or the next refactor quietly reintroduces it and nobody notices for two quarters.

The uncomfortable part of this finding is how mundane it is. There is no ranking algorithm to reverse-engineer and no prompt to optimise. A crawler asked for your page, your server sent an empty container, and the model wrote its answer from whoever sent text instead. That is the entire mechanism, which also makes it the cheapest fix in the stack. Content strategy takes quarters to compound. This one lands the day you deploy it.