How Does Googlebot Crawl, Render, and Index JavaScript?
Key takeaways:
- Googlebot crawls a URL, renders indexable 200 pages in evergreen Chromium, then indexes the rendered HTML.
- A noindex in the raw HTML stops rendering, so JavaScript can’t remove it.
- GPTBot and ClaudeBot download JavaScript without running it, so Google-indexed content can stay invisible to AI assistants.
Googlebot processes JavaScript pages in three stages: it crawls the URL, renders the page in headless Chromium, and indexes the result. What gets indexed is the page after the JavaScript runs, not the HTML your server sent, and the gap between JavaScript crawling and rendering is where most JavaScript SEO problems start.
JavaScript Crawling and Rendering: Three Stages, Three Different Jobs
JavaScript crawling and rendering are separate steps that read different versions of your page. Crawling is Googlebot requesting the URL and reading the raw HTML. Rendering is JavaScript execution by Google’s Web Rendering Service (WRS), a headless browser built on Chromium. Indexing is Google analyzing the rendered HTML.
Here’s how the three stages compare:
| Stage | What Googlebot does | What it reads | What stops it |
| Crawling | Checks robots.txt, requests the URL | Raw HTML and status code | A robots.txt disallow, a non-200 status |
| Rendering | Runs the JavaScript | Raw HTML plus JS, CSS, API data | A raw noindex, blocked JS files, script errors |
| Indexing | Analyzes the result | Rendered HTML | Missing content, a JavaScript-added noindex |
You can drop the old “two waves of indexing” model. Google’s Martin Splitt said in a 2020 AMA on Reddit’s r/TechSEO that the metaphor caused more confusion than it solved, and that you should assume Google crawls, renders and then indexes every page.
What Googlebot Checks Before Any JavaScript Runs
Four checks decide whether a page gets rendered at all. First, robots.txt: Google won’t request a disallowed URL or render JavaScript from blocked files.
Second, HTTP status codes: Google queues 200 pages for rendering, and in Vercel and MERJ’s testing, 3xx, 4xx and 5xx responses weren’t rendered. Third, a noindex tag in the initial HTML blocks rendering, so a product page that ships <meta name=”robots” content=”noindex”> and removes it with a script stays out of the index.
Fourth, size. Googlebot reads only the first 2MB of each HTML, JS or CSS file, measured uncompressed, and ignores the rest, which is a common failure point for large JavaScript bundles.
How Long Rendering Takes and What the Renderer Skips
JavaScript rendering usually follows the crawl within seconds, but it can take hours. Across 37,000+ renders on nextjs.org in April 2024, Vercel and MERJ measured a median of 10 seconds from crawl to render, while the 99th percentile reached about 18 hours. Google hasn’t published a fixed render timeout.
The renderer also differs from the browser you test in. Google’s JavaScript troubleshooting guide says WRS:
- Clears cookies, local storage and session storage between page loads.
- Declines permission requests such as camera access.
- Never clicks or scrolls, so content behind interaction is missed.
- May ignore caching headers, which is why Google recommends fingerprinted files like main.2bb85551.js.
- Uses HTTP only, with no WebSockets.
Anything your content depends on, whether a cookie, a click or a socket, needs a fallback that works on a plain page load.
When the Raw HTML and the Rendered HTML Disagree
Google indexes the rendered HTML, but the raw HTML wins for the most restrictive signals. A noindex that JavaScript adds is respected, while one that JavaScript removes isn’t. Google also asks you not to use JavaScript to change the canonical to a different URL.
Here’s what Google uses when the versions conflict:
| Signal | Raw HTML | Changed by JavaScript | What Google uses |
| Robots noindex | Present | Removed | noindex, page not rendered |
| Robots noindex | Absent | Added | noindex |
| Canonical | URL A | Changed to URL B | Conflicting signal |
| Title and meta description | Generic | Rewritten | Rendered version |
| Internal links | Missing | Injected as <a href> | Found after rendering |
HTTP Archive’s 2025 Web Almanac shows this isn’t rare: rendering changed the canonical on 3.02% of mobile sites. The safe rule is to put every SEO signal in the raw HTML and keep JavaScript from changing it.
Why AI Crawlers Miss JavaScript Content
Most AI crawlers read only the raw HTML. In a December 2024 analysis, Vercel and MERJ found that ChatGPT’s and Claude’s crawlers fetch JavaScript files (11.50% and 23.84% of their requests) but never execute them, while Gemini renders through Googlebot’s infrastructure.
So if a product name, price or article body exists only after client-side rendering, ChatGPT and Claude never see it, even when Google indexes the page fine. Server-side rendering puts it in the initial HTML response instead.
How to Check What Googlebot Actually Rendered
Compare the raw HTML with Google’s rendered copy on one URL from each page template:
- Compare view-source: with the rendered DOM in Chrome DevTools > Elements for the title, canonical, robots meta tag and main text.
- In Google Search Console, run URL Inspection > Test live URL > View tested page, and check the HTML, screenshot and JavaScript errors.
- Open Settings > Crawl stats > By file type to confirm Googlebot requests your JavaScript and JSON files.
- Search an exact sentence that appears only after rendering, with a site: query, to confirm indexing.
Manual checks like these cover one URL at a time, which gets slow on large sites.
EdgeComet works from the bot request path instead: its Log Analyzer attaches the rendered page, about 90 data points per request including structured data, to each bot visit, so you see what Googlebot and GPTBot received.
Its JS Rendering module (Business plan and above) serves bots cached, rendered HTML in under 15 milliseconds, a dynamic rendering setup Google calls a workaround, so bots must get the same content users see.
On JavaScript-heavy sites, start with the raw HTML: make it carry the title, canonical, robots meta, content and links, keep robots.txt open to the files pages need, and confirm the rendered result in URL Inspection rather than assuming it.