Can AI Crawlers Execute JavaScript?
Published November 16, 2026
The short answer is that, in general, AI crawlers do not render JavaScript. Even when they can, you shouldn’t rely on it.
We’re going to look at how we got here, and why crawling can feel like it has gone backwards since the explosion of AI. We’ll explore how Google’s investment in JavaScript rendering shaped the web, and what the next generation of AEO technologies might look like.
We’ll also get into the conflicts of interest that web crawling is sparking. Walled gardens are being scaled, and the old guard of the internet may be reluctant to relinquish control of their domains.
Now let’s take a journey through crawl and space to explore the past, present and future of JavaScript crawling.
Contents:
- Fetching JavaScript isn’t the same as executing it
- Different bots have different jobs
- Why rendering at scale is a substantial investment
- Direct integrations offer another route
- Browser agents don’t solve every discovery problem
- How to check what crawlers can access
- Make essential content available without JavaScript
- In Closing
Fetching JavaScript isn’t the same as executing it
There have been several experiments testing whether AI crawlers discover links generated through JavaScript. In August 2026, Vinicius Stanula ran a test: one site with ~2,400 pages, split into HTML-link sections and JS-link sections, run in two phases over 41 days. None of the JavaScript injected links were found by AI crawl bots.
Vercel and MERJ’s December 2024 research also found that, apart from Google and Apple, most AI crawlers were not rendering JavaScript, even when they requested JavaScript files.
A log showing a bot requesting bundle.js leaves you with Schrödinger’s script: you know the file was fetched, but you don’t know whether the code was run.
A crawler might collect JavaScript as source code, extract information from it as text, or request it without using it to construct a page. In our own tests, we’ve also seen AI fetchers extract content from JSON as plain text without appearing to interpret its structure.
For SEO, the question is whether important URLs, directives or content are being missed because they exist only in JavaScript-generated output.
Even if a crawler can execute JavaScript, how long does it wait? How does it decide the page has finished loading? Which events and interactions does it reproduce?
“The problem isn’t whether machines can display the page. The problem is deciding which version of that page represents the truth.”
Contradictory directives make this especially consequential. If the original response contains noindex, Google may skip rendering, so JavaScript that is intended to remove the directive may never run. A technically capable renderer cannot help if an earlier processing decision prevents the page from reaching it. Google’s JavaScript guidance explains this limitation.
The same principle applies to content that appears only after a particular interaction. Being able to execute JavaScript doesn’t establish that a crawler will scroll, click, accept consent or reproduce the state you inspected.
Different bots have different jobs
Tory Gray and Jessyca Frederick’s Sitebulb webinar highlighted the need to distinguish different kinds of automated visitors. Training crawlers, live fetchers and browser agents shouldn’t be treated as interchangeable.
A bulk collector can gather useful material from a large corpus without reconstructing every page. A search system answering a specific question has a stronger reason to recover a particular authoritative source. An agent completing a purchase may need to operate the website directly.
Perplexity’s documentation illustrates the distinction: PerplexityBot supports search discovery, while Perplexity-User retrieves pages in response to user actions. Neither is described as a foundation-model training collector.
These roles suggest different incentives, but they don’t disclose every operator’s private reasoning. We can establish that a retrieval path doesn’t execute JavaScript without claiming to know that cost is its only reason.
A simple expectation is that bulk data collection will tend towards cheaper crawling, while specialist actions are more likely to justify an in-depth browser session. That leaves room for selective rendering without assuming every crawler will eventually work like Googlebot.
Why rendering at scale is a substantial investment
Google’s history gives us a useful explanation of why a crawler would take on this work.
As websites increasingly used JavaScript to generate content, reading the original HTML no longer reliably showed Google what visitors could see. Useful information was missing from its index.
Its first major public workaround was the 2009 AJAX crawling proposal. Website owners supplied rendered HTML snapshots that search engines could fetch, while users were sent to the original interactive pages.
By 2014, Google publicly revealed that it had been rendering a substantial number of pages. Its explanation was straightforward: valuable JavaScript-generated content was being missed, and executing JavaScript at web scale was difficult but worthwhile.
In 2019, it introduced evergreen Googlebot, moving to a regularly updated Chromium rendering engine.
That was a decade between the AJAX crawling proposal and evergreen Googlebot. It illustrates the work involved, even for a company with Google’s resources and incentive to index the web.
AI companies can use browser technology that already exists. But operating browsers across billions of pages still requires substantial resources, infrastructure and expertise. That sits awkwardly with AI providers’ pressure to control costs as they compete on price.
Each page can involve additional requests, code execution and waiting for content to appear. The crawler also has to decide when the page is ready to read. Google’s own documentation describes a rendering queue that runs when resources allow.
It comes down to whether the juice is worth the squeeze.
A crawler’s ability to execute JavaScript doesn’t mean it has an incentive to do so on every page. Some systems will render selectively. Others may get enough information from HTML, feeds or direct integrations.
Direct integrations offer another route
Cheaper and more direct ways for agents to access information and services are emerging, including MCP and agent commerce protocols.
These integrations can let agents access services that would otherwise require navigating complicated JavaScript interfaces. Their usefulness, though, depends on websites adopting them and being willing to hand over some control of the customer experience.
There are already signs of resistance. In the Perplexity–Amazon dispute, a U.S. appeals court overturned a ruling that had temporarily barred Perplexity from using its AI-powered shopping tools on Amazon’s platform.
The risk of legal challenge could limit websites’ willingness to adopt protocols designed for agents. It could also strengthen the incentive for agents to use browsers and JavaScript to purchase products and services through existing interfaces.
I expect the strongest incentives for that browser use to come from specific user requests and commercial tasks, rather than bulk collection for training data. But the balance will depend on how both websites and AI platforms develop.
Browser agents don’t solve every discovery problem
Browser agents can interact with websites that require JavaScript. OpenAI’s agent announcement, for example, described both a visual browser and a text browser, alongside direct API access.
That doesn’t guarantee your content will be discovered through the search or retrieval system used to select a page.
An agent might follow a user-provided URL, use search results, navigate links or access a connected service. If an earlier discovery path misses your JavaScript-only content, the availability of a browser later in the process may not recover it.
Jessyca identified a related auditing trap: content can appear in Chrome’s accessibility tree while being absent from the original response.
The accessibility tree is useful for assessing an interface a browser has constructed. It doesn’t establish what a simpler fetcher received. You need to inspect both.
How to check what crawlers can access
Start with what your server actually sends.
Use a simple HTTP request or inspect the page source. Check whether the response contains the main content, meaningful links, page title, canonical and robots directives.
Disabling JavaScript and reloading in a browser is also a useful first check. Compare that with the response itself: the Elements panel shows a constructed document, rather than the original payload.
If you test with a bot user agent, compare it with an ordinary request too. Different responses can affect your conclusions, and a user-agent string alone doesn’t reproduce a verified crawler.
Look for content and links that appear only after execution. Sitebulb’s Response vs Render reporting helps identify these differences across templates and URLs, rather than relying on a few manual checks.
Give priority to product descriptions, article bodies, navigation and indexing directives. A missing decorative feature is a different problem from a missing product listing.
Look for evidence of execution
Verify crawler identity where possible using the operator’s published guidance.
Then look for evidence that JavaScript actually ran. Did the bot trigger a script-only endpoint or discover a link that had no other route? Treat requests for JavaScript files as evidence of fetching, not execution.
A lack of resource requests is suggestive, but caching, observation windows and different retrieval infrastructure can complicate the picture.
A dedicated test section can compare HTML links with JavaScript-injected links. Keep the groups comparable, control alternative discovery routes and record verified requests over time.
Converting the JavaScript links to HTML provides a useful follow-up: do previously undiscovered pages then receive requests?
That strengthens the evidence, but changing crawl demand and timing still deserve consideration. Report what happened in the test rather than claiming permanent behaviour from one experiment.
Make essential content available without JavaScript
Will Critchlow’s recommendation in our retailer-focused webinar was practical:
“If you want to be visible in these, particularly the real time fetching from things like ChatGPT, then your stuff needs to be available even without JavaScript enabled.”
That doesn’t require every website to abandon JavaScript or rebuild from scratch. Depending on the framework, server rendering or static generation may be applied to the content that matters while retaining interactive features.
A loading skeleton can improve the perceived experience for a visitor. If it is all the server returns, a non-executing fetcher may receive only the loading state.
Making content available earlier can benefit visitors too. Google also says faster loading and rendering can let it crawl more content, although crawl demand and content value still matter. Faster delivery doesn’t guarantee more frequent visits. Google’s crawl-budget guidance explains these considerations.
In Closing
Crawler behaviour will change. New browser tools will appear, retrieval systems will improve, and operators may add execution to paths that currently lack it.
That is a reason to date findings and keep testing. It is a poor reason to leave essential content inaccessible today.
Google’s history shows that a strong incentive can justify substantial investment in JavaScript execution. It doesn’t establish that every AI system will make the same investment or use the resulting capability on every page.
For website owners, the dependable approach is to put essential public content and links in the initial HTML, then use JavaScript to support the experience around them. That serves simpler fetchers now and leaves browser agents able to interact when the task requires it.
Your content shouldn’t depend on another company deciding it is worth executing your application.
Kevin is a design engineer at Sitebulb, where he brings design and code together to build user interfaces and create engaging digital experiences.
Articles for every stage in your SEO journey. Jump on board.
Related Articles
How to Prioritize Technical SEO Fixes on Ecommerce Sites (For Revenue)
Initial Time to Interactivity (ITTI): The Metric that Core Web Vitals Misses
Technical SEO Debt: How to Measure, Prioritise & Fix It
Sitebulb Desktop
Find, fix and communicate technical issues with easy visuals, in-depth insights, & prioritized recommendations across 300+ SEO issues.
- Ideal for SEO professionals, consultants & marketing agencies.
Try our fully featured 14 day trial. No credit card required.
Try Sitebulb for free
Sitebulb Cloud
Get all the capability of Sitebulb Desktop, accessible via your web browser. Crawl at scale without project, crawl credit, or machine limits.
- Perfect for collaboration, remote teams & extreme scale.
If you’re using another cloud crawler, you will definitely save money with Sitebulb.
Explore Sitebulb Cloud
Kevin Letchford