AI Crawlers Do Not Render JavaScript

Most AI crawlers, including GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, read only the raw HTML your server sends and do not execute JavaScript. Anything that appears only after client-side rendering is invisible to them. Googlebot, which feeds Gemini, and Applebot are the exceptions. To be cited, put your content in the initial HTML.

There is a technical assumption embedded in most marketing and engineering teams that is quietly costing brands their AI search visibility: if Google can crawl and render our site, AI crawlers can too.

It is not a safe assumption. Google has spent years building the infrastructure to render JavaScript at scale. The crawlers that feed ChatGPT, Claude and Perplexity have not been shown to do the same, and that gap is where a large share of AI citation invisibility starts. Closing it is one of the highest-leverage technical moves a marketing team can make in 2026.

It is worth closing because the traffic is valuable. Lantern's monitoring data shows AI-referred visitors converting at 14.2%, versus 2.8% for standard Google clicks. None of that is available if AI crawlers can't read the page.

What AI Crawlers Actually See

When GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot fetches a page, it receives the raw HTML response before any JavaScript executes.

For a server-rendered site, that HTML contains the whole page: headings, body copy, structured data, schema markup and internal links. Everything an AI engine needs to understand and cite it.

For a single-page application or a heavily client-rendered site, the raw HTML often contains only:

• An empty container such as <div id="app"></div>
• Script bundle references
• Minimal metadata and boilerplate navigation
• No meaningful body content

Product descriptions, feature comparisons, documentation, pricing and FAQ answers then exist only in the rendered DOM after JavaScript runs. A crawler that doesn't execute JavaScript never sees them.

What the evidence shows

The most comprehensive public measurement is a December 2024 analysis by Vercel and MERJ. In one month across Vercel's network it recorded 569 million GPTBot requests and 370 million from Anthropic's crawler, and it found that none of the major AI crawlers it tested rendered JavaScript. That covered OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot.

Two details matter:

• ChatGPT's and Claude's crawlers do download JavaScript files (in 11.5% and 23.84% of requests) but don't execute them.
• Content that is already in the initial HTML response, such as embedded JSON, can still be read.

The exceptions are Google, whose Gemini uses Googlebot's rendering infrastructure, and Applebot, which renders pages in a browser-based crawler.

What is not measured. We haven't found documentation from any AI vendor describing JavaScript rendering for its crawlers, and Anthropic's Claude-SearchBot and Claude-User weren't listed in the 2024 study. The safe assumption is that they behave like the rest and read raw HTML only.

Which crawlers render JavaScript?

The Training vs. Retrieval Distinction That Changes Everything

AI crawlers are not one category. Each major vendor runs separate crawlers for three different jobs, and you can control them independently:

• Training crawlers collect content that may be used to train foundation models.
• Search crawlers build the index behind AI search answers and citations.
• User-initiated fetchers load a page live when someone asks a question or shares a link.

What the vendors' own documentation says:

• OpenAI: Each setting is independent. You can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot to keep content out of training. Sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links. ChatGPT-User isn't used for automatic crawling or to decide Search inclusion, and because its requests are user-initiated, robots.txt rules may not apply. Changes can take about 24 hours to take effect.
• Anthropic: Three bots, each with its own robots.txt entry. Restricting ClaudeBot signals that your future content should be excluded from training. Blocking Claude-SearchBot may reduce your visibility in Claude's search results, and blocking Claude-User stops user-directed fetches. Add rules for each bot and each subdomain. The older Claude-Web and anthropic-ai user agents are deprecated.
• Perplexity: PerplexityBot surfaces and links sites in Perplexity search results. Perplexity-User supports individual user requests. Settings work independently and can take up to 24 hours to apply.

The strategic implication. An operator who wants to appear in ChatGPT and Claude answers without contributing content to model training can allow the search crawlers and disallow the training crawlers. That is a real choice most teams haven't made deliberately.

1# Appear in AI search, opt out of model training2User-agent: OAI-SearchBot3Allow: /4 5User-agent: Claude-SearchBot6Allow: /7 8User-agent: PerplexityBot9Allow: /10 11User-agent: GPTBot12Disallow: /13 14User-agent: ClaudeBot15Disallow: /

If you want maximum presence, you don't need to add anything: a crawler with no matching rule is allowed. Explicit Allow: / lines are optional, but they make your policy easy to audit. Leave ChatGPT-User and Claude-User unblocked unless you have a reason, since they are how users' questions reach your pages.

Two more things worth knowing:

• The bigger risk is usually upstream. A WAF rule, CDN bot setting or rate limit that returns 403 or 429 to AI crawlers blocks them before robots.txt matters. OpenAI and Perplexity publish IP ranges you can allowlist.
• Google-Extended doesn't affect Search. It controls whether Google uses your content for Gemini, and it isn't a separate crawler. Google Search, AI Overviews and AI Mode follow Googlebot.

None of these crawlers, training or retrieval, has been shown to execute JavaScript. The rendering gap applies across all of them.

Why Google Rankings Don't Predict AI Citations

Technical SEO teams that have solved JavaScript rendering for Google can't assume they've solved it for AI crawlers. The two situations aren't equivalent.

Googlebot queues pages for its Web Rendering Service, runs them in an evergreen version of Chromium and indexes the result, with delays that can affect freshness and crawl budget. On March 4, 2026, Google removed the "Design for accessibility" section from its JavaScript SEO basics documentation, calling the advice out of date, and it has said it renders all pages it indexes. That change is accurate, and it applies only to Google.

AI crawlers are optimized for a different task: retrieving content quickly to inform an answer. In practice:

• A site that fixed rendering for Googlebot alone, using dynamic rendering (which Google itself calls a workaround), may still be serving AI crawlers an empty shell if they're not included in the rendering logic.
• A page can rank on the first page of Google and convert visitors while being absent from AI-generated answers.
• These two facts coexist silently, which is why the problem compounds before most teams notice it.

The Pages Most at Risk

Not all pages carry equal risk. The pages where rendering limits do the most damage are the ones with the highest citation value.

Product and feature pages. Product pages account for 20.1% of ChatGPT citations, according to Lantern's data. If their descriptions, specifications and pricing load client-side, the engine most likely to cite them can't read them. These are also the pages AI-referred visitors land on when they convert.

Comparison and alternative pages. Comparison content earns 9.3% of AI citations in Lantern's data and carries strong purchase intent. A comparison table rendered client-side is invisible to AI crawlers, so the content most likely to influence a buyer in active evaluation is the content most at risk.

Documentation and FAQ pages. AI engines favor structured Q&A content for informational and evaluation queries. FAQ and documentation pages that render via JavaScript underperform their citation potential however well they are written. Lantern's Site Audit identifies pages where JavaScript dependency is the likely cause of citation underperformance.

Category and landing pages. These establish the topical authority signals AI engines use to judge a brand's expertise. If their substance is client-rendered, that signal is weakened or absent.

Your highest-traffic pages. Vercel found that pages with higher organic traffic get more frequent crawler visits. Traffic only amplifies what crawlers can read, so if your busiest pages are client-rendered, volume doesn't turn into citations.

How to Test Whether AI Crawlers Can Read Your Site

Don't assume. Test with methods that approximate what AI crawlers actually receive.

1. View Source test. Open a critical page and choose View Source, not Inspect Element. Inspect shows the rendered DOM. View Source shows the raw HTML response. Look for your primary heading, body copy, product descriptions, FAQ content and internal links. If they're missing from View Source but visible on the page, you have a rendering gap.

2. cURL test. A cURL request returns the raw HTML the way most bots receive it on their first fetch:

1curl -sL https://your-domain.com/your-page | grep -i "your primary heading"

To see what a crawler-style request gets, send a crawler user agent as well. This confirms the content is in the HTML. It doesn't prove your firewall lets the real crawler through, because many WAFs verify crawlers by IP address:

1curl -sL -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" \2  https://your-domain.com/your-page | grep -i "your primary heading"

3. JavaScript-disabled test. Load the page with JavaScript turned off in your browser. If it becomes an empty shell or loses its substance, crawlers are likely seeing the same thing.

4. Ask an AI engine. Ask ChatGPT, Claude or Perplexity to read the URL and quote your H1 and first paragraph. A failure is informative. A pass is not proof, because some assistants use a real browser or a search index rather than a raw fetch.

5. Server log analysis. Look for visits from AI user agents and check the status codes and response sizes:

1GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User

Frequent visits to pages with thin raw HTML point to rendering-related visibility loss. Also watch for 404s. Vercel found ChatGPT and Claude spent roughly a third of their fetches on 404 pages, so keep sitemaps current and redirects clean.

6. Know which tests mislead. Google Search Console's URL Inspection and the Rich Results Test render JavaScript, so they show what Googlebot sees, not what AI crawlers see. JSON-LD injected client-side, for example by a tag manager, can pass the Rich Results Test and still be invisible to non-rendering crawlers.

Lantern's Site Audit surfaces rendering-related visibility gaps automatically. It identifies pages where JavaScript dependency is likely reducing AI crawler access and flags them alongside citation performance data, so the technical problem and the business outcome sit in one dashboard instead of across crawl tools, server logs and analytics.

The Technical Fixes

The goal isn't to abandon modern frameworks. It's to make sure the content AI engines need to cite you is delivered in a form they can read. A reliable rule: if the content matters for citation, trust or conversion, it belongs in the initial HTML.

Whichever you choose, check these too:

• Metadata, canonical tags and JSON-LD must be in the initial HTML, never injected by JavaScript.
• Navigation and internal links should be real <a href> links present in the HTML, including menus and pagination.
• Tabs, accordions and "load more" content that fetch after page load are invisible. Ship the content in the HTML and collapse it with CSS.
• Client-side rendering is still fine for non-essential elements such as view counters, chat widgets and interactive UI.
• robots.txt and firewall rules should let the search crawlers in. See the example above.

What Readable HTML Must Contain

Once the rendering architecture is right, the initial HTML for every high-value page should contain:

1. A primary H1 that makes a specific, descriptive claim, not a generic keyword.
2. An opening answer that directly answers the page's main question in roughly 40 to 60 words. In Kevin Indig's analysis of 1.2 million ChatGPT responses and 18,012 verified citations, 44.2% of citations came from the first 30% of a page's content. Give every section a self-contained answer before the supporting detail.
3. Core body copy with specific outcome language: numbers, named entities and concrete claims.
4. Comparison tables in HTML <table> format. Tables are easy for crawlers and models to parse. A table saved as an image is invisible, and comparison data written as prose is harder to extract.
5. FAQ sections with specific question-and-answer pairs, marked up with FAQPage schema.
6. Visible author and date signals: author name and credentials, publication date and last-updated date. These are direct trust signals for Google and AI engines.
7. Pricing context where appropriate, delivered in the raw HTML.
8. Internal links to related pages, including Lantern's AI Visibility monitoring, Citation tracking and Site Audit documentation, and related posts such as Site Audit Signals That Matter for AI Search and Why Your Old Content Is Losing AI Citations.
9. Canonical tag and metadata in the <head>, never injected by JavaScript.
10. Structured data (JSON-LD) as a supporting layer alongside visible text.

On schema markup: it reinforces visible content; it doesn't replace it. AI crawlers extract meaning from the visible text of a page first, so excellent schema over thin visible content is not a well-optimized page. JSON-LD in a <script type="application/ld+json"> tag in the server-rendered HTML is readable without executing JavaScript.

On llms.txt: as of September 2026 it has no demonstrated effect on AI citations. An SE Ranking analysis of about 300,000 domains found no measurable relationship between having the file and being cited, and none of the crawler documentation from OpenAI, Anthropic or Perplexity presents it as a citation signal. It's cheap to add and may matter later, but it doesn't substitute for readable HTML. Fix the rendering architecture first.

Measuring the Impact with Lantern

The rendering gap is invisible in most analytics setups. A page can look healthy in Google Search Console, ranking well and earning clicks, while being completely absent from AI-generated answers because its content is unreachable by AI crawlers.

Lantern's Search Console integration shows GSC performance next to AI citation data, so pages with strong Google performance but low AI citation rates, the first place to look for rendering gaps, stand out. AI citation monitoring tracks which pages are cited in ChatGPT, Perplexity, Gemini and Claude and compares citation rates with each page's rendering characteristics.

Two failure patterns emerge, and their fixes are completely different:

• Server-rendered pages that aren't being cited despite strong topical authority. This is a content or structure problem. Add answer capsules, FAQ sections, comparison tables and specific data.
• Client-rendered pages that aren't being cited. This is a rendering problem. Use SSR, prerendering or SSG.

Lantern's AI Visibility Score, a 0 to 100 measure of how prominently your brand appears in AI-generated responses, combined with the Site Audit's rendering gap detection, makes that diagnosis automatic.

If your AI Visibility Score is rising but AI referral traffic isn't, AI models are mentioning you without linking to you. That is a content and structured data problem, separate from rendering.

Frequently Asked Questions

Do AI crawlers render JavaScript?

Mostly no. Vercel and MERJ's December 2024 analysis found that OpenAI, Anthropic, Perplexity, Meta and ByteDance crawlers fetch raw HTML and don't execute JavaScript. Googlebot, which feeds Gemini, and Applebot do render it. Assume any other AI crawler reads only the initial HTML.

Does ChatGPT render JavaScript?

Its crawlers (GPTBot, OAI-SearchBot and ChatGPT-User) did not execute JavaScript in the 2024 measurement. ChatGPT agent mode can drive a real browser, but in one test 46% of its visits used a text-based reading mode. Put important content in the initial HTML.

Can Claude or ClaudeBot read client-side rendered pages?

Not reliably. ClaudeBot downloaded JavaScript files in about 24% of requests in the Vercel study but did not execute them. Anthropic doesn't document JavaScript rendering for Claude-SearchBot or Claude-User, so assume both read raw HTML only.

Why does my page rank on Google but never get cited by AI?

Google renders JavaScript and AI crawlers generally don't, so a client-rendered page can rank well while crawlers see an empty shell. Check with View Source or cURL. If your content is in the raw HTML, the cause is more likely content structure, authority or Bing indexing.

Should I use SSR, SSG or prerendering?

Use SSR for dynamic, content-heavy pages, SSG or ISR for stable content such as blogs and docs, and prerendering as a bridge for an existing SPA. Any of them works, as long as headings, body copy and structured data are in the first HTML response.

Should I block GPTBot and ClaudeBot?

That's a business decision about model training, separate from AI search visibility. Blocking GPTBot and ClaudeBot doesn't block OAI-SearchBot or Claude-SearchBot. If you want to be cited in AI answers, keep the search crawlers allowed.

Does llms.txt help AI crawlers read my site?

No. It doesn't fix rendering, and an SE Ranking analysis of about 300,000 domains found no measurable link between having llms.txt and being cited by AI. Fix the HTML first.

Key Takeaways

1. Most AI crawlers read raw HTML only. Vercel and MERJ measured none of the major ones executing JavaScript. Googlebot (and so Gemini) and Applebot are the exceptions.
2. Training, search and user-initiated fetching are separate jobs. You can control each in robots.txt, though user-initiated fetchers such as ChatGPT-User may not follow it.
3. Google's JavaScript progress doesn't transfer. Google dropped a JavaScript SEO section in March 2026 because Googlebot's rendering is mature. That applies only to Google.
4. ChatGPT search leans on Bing.
5. The highest-citation-value pages are the highest risk. Product pages (20.1% of citations in Lantern's data), comparison pages (9.3%) and FAQ and documentation pages are the ones most likely to be client-rendered.
6. View Source and cURL are the fastest diagnostics. If your content isn't in the raw HTML, assume it isn't being read.
7. SSR, SSG and prerendering are the fixes. Choose per content type, and keep metadata, JSON-LD and links in the initial HTML.
8. Lead with the answer. 44.2% of ChatGPT citations in Indig's study came from the first 30% of a page.
9. llms.txt has no demonstrated effect on AI citations. Fix rendering first.
10. Separate rendering problems from content problems. Lantern's citation monitoring and Site Audit show which pages need a technical fix and which need a content fix.

Lantern monitors your AI search visibility across ChatGPT, Perplexity, Gemini, and Claude, and connects citation performance directly to technical audit data. Start your free trial at asklantern.com.