apprun.icuGuides

Guide · AI search & the web

How AI crawlers read your page: JavaScript, chunks and citations

AI crawlers are separate bots with separate purposes. The evidence that ChatGPT and Claude crawlers skip JavaScript is one dated measurement, not a rule. Check the HTML your server sends, then see what a text pipeline keeps, chunks and can cite.

By apprun.icuUpdated Read 9 minIncludes a free browser toolSources 9 cited
On this page
  1. The short answer
  2. What each named bot is for
  3. What is verified about JavaScript, and what is not
  4. How a typical pipeline turns HTML into chunks and citations
  5. View source and the rendered DOM are different things
  6. Check your page with the inspector
  7. What this cannot tell you
  8. Common questions
  9. Sources and further reading
General pipeline, not any vendor's internals: fetched HTML, main content, chunks by heading, then a cited anchorFetched HTMLview-source textMain contentkept by extractorChunks by headingwith heading pathCited chunkURL + #anchor
A general pipeline, labelled as such: no vendor in this review documents these steps as its own design.

What each named bot is for

Vendors document their crawlers separately, so "block AI bots" is too coarse a rule. The table gives each bot's purpose in the vendor's own words, with the robots.txt note the vendor gives. Where the cited page says nothing, the table says so.

Named AI bots, their documented purpose and robots.txt notes, as of 17 October 2026
BotVendorDocumented purposerobots.txt note
GPTBotOpenAITraining: "used to make our generative AI foundation models more useful and safe.""Each setting is independent of the others."
OAI-SearchBotOpenAISearch: "is for search."Search results can take "~24 hours" to adjust after a robots.txt update.
ChatGPT-UserOpenAIUser-triggered visits: "not used for crawling the web in an automatic fashion.""robots.txt rules may not apply", because a user starts the action.
ClaudeBotAnthropicCollects web content that "could potentially contribute to their training."Has its own product token.
Claude-SearchBotAnthropic"navigates the web to improve search result quality for users."Has its own product token.
Claude-UserAnthropic"supports Claude AI users."Has its own user agent. Anthropic says its bots, Claude-User included, respect "do not crawl" signals by honoring robots.txt.
PerplexityBotPerplexitySearch: "is designed to surface and link websites in search results on Perplexity.""It is not used to crawl content for AI foundation models."
Perplexity-UserPerplexityA fetch requested by a user."generally ignores robots.txt rules."

A rule for one token does not change another. Blocking GPTBot leaves OAI-SearchBot and ChatGPT-User under their own rules, because OpenAI documents each one separately. Decide which purpose you want to allow or limit, then write the rule for that token and no other.

What is verified about JavaScript, and what is not

The main evidence is a December 2024 analysis by Vercel with data from MERJ. Its central claim is that "none of the major AI crawlers currently render JavaScript" (Vercel, "The rise of the AI crawler", 17 December 2024). The primary data came from monitoring nextjs.org and the Vercel network. To validate the findings, the post also analysed two job-board sites. The finding describes those logs at that time, not the whole web.

The same post says the ChatGPT and Claude crawlers fetched JavaScript files but did not execute them. In its words, "they don't execute them. They can't read client-side rendered content." If your article text is added by a script, the response those crawlers received did not contain it, according to this measurement.

Three things are not verified. The three vendor pages cited say nothing about rendering, so that silence is not a yes or a no. None of the other sources reviewed here repeats or updates the measurement. The post also says AppleBot renders JavaScript through a browser-based crawler, similar to Googlebot, and that Gemini uses Googlebot's infrastructure, which it describes as enabling full JavaScript rendering.

Google's own documentation is narrower. It says that "Google Search runs JavaScript with an evergreen version of Chromium" and that "Googlebot queues pages for both crawling and rendering" (Google Search Central, "Understand JavaScript SEO Basics", updated 4 March 2026). That describes Google Search only. Google still advises server-side rendering or pre-rendering, because "not all bots can run JavaScript."

How a typical pipeline turns HTML into chunks and citations

No vendor in this review documents its internal pipeline. The steps below are a general retrieval model, labelled as such, and not any company's design. A common retrieval design, which no vendor here documents, does four things.

  1. Extract the main content. Remove scripts, styles, navigation and footers, then pick the block that looks like the article. Mozilla's Readability library, which powers Firefox Reader View, works by modifying the DOM it is given. It strips JavaScript and CSS, scores candidate blocks by tag and by class or id hints, and lowers the score of blocks with many links.
  2. Split by headings. Group paragraphs under the nearest heading and keep the heading path, such as "Refund policy > Refunds". Split long sections again by size.
  3. Summarise each chunk. A pipeline can store a short summary per chunk. An extractive summary copies representative sentences rather than writing new ones.
  4. Attach a citation. Link each chunk to the page URL. If the chunk sits under an element with an id, the link can carry that id as a fragment.

Worked example (invented page)

This page is invented for the example and is not from a real site.

<article>
  <h1>Refund policy</h1>
  <h2 id="refunds">Refunds</h2>
  <p>Refunds are issued within
     14 days of purchase.</p>
</article>
<div class="sidebar">Subscribe
  to our newsletter</div>

Under the general model, the sidebar drops out, because its class name matches "sidebar", one of the patterns Readability treats as unlikely to hold main content. The chunk becomes "Refund policy > Refunds: Refunds are issued within 14 days of purchase." Its citation candidate is the page URL with #refunds added. The 14 days are part of the invented example, and this is not the inspector's output for this page.

View source and the rendered DOM are different things

View source shows the HTML document the server returned. To open it, choose view page source from your browser's menu. The DOM in the DevTools Elements panel is the page after scripts have run and changed it. For a server-rendered article the two usually contain the same article text, though DevTools can also show markup that scripts added. For a client-rendered app, view source may hold only an empty container such as <div id="root"></div> and some script tags, while the Elements panel shows the full article.

That is why the AI Crawler HTML Inspector asks for the view-source HTML. A fetch without JavaScript receives something close to what view source shows. A bot can still get a different response from the same URL, depending on user agent, cookies or location. Pasting the outerHTML copied from DevTools tests the page that scripts built, which a non-rendering reader never sees. If the two versions differ in their text, the difference is the part a non-rendering reader misses.

Anchors matter for citations as well. The HTML standard resolves a URL fragment to "the first such element in tree order" with a matching id (WHATWG, HTML Standard, "Scrolling to a fragment", living standard). A text fragment needs no id. It is written #:~:text= and highlights matching text. MDN notes that when nothing matches, "the whole text fragment is ignored and the top of the document is linked", and that browser support varies (MDN Web Docs, "Text fragments").

Check your page with the inspector

Paste your page's view-source HTML into the AI Crawler HTML Inspector. It shows which text survives a non-rendering read, how that text would be split into chunks, a one-sentence summary for each chunk and which anchors could carry a citation. It runs in your browser.

Start with the render gap. If the article text is missing from the pasted HTML, the chunks will be missing it too, and no heading work fixes that. Then compare the main content the extractor kept with the text you meant to publish. Last, read the anchors. A chunk with no nearby id has to be cited by page URL or by a text fragment, and the fragment depends on browser support.

To see how the split changes with the chunk size, run the same text through the Chunking Strategy Comparator, and read the chunking strategies guide for the trade-offs. Share previews are a separate job: the Open Graph Generator builds the tags that control them, and citation does not depend on them.

For the rest of the checklist, the guide to making a page easy for AI answers to cite covers the twelve technical points, from headings and ids to robots rules.

What this cannot tell you

  • It does not fetch your page. You paste HTML, so it cannot see HTTP headers, redirects or what a particular bot receives.
  • It does not show what any AI system does with your page. It applies rules and a general pipeline. The December 2024 measurement is the only evidence on bot rendering here, and it is dated.
  • Its extraction is approximate. Mozilla says that Readability's own readability check "is likely to produce both false positives and false negatives". The inspector uses a Readability-style heuristic, so it can keep boilerplate or drop real content.
  • Its summaries are extractive. They copy sentences from a chunk and are not written by a language model.

Common questions

Do AI crawlers run JavaScript?

Not as a verified blanket rule. In December 2024 Vercel, with MERJ data, reported that none of the major AI crawlers it measured rendered JavaScript, and that the ChatGPT and Claude crawlers fetched JavaScript files without running them. The data came mainly from nextjs.org and the Vercel network, plus two job-board sites. Google documents that Googlebot renders JavaScript, and that statement covers Google only.

Should I block GPTBot to stop ChatGPT citing my page?

Not necessarily. OpenAI documents GPTBot as the training crawler and OAI-SearchBot as the search crawler, with ChatGPT-User for user-triggered visits. Each is controlled separately in robots.txt, so blocking GPTBot does not change the other two. OpenAI also says that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links." Search results can take about 24 hours to reflect a robots.txt update. Decide which purpose you want to allow.

Does Googlebot render JavaScript?

Google says Google Search runs JavaScript with an evergreen version of Chromium, and that Googlebot queues pages for crawling and rendering. Google also says not all bots can run JavaScript, so server-side rendering or pre-rendering for key pages is still a good idea. That guidance is about Google's crawler, not the AI bots.

Is Readability what ChatGPT or Claude uses to pick content?

No source here says so. Readability is an open-source Mozilla library used in Firefox Reader View, and it is a useful example of how extraction can work. The vendor pages reviewed describe their crawlers' purposes, not their extraction methods, so treat any link between the two as a guess.

Why does the inspector show little text when my page looks full?

The browser shows the DOM after scripts have run. If you pasted view-source HTML and the article text is added by a script, that text is not in the HTML the inspector received. That is the same gap a non-rendering crawler sees. Check whether the text is in the first response, and move it there if it matters.

Sources and further reading

Share this guide