AI Crawler Behavior on Documentation Sites
Different AI crawlers serve different purposes, and blocking one doesn't block the others.
The first instinct, when someone mentions "AI crawlers," is to picture a single type of bot doing a single type of thing. That instinct is wrong, and acting on it is expensive. But the more disorienting possibility is that the crawlers you're blocking aren't even the ones causing the problem.
OpenAI alone runs three distinct crawlers. GPTBot handles bulk training data collection. OAI-SearchBot indexes pages for ChatGPT's search functionality. ChatGPT-User fires only when a human explicitly asks ChatGPT to retrieve a specific page, making it the most targeted of the three and the one most likely to produce a direct interaction with your actual documentation. A documentation team that blocks GPTBot to prevent training ingestion has not necessarily blocked OAI-SearchBot, and blocking both leaves unaddressed the live-retrieval scenario that ChatGPT-User represents. These are separate decisions with distinct consequences, and conflating them is how policies get written that solve the wrong problem.
Cloudflare's October to November 2025 data offers a useful sense of scale. Googlebot reached 11.6% of unique web pages in that period, more than three times GPTBot's 3.6%, and nearly 200 times PerplexityBot's 0.06%. The breadth gap between Google and everyone else remains enormous. But Googlebot carries a structural complication most other crawlers do not: it serves dual purposes, indexing for search and feeding AI training data. Google-Extended is the directive that controls Gemini training use specifically, but it produces no separate user-agent entry in server logs. An operator cannot verify from logs alone whether Google-Extended is being honored. For a documentation team trying to make principled decisions about AI training access, that is a genuine compliance gap, not a technicality.
Meta-ExternalAgent sits in an uncomfortable position for documentation owners: second-highest volume AI crawler in 2025, zero referral traffic sent. The crawl is pure extraction. Bytespider, operated by ByteDance and feeding TikTok, Douyin, and Toutiao, has a well-documented history of ignoring robots.txt directives entirely. It dominated AI crawler traffic share at over 40% in 2024, collapsed to under 3% after platforms began blocking it by default in mid-2025, then climbed back. That volatility makes it difficult to plan around and harder still to enforce against through conventional means.
Month-to-month ranking shifts among the major crawlers are significant enough to complicate any static strategy. ClaudeBot led GPTBot in AI bot HTTP requests in April 2026, 11.69% to 9.84%. By May 2026, GPTBot led ClaudeBot, 11.48% to 9.73%. The mix moves regularly. Any documentation policy built around a fixed threat landscape will need revision before it's fully implemented.
What AI Crawlers Are Doing with Documentation Content Once They Fetch It
The destination of fetched content determines almost everything about whether a crawl produces any value for the site being crawled. Cloudflare data covering the twelve months to July 2025 shows that 80% of AI crawling was for model training, 18% for search, and 2% for user-triggered actions. In the final six months of that window, the training share rose further to 82%, search dropped to 15%, and user-triggered actions edged up to 3%.
Training crawls generate no referral traffic. The content is consumed, synthesized into model weights or retrieval corpora, and the original page receives no visit from any human as a result. This inverts the traditional search-crawl bargain entirely: you allow crawling because crawling drives discovery, which drives traffic, which drives product adoption. That chain still holds for search crawlers. For the majority of AI crawling, it does not. Model training may still benefit the site indirectly — better AI answers mean better-informed users — but that benefit is difficult to measure and entirely outside the documentation team's control.
The user-action category, represented most clearly by the ChatGPT-User pattern, is the exception worth tracking carefully. It was the fastest-growing category in absolute terms: volume increased more than fifteen-fold from January through early December 2025. Cloudflare's data showed a weekly rhythm in that growth starting in mid-February 2025, consistent with school and workplace usage patterns. A human prompted the retrieval; a human is presumably reading the result.
Training crawls and retrieval crawls require different responses from documentation teams. A page structured for training ingestion, dense with information and long-form prose, is not necessarily the same as a page structured for real-time retrieval, where currency, version signaling, and structured metadata matter more. The mode of consumption shapes what "good documentation" actually means for the AI audience, and that audience spans purposes, timelines, and technical behaviors that resist reduction to a single profile.
The Crawl-to-Referral Gap and What It Costs Documentation Sites
Anthropic's ClaudeBot made 38,000 crawl requests per visitor referred back to origin sites in July 2025. In January 2025, that ratio was 286,000 to one. The improvement is real, but the gap is still vast. The median AI referral traffic rate sits at 0.005%, compared to Google's 22.82% referral rate. A difference of more than four thousand times. So what is actually being exchanged?
The cost side is less frequently discussed but equally concrete. WP Engine's 2025 Traffic Trends Report found that AI crawlers can consume up to 70% of the most costly hosting resources. A Fastly report documented a single fetcher bot making 39,000 requests per minute to one site at peak, producing what the report described as "DDoS-like effects even without malicious intent." A Clutch survey found 42% of small businesses reported performance or bandwidth strain from bot traffic in the last twelve months as of 2025.
Modern AI bots are more expensive to serve than traditional search crawlers on a per-request basis. They execute JavaScript, simulate user interactions, and follow deep link chains. Serving a JavaScript-heavy bot costs meaningfully more than serving a static HTML response. For documentation sites with extensive versioned content, deep navigation hierarchies, and dynamic rendering, the cost exposure scales with complexity. The sites most worth crawling, because they contain the most useful technical information, tend to be the most expensive to serve.
The Wikimedia situation illustrates the specific exposure for content-rich sites. A 50% surge in multimedia bandwidth from bulk image scraping shows how documentation sites carrying rich assets, API diagrams, code examples, embedded video, face disproportionate infrastructure strain relative to text-only pages.
None of this leads automatically to "block everything." A developer documentation site may actively want its content ingested by AI systems, because the downstream effect is better AI-assisted answers for developers using the product. The calculation depends on what the documentation is for, who benefits when it's accurate, and what the organization loses if it's wrong or missing from AI knowledge bases. The gap between crawl volume and referral value is real; what it means for any given documentation team resolves differently depending on those variables.
How AI Crawlers Actually Parse Documentation Pages, and Where They Fall Short
Vercel's analysis of developer documentation on its network showed AI crawlers frequently attempting to fetch outdated assets from static folders, generating high rates of 404 errors and redirects. Googlebot, by contrast, spends only 8.22% of fetches on 404s and 1.49% on redirects. Google has invested years in crawler efficiency. Most AI crawlers have not yet matched that maturity, and the documentation sites bearing that cost are doing so without receiving commensurate value.
The JavaScript rendering gap is the more fundamental limitation. ChatGPT and Claude do not execute JavaScript at retrieval time. For documentation that renders content dynamically, API references built from database queries, versioned content generated client-side, changelogs assembled from a CMS at runtime, a significant portion of the actual information may simply not exist in what the crawler sees. Even when a user asks one of these models for current documentation, immediate fetches are often absent from server logs, suggesting reliance on cached or training data rather than live retrieval. That raises an uncomfortable question: if the crawler sees the content, does it matter how well that content is written?
Crawl frequency correlates with existing organic search visibility, not documentation quality. Pages that already attract search traffic get revisited more often. Structured data changes this dynamic: sites providing richer, more complete structured information are crawled more often and more accurately. Explicit labeling of content types, version relationships, and page function helps AI systems identify what they're reading without inference.
Documentation sites occupy a strategically distinct position from media publishers on this question. A media publisher wants to protect ad revenue and has a clear incentive to minimize AI training access. A technology company may want its developer documentation ingested, because accurate AI answers about the product are a support and adoption benefit. The same crawler behavior means different things depending on the site's operational goals, and that distinction should drive the response rather than a blanket posture borrowed from a different industry.
Why robots.txt Is an Unreliable Control Mechanism for Documentation Teams
An arXiv study found AI-blocking by reputable sites rose from 23% in September 2023 to nearly 60% by May 2025, with those sites averaging 15.5 AI user-agent entries in their robots.txt files. More than 2.5 million sites had fully disallowed AI training crawlers as of August 2025. GPTBot appeared in 5.52% of DISALLOW rules in Q1 2026, making it the most-blocked AI crawler. The adoption of these controls is real. The enforcement is not.
robots.txt is a voluntary protocol. A 2024 Digital Content Next study analyzed 42 identified AI training crawlers and found only 8 consistently respected robots.txt directives. The rest ignored the file, spoofed user-agents, or parsed selectively. Cloudflare documented in August 2025 that Perplexity uses undeclared crawlers that rotate user-agents and IP addresses to circumvent robots.txt. Bytespider ignores Disallow rules entirely; the practical remedy is a 403 response at the server or WAF layer, before any content is served, because the file-based instruction reaches nothing that will act on it.
Maintenance is an underappreciated problem. Blocking all known AI crawlers in robots.txt required approximately 40 to 50 distinct user-agent entries as of early 2025, and new crawlers appear regularly with no public documentation. That file requires active curation to remain effective, and its effectiveness against non-compliant actors is zero regardless of how well it's maintained. A documentation team investing meaningful time in robots.txt hygiene is doing that work primarily for the compliant minority.
The Google bind deserves a separate mention because it shapes the robots.txt decision for almost every documentation site. Blocking Googlebot entirely to prevent AI training use also removes the site from search indexing. Google-Extended targets Gemini training specifically without affecting search, but because it produces no separate log entry, there is no server-side verification that it's being honored. Documentation teams are asked to trust a directive they cannot audit.
Among top news sites, 79% block AI training bots and 71% block AI retrieval bots via robots.txt. Documentation sites, many of which want retrieval access to work because it benefits their users, face a more granular decision than news publishers. Blocking everything is not the same as protecting the right things, and robots.txt cannot currently make that distinction reliably.
What Documentation Structure Actually Determines About AI Readability
The JavaScript rendering problem is the most direct structural consequence for documentation teams to address. If content only exists after JavaScript executes, most AI crawlers will miss it entirely. Server-side rendered or statically generated documentation is more reliably parsed. For API references, versioned docs, and any content assembled dynamically from a database or CMS, this is not a theoretical concern; it determines whether the content exists at all from a crawler's perspective.
Structured data is not just an SEO mechanism, though it functions as one. It directly affects crawl frequency and accuracy for AI systems. Explicit labeling of content types, the relationships between pages, and version metadata gives AI systems a way to identify what a page contains without inference. Per the Vercel analysis, as more structured fields are present, both crawl rate and crawl frequency increase. A documentation page that announces what it is, what version it covers, and when it was last updated is a more tractable object for an AI system than a page that requires inference to interpret.
Content freshness is a particular vulnerability. AI models may rely on cached or training data even when queried about current product versions. Documentation that signals its own version and last-modified date gives AI systems and their users a legible way to recognize when information may be stale. Version-aware structure doesn't guarantee accuracy; it creates the conditions under which accuracy is possible.
Flat, explicit prose outperforms layered navigation for AI consumption. A crawler that doesn't execute JavaScript and doesn't follow all redirect chains will retrieve more usable content from a page where the substance is directly in the HTML. Long navigation hierarchies that require interaction to expose content are obstacles for human readers and largely invisible to most AI crawlers.
The crawl-frequency correlation with organic traffic creates a structural equity problem for recently shipped features. New documentation pages, with little existing traffic, may be underrepresented in AI model knowledge simply because they haven't accumulated the visibility that drives revisits. This argues for active structured-data signaling as a way to accelerate discovery, rather than waiting for traffic momentum to develop organically. This gap may compound over time: a feature that launches with sparse documentation may remain underrepresented in AI systems long after the documentation is complete, simply because the crawl cycle never caught up.
The goals of AI readability and human readability are more aligned than they initially appear. Documentation that is structured explicitly, rendered in HTML, and version-aware is clearer for both audiences. The adaptations tend to improve human usability rather than compromise it.
The Emerging Permission and Compensation Layer Around AI Crawling
On July 1, 2025, Cloudflare announced it would block AI crawlers accessing content without permission or compensation by default, making it the first major infrastructure provider to take that position. New domains on Cloudflare are now asked at sign-up whether to allow AI crawlers. Permission is the starting point, not a setting buried in an opt-out process.
The architecture Cloudflare has introduced goes further than binary allow-or-block decisions. AI companies can now declare purpose within the system: training, inference, or search. Site owners can make granular decisions based on declared purpose, allowing retrieval bots while blocking training bots, for example. Cloudflare has also introduced a pay-per-crawl capability, formalizing what has been, for most of AI's training data collection period, an entirely uncompensated extraction process.
The practical implications for documentation teams are layered, and some are in tension. Blocking all AI crawlers protects bandwidth and avoids documented infrastructure costs. It also degrades the accuracy of AI systems that answer questions about the product, because those systems operate from incomplete or outdated training data rather than current documentation. Selectively allowing retrieval bots while blocking training bots is a more defensible posture for most developer-facing documentation, but enforcing that distinction reliably requires infrastructure support rather than a text file.
The direction of travel is toward infrastructure-enforced permission models supplementing, and in some cases replacing, the informal robots.txt social contract. The documentation team's relationship with AI crawlers will increasingly be managed at the platform layer.
Whether that shift resolves the fundamental problem remains unclear. Infrastructure-level controls are more enforceable than file-based directives, but they depend on AI companies accurately declaring purpose, and there is no established mechanism for auditing those declarations. The documentation team that invests in machine-readable structure and explicit metadata is at least working with a constraint that is real and addressable. Whether the permission layer matures quickly enough to make the rest of this tractable is, at this point, an open question.
