Making Docs Discoverable to AILong read
Documentation Site Architecture for Agent Crawlability
Agents now consume 45% of documentation traffic—but most sites are architected only for humans.
Features Editor · · 9 min read
## Why Architecture Built for Human Readers Fails Agents at the Structural Level
The problem is not content quality. That distinction changes where you look for the fix.
No matter how well-written the documentation is, if an agent cannot access or parse the page cleanly, the content is invisible to it. The failure happens before a single word of prose is evaluated.
JavaScript rendering is the primary failure mode. Most modern documentation sites build their visible content client-side, which means a plain HTTP GET against those pages returns empty divs. Googlebot has developed reasonable JavaScript rendering capability over years of investment; most AI crawlers have not. Document360's getting-started page is instructive here: the HTTP response returned over a million bytes of HTML, but the stripped text content was 1,279 bytes, a text ratio of approximately 0.001, because the body of the article was never present in the HTTP response at all. The content exists for a human with a browser. For an agent issuing a GET request, the page is essentially blank.
HTML token overhead is a second structural problem, distinct from rendering, that surfaces even when content is successfully returned. A plain `##` heading in Markdown costs roughly three tokens. Its HTML equivalent, with wrapper elements, burns 12 to 15 before accounting for navigation bars, script tags, and layout chrome. Agents consuming documentation through constrained context windows pay for structure instead of substance. Every token spent on scaffolding is a token not spent on the API reference or the error message the agent was sent to find.
Human readers tolerate slow renders, collapsed navigation, and visual chrome. They scroll past it. Agents cannot skip or scroll; they process whatever the server returns, or they receive nothing at all.
These are not content problems. They are architectural decisions made without agents in mind, at a time when overlooking that consumer was entirely reasonable. Mintlify analyzed 790 million requests across its network over a 30-day period and found that 357.6 million, or 45.3%, came from AI agents. Claude Code alone generated 199.4 million requests, surpassing Chrome on Windows at 119.4 million. HUMAN Security's telemetry recorded verified AI agent traffic growing more than 6,900% year-over-year in 2025. A documentation site architected only for human readers is already failing a substantial portion of its traffic, and the proportion is growing. That time for overlooking agents has passed.
## What robots.txt Actually Controls, and What It Doesn't, When AI Crawlers Are Involved
The first thing practitioners tend to reach for when thinking about agent access is robots.txt. It is the right instinct, but the file is more nuanced in 2026 than most configurations reflect, and the gaps are compounding quietly.
Major AI platforms have separated their crawling activity into distinct bot identities corresponding to distinct purposes. OpenAI operates GPTBot for training data collection, OAI-SearchBot for search indexing, and ChatGPT-User for user-triggered retrieval. Anthropic runs ClaudeBot and anthropic-ai for training, with claude-web and Claude-SearchBot handling user-triggered requests. Google maintains Google-Extended specifically for AI training use, separate from its core crawling identity. Perplexity operates PerplexityBot alongside Perplexity-User. Common Crawl's CCBot feeds training datasets for many models.
The strategic distinction most configurations collapse: blocking training crawlers prevents content from entering model training. Blocking search and user-triggered bots removes content from eligibility for AI-generated answers and citations. Those are different outcomes with different business implications, and conflating them is expensive. Adding `Disallow: /` for Google-Extended blocks only training use by that specific crawler; core crawling is unaffected. The directive does exactly what it says, nothing more.
A CapstonAI audit of 86 B2B customers in Q1 2026 found that 41% still block at least one major AI bot, with each blocked bot estimated to cost 18 to 34% of potential AI citations on that engine. The prevailing posture among well-configured developer-facing sites is to block training crawlers and explicitly allow search and user-triggered bots. Most sites have not reached that posture yet.
The harder problem is compliance. robots.txt is a request, not a lock. Cloudflare's August 2025 report documented Perplexity using undeclared crawlers that rotate user-agents, IPs, and ASNs to evade directives. For non-compliant bots, WAF-level blocking is the only reliable defense.
One failure mode that receives less attention than it deserves is redirect chains. AI crawlers generally require a 200 OK within one to three hops. Chains optimized over years for Googlebot have rarely been tested against GPTBot or PerplexityBot, and chains that perform reliably for human traffic can silently block agent access at the network level before robots.txt even enters the picture.
## What llms.txt Is, What It Isn't, and Where Its Real Value Sits
An analogy that has proven useful: sitemap.xml is the full library catalogue; robots.txt marks the restricted shelves; llms.txt is the librarian's recommended reading list. It is a curated directory, not a content delivery mechanism. That distinction defines both the file's value and its limits.
The problem llms.txt addresses is real. LLMs face context window constraints that make full-site crawling impractical. HTML-heavy pages are expensive to parse and frequently return incomplete content. llms.txt provides a structured Markdown index of the most important pages, giving agents a starting point rather than asking them to navigate an entire site from scratch. Its companion, llms-full.txt, provides the complete site content in a single Markdown document for deep ingestion; most sites need only the index file.
Where the format demonstrably earns its place is in inference-time retrieval. AI coding tools like Cursor and Claude fetch documentation in real time during active sessions, and llms.txt helps them find the right pages with less token waste. Mintlify recorded 436 visits from AI crawlers after publishing their llms.txt, most from ChatGPT.
The adoption picture is worth examining carefully. As of July 2025, only 951 domains had published an llms.txt file per NerdyData. By early 2026, BuiltWith tracked over 800,000 implementations, but the vast majority are auto-generated by SEO plugins. Hand-curated implementations remain closer to the original count. Anthropic, Cloudflare, Stripe, and Mintlify are among the major publishers using it with apparent intent.
The significant unresolved question: as of Q1 2026, no major AI provider, not OpenAI, Google, Anthropic, nor Meta, has publicly confirmed that their production systems read or act on llms.txt. Google's position is explicit. Gary Illyes stated in July 2025 that Google does not support llms.txt and is not planning to; John Mueller compared it to the keywords meta tag. A counter-signal worth noting: Google included llms.txt in their experimental A2A protocol, suggesting interest at the research layer even while the search organization rejects it publicly. What to make of that tension is genuinely unclear, and anyone claiming certainty about it is probably overstating their knowledge.
The critical implementation caveat: llms.txt points to pages but does not change what those pages serve. If the linked pages still return JavaScript-rendered HTML, the navigation signal is sound but the content retrieval still fails. The directory is only as useful as the destination.
## Serving Markdown Directly to Agents Through Content Negotiation
If JavaScript rendering is the primary structural failure, Markdown delivery is the structural solution most worth understanding, because the performance difference is not marginal.
A typical article in Markdown compresses to 20 to 30% of its HTML size. HTML-to-text conversion is lossy; tables, code blocks, and footnotes frequently arrive mangled in downstream summaries. Research on this point is striking: GPT-3.5-turbo's accuracy on a code translation task swung by up to 40% based on input format alone, with no change to the underlying content. GPT-4 showed a consistent preference for Markdown, attributed to heavier pretraining on structured text. The content is the same; the container changes what the model can do with it.
Content negotiation is the mechanism. It is a 27-year-old HTTP standard in which clients declare their preferred format via the Accept header and servers respond accordingly. The same URL serves HTML to browsers and Markdown to agents: no separate URLs, no content duplication, no maintenance of parallel doc trees. When an agent sends `Accept: text/markdown`, the server returns clean Markdown. When a browser sends `Accept: text/html`, it gets the rendered page. Availability is advertised via `Link: rel="alternate"` response headers and `` tags in the ``.
A simpler alternative is the URL-level Markdown pattern: appending `.md` to any page URL returns clean Markdown. Mintlify's architecture already uses this. It is easier to implement, easier to test, and easier to document for developers integrating against your docs programmatically.
Cloudflare released a "Markdown for Agents" feature in February 2026 that handles HTML-to-Markdown conversion at the CDN edge, adding a Content-Signal header indicating permitted uses including training, search, and agent input. For teams without server-side control, this lowers the implementation barrier considerably.
One practical caveat: not all AI crawlers currently send `Accept: text/markdown`. The implementation should serve Markdown when the header is present and fall back gracefully to HTML. An implementation that breaks one population to serve the other has failed to solve the problem.
The connection back to llms.txt is direct. llms.txt provides the directory; content negotiation provides the actual content. Both are needed for the full retrieval path to function.
## How Semantic HTML Structure and JSON-LD Reduce the Work Agents Have to Do
Not every team can implement content negotiation immediately, and not every platform exposes server-side response logic. In those cases, semantic HTML structure and JSON-LD become the most practical near-term investments, with one important caveat on ordering.
When agents receive HTML, a clear heading hierarchy, h1 through h2 through h3, reduces token noise and helps them identify section boundaries without heuristic guessing. Flat `
` nesting with no landmark elements forces agents to parse visual layout instead of logical structure, the equivalent of reading a document with all formatting stripped and then trying to reconstruct its organization from indentation alone.
JSON-LD's role has shifted in 2026. Its primary value is no longer rich snippets in Google Search. It is machine-readable metadata that AI systems parsing a page can extract without reading prose. Relevant schema types for documentation include TechArticle, HowTo, and FAQPage. Structured data surfaces the topic, author, date, and relationships between pages without requiring the agent to infer them from body text. That inference is expensive and error-prone; the structured data removes the burden.
Here is the ordering problem that is easy to miss: well-structured semantic HTML is worthless if it never appears in the HTTP response. Semantic structure only helps after the JavaScript-rendering and content-negotiation problems are addressed. Teams sometimes reach for JSON-LD first because it feels tractable and concrete; it is, but the gains are invisible until the upstream access problems are resolved. The sequence matters more than most documentation on these tools acknowledges.
## Treating Agent Crawlability as an Ongoing Infrastructure Requirement, Not a One-Time Configuration
The temptation, once these changes are made, is to treat the problem as solved.
Bot identities, crawler behaviors, and format preferences are shifting on a months-long cadence. The robots.txt configuration that was correct in 2024 was causing citation losses by Q1 2026. The landscape with named bots from OpenAI, Anthropic, Google, and Perplexity organized into training, search, and user-triggered tiers did not exist in its current form two years ago. It will look different again, probably sooner than anyone expects.
Documentation drift is the recurring failure that catches teams after the initial configuration is complete. Agents that retrieved accurate documentation at deployment retrieve stale or contradictory information as the product changes. The architecture that serves content must be connected to the system that updates it. This is a synchronization problem, not a documentation quality problem, and it requires architectural solutions: Markdown endpoints, llms.txt, and sitemaps that stay current with page additions and removals automatically rather than through manual curation cycles that inevitably lag.
Crawl logs are the feedback mechanism most teams are not reading. Which bots are arriving, which pages are returning errors, where redirect chains are breaking: this information exists and is actionable, but only if someone examines it on a regular cadence. In practice, this almost never happens until something breaks visibly enough to prompt an investigation.
Agent crawlability is a property of the whole documentation system. Access policy, navigation signals, content format, semantic structure, and update cadence all interact, and a gap in any one of them degrades the others. No single file or setting produces or maintains it, and the teams that treat it as a checklist item rather than an operational discipline are the ones whose configurations quietly age out of relevance.
Sources
Filed underMaking Docs Discoverable to AI
