Documentation Site Architecture for Agent Crawlability

Roughly half the traffic hitting documentation sites now comes from AI agents, not people. Cursor pulls an API reference mid-task, Claude Code checks a function signature, ChatGPT fetches a page in real time to answer a question. That shift changes what "good documentation architecture" means, and this piece walks through each layer of it, from robots.txt to heading tags, to lay out what agents need that human readers never did.

The shift wasn't gradual. It happened fast enough that most documentation sites were built for an audience that's already a minority of their traffic, and for API-first products, agent traffic sometimes beats human traffic outright. Cloudflare Radar data (arXiv:2604.02544) found something structural underneath that shift: agent visits compress what used to be a multi-page browsing session into one or two HTTP requests. A human reader clicks through multiple pages, hovers, scrolls, backtracks. An agent fetches a URL, reads the raw response, and moves on. There's no session to track, because there was never a session in the human sense to begin with.

That single fact breaks a lot of assumptions. Session depth, time-on-page, click paths, bounce rate: these metrics were built to measure human attention, and none of them register what an agent does when it grabs a page and leaves. Dachary Carey of MongoDB put it plainly in GitBook's State of Docs Report 2026: "Good for humans is not good for agents." The design choices that make documentation pleasant to read, progressive disclosure, collapsed sections, script-driven navigation, actively work against a system trying to pull structured facts out of a single HTML payload. That's the wrong tradeoff to keep making by default, and most teams are still making it, because nobody told them the audience had changed. Every section below covers one specific way agents diverge from human browsers in how they find, fetch, parse, and pull meaning from a page.

The crawler ecosystem documentation teams are now designing for

GPTBot showed up on August 7, 2023, and that date matters more than it looks like it should. It's the point where LLM publishers started giving site owners an explicit way to opt in or out, instead of scraping in the shadows. ClaudeBot followed in March 2024. OAI-SearchBot arrived that July alongside the SearchGPT prototype, and Anthropic formally documented Claude-SearchBot in February 2026. Four bots, four different jobs, and most documentation teams get their configuration wrong somewhere in the gap between them.

The growth curve alone should give pause. GPTBot's share of AI crawler traffic went from 5% to 30% between May 2024 and May 2025, and training crawlers now account for a substantial majority of AI-driven traffic by volume as of mid-2025. That's the majority of the activity, and most teams still lump it in with search indexing instead of treating it as its own category. That's backwards, and it's the mistake that costs the most later: a training crawler and a search bot do fundamentally different things with the same page.

Split the ecosystem into three groups, because each one relates to a documentation site differently. Training crawlers (GPTBot, ClaudeBot) absorb pages into model weights during a training run; whatever gets ingested becomes part of a model's baked-in knowledge, for better or worse, for a long time. Retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the citation index that answers pull from at query time, closer in spirit to a traditional search index. User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) only show up when an actual person asks an AI to go read a specific URL right now.

Compliance across these three groups isn't uniform, and that's the detail most configuration guides skip past. Anthropic states plainly that all three of its bots honor robots.txt, including the non-standard Crawl-delay extension, which is a firmer commitment than most competitors make. OpenAI's own documentation warns that robots.txt "may not apply" to ChatGPT-User on user-initiated fetches, so treat that as best-effort, not a guarantee. Perplexity has gone further and said Perplexity-User is "an agent, not a bot," and therefore isn't bound by robots.txt at all. Cloudflare backed that concern with evidence on August 4, 2025, publishing findings that Perplexity was running undeclared crawlers that rotate user-agents and IPs and ASNs specifically to get around no-crawl directives. That's not an edge case worth a footnote. A company got caught, on the record, dodging the one file the whole ecosystem is supposed to respect.

Underneath all of this sits a quieter problem: plenty of documentation infrastructure is blocking these bots by accident. Sites built or last audited before 2023 often run CDN configurations aggressive enough to treat OAI-SearchBot and PerplexityBot the same as malicious scrapers, denying them outright. Cloudflare shifted to a default block policy for AI crawlers on newly added domains in 2025, raising the stakes for anyone who hasn't checked their bot rules recently. Per Cloudflare data, GPTBot requests alone grew more than 300% as AI systems steadily take over a slice of web consumption that used to belong to humans. At that growth rate, an accidental block doesn't stay a minor gap. It compounds.

Carry this into the next section: robots.txt is a dependable signal for some bots, a best-effort signal for others, and effectively invisible to the ones that have already decided not to honor it. Architecture has to account for all three cases at once, not just the first one, and treating them as one problem is the error most teams make.

Writing a robots.txt strategy that distinguishes training from retrieval

Most documentation publishers land on some version of the same split: block the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) while leaving the retrieval and citation bots open (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot). That's the operational meaning behind "opting out of AI training" for most sites. It keeps a documentation set visible in AI answers and citations without feeding its content into someone's next training run.

Is that split actually the right call, though? Evidence from late 2025 complicates it. Brands that allow training crawlers may build a kind of long-term presence inside model weights, though the effect on citation rates in later generations remains an open question. Blocking training crawlers might protect content today while quietly reducing how often a brand gets recognized, and cited, tomorrow. Most teams don't weigh that tradeoff on purpose. They copy a robots.txt template off a blog post and call it a policy, defaulting to block-everything because it feels safer, when the bigger risk might run the other way entirely.

One failure mode hides in plain sight: redirect chains. To stay eligible for AI Overviews or Perplexity Answers, a URL needs to resolve to a 200 OK within a small number of hops. A redirect chain that a human browser handles invisibly, even just a few bounces before landing on the real page, can quietly break retrieval for an agent working under tighter latency tolerances. Nobody notices, because the human experience looks fine the whole time.

For crawlers that don't honor robots.txt at all (Perplexity-User being the documented case as of Cloudflare's August 2025 report), the file isn't a defense. Server-level controls, Cloudflare Bot Management, WAF rules, nginx deny directives, are the only mechanisms that actually stop unwanted access once a bot has decided the rules don't apply to it.

Before writing a single bot-specific rule, though, there's a more basic audit to run: raw HTML access, site structure, internal linking, log-verified crawlability. These determine whether a bot can read and revisit a site at all. Fine-grained allow and disallow directives mean nothing if the underlying content isn't reachable in the first place, which is really the throughline for the rest of this piece: robots.txt is one signal among several, and not the one doing the heaviest lifting.

What llms.txt does and does not do for documentation sites

Jeremy Howard of Answer.AI proposed llms.txt in 2024: a plain Markdown file at /llms.txt that gives a structured summary of a site's most important content, written specifically for large language models to read. Picture a documentation site as a library. sitemap.xml is the full catalogue, robots.txt marks off the restricted shelves, and llms.txt is closer to a librarian's recommended reading list, a curated path through the material rather than an exhaustive index.

There's a companion format, llms-full.txt, which dumps the entire site's content into one Markdown document for deeper ingestion. Most sites don't need it; llms.txt alone covers the intended use case. One small but real detail: the file has to be named exactly llms.txt, not llm.txt, or cross-platform tooling won't recognize it.

The strongest real-world case for llms.txt shows up in developer tooling. AI coding assistants like Cursor, GitHub Copilot, and Claude fetch documentation live, mid-task, and a well-structured llms.txt helps them grab the right page without burning tokens on irrelevant navigation chrome. That's a genuine, demonstrated benefit, not a theoretical one.

Adoption is uneven across the platform landscape. GitBook and Fern both support generating llms.txt. Yoast SEO and Rank Math added native generation in 2025, available as a toggle inside each plugin's dashboard. Other platforms in the documentation space still don't offer it, which limits how precisely models can cite content hosted there.

Now for the skepticism, and it's worth sitting with instead of glossing over. In July 2025, Google's Gary Illyes confirmed that Google does not support llms.txt and has no plans to. John Mueller compared it to the old keywords meta tag, a format search engines eventually stopped trusting. No major LLM provider, not OpenAI, not Anthropic, not Google, not Meta, not Mistral, has publicly committed to using llms.txt as a production signal for search or answer generation. In one monitored sample of AI bot visits, only a small fraction targeted llms.txt directly. The bots haven't caught up to the proposal, and there's no strong evidence they're going to.

There's also a hard limit that's easy to miss: llms.txt is a navigation aid, not an access control. It cannot block a crawler or stop any AI system from reading a site's content. Teams that install it expecting some kind of gatekeeping function will be disappointed, because that was never what the format does.

So the verdict here actually cuts against the current hype: build llms.txt for developer tooling workflows, where the benefit is proven, and stop there. Don't let it eat time that belongs to the structural and markup work covered in the rest of this piece. That work carries far more of the actual load, and teams that spend a sprint polishing llms.txt while their pages still render blank to a crawler have their priorities backwards.

URL structure, page depth, and site hierarchy as the foundation agents navigate by

Every important documentation page should sit within three or four clicks of the homepage. The pattern that works: Home, Category, Subcategory, Page. Once a URL nests deeper than that, something like /products/category/subcategory/option/page.html, crawl coverage of whatever lives down there becomes harder to guarantee. Depth carries weight. It's a tax on discoverability, and teams that let their docs sprawl five and six levels deep are paying that tax without knowing it.

URL hygiene matters more than it gets credit for. Keep URLs short, descriptive, and consistent with the content hierarchy they represent. Avoid parameter-heavy or dynamically generated URL patterns that fragment what's really one canonical page into a dozen slightly different addresses, because that fragmentation confuses both search indexing and agent retrieval. Breadcrumb navigation reinforces the hierarchy for human users and crawlers alike, and it maps directly onto BreadcrumbList schema, worth setting up correctly the first time rather than patching later.

One might treat hierarchy as a housekeeping concern, filed under tidiness rather than strategy, but the topical authority angle says otherwise. Organized, comprehensive coverage of a subject is generally easier for crawlers to index and retrieve than scattered pages that touch on it in passing. A documentation site that groups all its API reference material under one coherent structure is making an argument about trustworthiness, not just tidying up a sidebar. The clustering itself becomes part of the signal.

XML sitemaps function as the explicit map layered on top of all this. Include only indexable pages, keep the sitemap updating automatically as pages get added or retired, and for large documentation sets, split the sitemap by content type (API reference, guides, changelog) so crawlers can prioritize the sections that matter most for a given query. None of this replaces internal linking. It just gives crawlers an index to check their work against, and internal links are where the next section picks up, because they're not simply navigation. They're the paths agents actually follow.

Why the internal link topology of documentation is a retrieval signal, not just navigation

A May 2025 paper (arXiv:2605.07517) found something that should reshape how documentation teams think about internal linking. Standard embedding-based retrievers tend to treat structured technical manuals as flat collections of passages, ignoring the hyperlink topology that human readers actually rely on to navigate the same material. The links an author places between pages carry structural meaning, and most retrieval systems throw that meaning straight in the trash.

The paper's method, LARAG (Link-Aware RAG), tested this directly against a benchmark of twenty expert-designed queries over the Rulex Platform's technical documentation. It consistently produced better answers, hit the highest BERTScore F1 of the systems tested, and did it while retrieving fewer chunks and generating fewer tokens than a baseline retrieval architecture. Better answers with less computational overhead isn't a marginal result. That's the kind of tradeoff most engineering teams would kill for.

Why does this work? LARAG encodes the hyperlink relationships between pages as metadata attached to each chunk's representation, which gets something close to graph-based retrieval without the overhead of actually building a graph. The links were already sitting there in the HTML the whole time. LARAG just stops throwing that information away.

So are the internal links in a given documentation set placed as an afterthought, or engineered as a real signal? Because per this finding, they're not decoration. They're consumed by retrieval systems as semantic evidence, and well-linked documentation demonstrably produces better machine-generated answers, while poorly linked documentation produces worse ones even when the underlying content is accurate.

A few practices follow pretty directly from that. Link related concepts inside the prose itself, not only through sidebar navigation or top-level menus, since prose links carry more contextual weight. Use anchor text that describes the destination ("technical SEO guide" rather than "click here"), because the anchor text itself becomes part of the semantic signal a retriever picks up. Link outward from high-traffic or high-authority pages toward the pages that most need surfacing. Audit internal links on a schedule, because broken or redirected URLs don't just annoy human readers, they sever the graph a retrieval system depends on.

There's a preprocessing layer that amplifies all of this. Cleaning up document structure before chunking, detecting real headings, keeping section context attached to the text underneath it, stripping out navigational clutter, makes a well-linked topology work even better. Structure and links aren't separate concerns. They reinforce each other, which is exactly the argument the next two sections build on.

Server-rendered HTML and the rendering pipeline agents actually depend on

Here's the mechanical problem underneath everything else in this piece: AI crawlers and retrieval bots don't consistently run JavaScript. A documentation site that renders its content client-side, through a JS framework that builds the page in the browser after load, can look nearly blank to an agent reading only the raw HTML response. The page exists for a human. It might not exist at all for the bot, and no amount of good linking or clean headings fixes that if the content never arrives in the first response.

This connects straight back to the Cisco DevNet finding from earlier. Agents compress multi-page navigation into one or two HTTP requests because they're not opening a browser session, they're fetching HTML directly. If the content isn't present in that initial response, there's no second chance for it to load later. It simply wasn't retrieved, full stop.

Server-side rendering or static site generation solves this at the source, guaranteeing that the full content of every page exists in the HTML delivered on the first request, before any client-side script gets a chance to run. That's the difference between a page an agent can read and a page an agent sees as empty. It's not a small architectural preference. It's the whole ballgame for documentation platforms that lean hard on client-side scripting, and teams that treat it as a nice-to-have are the ones showing up empty in retrieval logs six months later.

A short checklist makes this concrete rather than abstract. Check that page content shows up in curl or wget output, not just in a browser with JavaScript turned on, since that's the closest approximation to what an agent actually receives. Make sure CSS and JavaScript resources aren't accidentally blocked in robots.txt, because crawlers use them to understand layout and avoid misreading content boundaries. Confirm navigation elements, code blocks, and headings appear in the raw HTML rather than getting injected after the page loads.

The redirect-chain problem from earlier resurfaces here too. A final URL that takes several hops to resolve risks timing out for an agent working under tight latency limits, and the 200 OK response needs to arrive fast, with the full content attached, not staged in afterward.

Log analysis is really the only trustworthy way to verify any of this. Check server logs for agent user-agents directly, and confirm they're reaching the intended pages and getting clean 200 responses back. CDN or WAF configurations that quietly return challenge pages or soft blocks won't always show up in standard analytics dashboards, so the logs are where the truth actually lives, not the dashboard.

Semantic HTML and heading structure as the parsing layer agents use to segment content

Agents don't skim documentation the way a person does, scanning for bolded terms or scrolling to the section that looks relevant. They parse the DOM looking for semantic boundaries, and heading hierarchy, h1 down through h2 and h3, is the primary signal telling an agent where one concept ends and the next one starts. Get that hierarchy wrong, and the agent's sense of where a topic begins and ends gets wrong right along with it.

This has a direct, measurable consequence at the chunking stage of retrieval. Preprocessing pipelines that detect real headings and keep the text underneath attached to its correct section produce meaningfully better retrieval accuracy than pipelines working against flat or malformed HTML. That's the implementation-level reason heading structure matters beyond just making a page look organized to a human eye.

A few specific practices follow, and teams that skip them pay for it downstream. Use exactly one h1 per page, matched to that page's primary topic or function. Nest h2 and h3 tags to reflect the actual structure of the content, not whatever visual size looks good in a given theme, because agents read the tag itself, not the font size it happens to render at. Don't use heading tags purely for styling, and don't skip levels: jumping from h1 straight to h3 without an h2 in between breaks the hierarchy an agent depends on. Keep the text right after each heading genuinely on-topic for that heading, since that's the exact context bundled with the chunk during retrieval preprocessing.

The same logic extends to code blocks, parameter tables, and API endpoint definitions. These belong in semantically correct elements (pre, code, table) rather than in generic styled divs that look the same to a human reader but carry no structural meaning to a parser. Agents built to read HTML semantics extract content correctly from the former and often mangle or miss it entirely in the latter. It's a small markup choice with a disproportionate consequence, and it's the kind of detail that never shows up in a design review but shows up constantly in a retrieval log.