Implementing llms.txt for a Documentation Site

The specification is deliberately minimal; depending on what you expected it to do, that's either its greatest strength or its primary liability.

The file lives at the root path of a domain, typically https://example.com/llms.txt, though subpath placement is permitted. Format is Markdown, a choice that reflects who will actually read it: language models and agents, not parsers expecting XML or JSON.

Required sections proceed in a defined order. An H1 heading carries the project or site name and is the only element the spec treats as truly mandatory. A blockquote immediately following provides a short project summary, supplying the contextual grounding a model needs to interpret everything downstream. Below that, any number of non-heading Markdown blocks may offer additional context. The substantive content lives in zero or more H2-delimited sections, each functioning as a file list: Markdown entries in the form name, optionally followed by a colon and a short description.

One section heading carries special semantic weight. Any H2 titled "Optional" signals that the URLs listed beneath it can be skipped when an agent needs to conserve context. This is the specification's primary mechanism for communicating priority to downstream agents, and implementers who treat it as a miscellaneous catch-all forfeit the most useful tool the format provides.

The spec also proposes a companion convention for individual pages: exposing a clean Markdown version at the same URL with .md appended. A page at /docs/authentication would have its counterpart at /docs/authentication.md. Agents retrieving linked pages prefer clean Markdown over HTML; the .md convention eliminates lossy HTML-to-text conversion at the page level rather than only at the index level.

Description quality is where most implementations quietly fall apart. The one-line description following each URL is not a marketing phrase, and it should avoid verbatim repetition of the page title. Its job is to tell a model what the page covers and, more usefully, when to retrieve it. "Core API authentication methods including API keys, OAuth 2.0 flows, and token refresh behavior" is actionable; "Learn about authentication" is not.

Deciding What to Include: Curation Principles for the File List

A documentation site may have hundreds of pages. The llms.txt index is only valuable if an agent can use it to identify the right page quickly without fetching everything. The curation decisions made during authoring determine whether the file is useful or merely present.

Primary sections should carry the pages an agent will reach for most often: what the product is, how its architecture works, the foundational mental models a developer needs before specifics make sense. API reference pages, stable endpoint documentation, getting-started guides, authentication docs, and configuration references all belong here because questions about credentials, environment variables, and setup sequences recur constantly in agentic workflows.

The "Optional" section is a deliberate second tier, not a dumping ground. Changelog entries, edge-case integration guides, deprecated feature documentation, and specialized tooling docs belong there. Stripe's published llms.txt places Stripe Climate documentation under "Optional," which is a sensible editorial call: most developers interacting with an AI assistant about Stripe are working with payments, not carbon removal. That kind of judgment, informed by knowing who actually uses your docs and why, is what separates a file that helps from one that merely exists.

Some pages should be omitted entirely. Navigation pages and category landing pages containing no substantive content add tokens without adding information. Pages that duplicate content covered more completely elsewhere create retrieval ambiguity. Marketing-oriented pages introduce noise.

The H2 grouping structure should mirror the logical organization of the documentation as a developer experiences it, not the site's navigation hierarchy. Cloudflare's published llms.txt organizes by product, a reasonable approximation of this principle for a platform with many distinct services: an agent helping a developer with Workers has no reason to load Cloudflare for Teams sections to answer its immediate question.

Keeping the Index File Within a Workable Token Budget

Token cost is real and immediate. Every URL and description in the index consumes context window space when an agent loads the file, and that space comes directly at the expense of context available for the developer's actual question and the retrieved page content. Most teams fail to think about this until they've shipped a file that is three times too large.

Turborepo's llms.txt has been widely cited as a cautionary example: at 116,000 tokens, it represents what happens when curation discipline is absent. A practitioner-recommended ceiling sits around 3,000 tokens for the index file; Anthropic's own llms.txt covers their entire API documentation at 8,364 tokens, a reasonable upper bound for a large, complex documentation site. The gap between those two reference points is instructive, and not in the way some teams interpret it. Bigger sites do not simply get to go bigger. 116,000 tokens is a documentation failure, not a documentation decision.

Draft the file, run it through a tokenizer before publishing, and trim from the least impactful elements first. Descriptions are usually where to cut: one tight sentence, not two. Borderline pages should move to "Optional" before being removed entirely, because retaining them in a lower-priority tier preserves access when they are relevant while reducing their token cost at the index level. The index file's job is navigation; content lives in the linked pages and, for sites that ship it, in llms-full.txt.

When and How to Add llms-full.txt Alongside the Index

The two-file architecture reflects a real tradeoff. Fetching individual pages requires multiple round trips; loading full content upfront is token-expensive and may exceed practical limits for large sites. llms.txt and llms-full.txt serve opposite ends of that tradeoff, and which you need depends on what kind of site you're running and who its AI readers actually are.

llms-full.txt embeds the actual content of the linked pages so an agent can ingest everything in a single fetch, without following links. For a small marketing site or a simple blog, shipping only llms.txt is defensible; for documentation sites, API references, and technical knowledge bases, the case for shipping both is strong: a developer's AI assistant answering technical questions needs content, not just a map.

The scale differential is instructive. Anthropic's llms-full.txt is 481,349 tokens, roughly 57 times the size of their index file. That ratio illustrates how much content the companion file is expected to carry, and why token discipline in the index matters: a bloated index compounds when the full file is 57 times its size. Per data from Profound, AI agents visit llms-full.txt at more than twice the rate of llms.txt on documentation sites where agents are the primary consumer; in practice, the full file is often the higher-traffic artifact.

Curating llms-full.txt should apply the same prioritization logic as the index. Avoid embedding every page; limit to highest-value pages; resist the temptation to treat the file as a complete site dump. NVIDIA's approach illustrates one clean solution to the scoping problem: separate files for technical documentation and for the main site, with distinct index and full-content files for each, matching different audience needs rather than collapsing everything into a single artifact.

Placing and Serving Both Files Correctly

Both files belong at the domain root: /llms.txt and /llms-full.txt. Subpath placement is permitted by the spec but reduces discoverability for agents that look only at the root. The cost of placing them correctly is zero.

Serving failures are silent, which makes them worse than obvious failures. The file exists; agents cannot use it; nobody notices until someone asks why their AI assistant is returning garbage. Files must be served as plain text with UTF-8 encoding, no authentication gate, correct MIME type (text/plain or text/markdown), over HTTPS. For teams building on Docusaurus and similar static site generators without built-in support, placing both files in the static/ directory is the simplest correct approach: the build pipeline copies them to the output root unchanged, no templating required.

Where pages already expose .md versions, the URLs in llms.txt should point to those Markdown versions. Agents prefer clean Markdown over HTML, and pointing to the .md URL delivers exactly what the spec's companion convention is designed to provide.

One interaction that catches teams off guard: llms.txt is not governed by robots.txt, but it can be blocked by it. If robots.txt rules targeting AI bots also cover the llms.txt path, the coding assistants and MCP clients that actually use the file will be unable to fetch it. The bots most teams are trying to stop with aggressive robots.txt rules are search crawlers and scraping bots, not the inference-time agents that llms.txt was designed to serve. Blocking the file to stop one class of bot inadvertently disables it for the other.

llms.txt also goes stale. A documentation site that ships features regularly will have sections restructured, pages moved, and URLs changed; a file pointing to dead endpoints sends agents to dead ends with apparent authority, which is worse than having no file at all.

Automating Generation and Keeping the Files in Sync with Docs Changes

A docs site that ships features weekly will have an llms.txt that diverges from reality within weeks if updates are manual. Teams that generate the file once and move on will end up with a file that lies. The typical timeline before someone notices something is wrong is about two months, and by then the damage is diffuse enough that tracing it back to the stale index is difficult.

Platform-level tooling has emerged as the most reliable solution for teams on managed documentation hosts. Mintlify generates both llms.txt and llms-full.txt automatically for every hosted documentation site. GitBook added llms.txt support in January 2025, followed by llms-full.txt and per-page .md support in June 2025, both requiring zero configuration. Fern, Vercel Docs, and Supabase Docs now ship the file by default. That list is a signal about where documentation tooling is heading, not an exhaustive inventory.

For teams building on static site generators without built-in support, the correct approach is a build-time script that reads the docs content tree and generates llms.txt from it, regenerated on every build and never edited manually. The script's inclusion criteria, which pages qualify for primary sections, which for "Optional," which are omitted, should live in a configuration file that content owners can adjust without touching the generator logic. This separation keeps the tooling stable while keeping editorial decisions accessible to the people who should be making them.

The most defensible CI/CD integration adds a validation step that checks the generated file's token count against a defined ceiling and fails the build if it exceeds the threshold; this enforces curation discipline automatically, rather than relying on individual contributors to remember the token budget during a documentation sprint when they are thinking about seventeen other things.

Where llms.txt Actually Gets Used Today, and Where It Doesn't

Most published tutorials skip this entirely; skipping it leads teams to implement the file for the wrong reasons and then measure its success against outcomes it was never designed to produce.

Analysis of over 500 million LLM bot traffic events found that AI search crawlers, including GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and Google-Extended, almost never fetch /llms.txt. They crawl HTML directly. The mechanism llms.txt relies on, active retrieval at inference time by a reasoning agent that reads the index and follows links, is not how these crawlers operate. They are simply the wrong audience.

Google's position was confirmed by Gary Illyes in July 2025: Google does not support llms.txt and has no plans to. John Mueller's comparison to the discredited keywords meta tag is pointed and, so far as the evidence goes, accurate. The file carries no weight in search, no effect on AI Overviews. An SEO rationale for implementing it is unsupported by data.

SERanking's analysis of 300,000 domains, published in November 2025, found that having an llms.txt file did not measurably correlate with being cited by ChatGPT, Claude, Gemini, or Perplexity. More pointedly, an XGBoost model trained on AI-citation data performed better when the llms.txt variable was removed entirely, suggesting the file's presence is not even a weak positive signal in citation prediction. The intuitive assumption is that structured, machine-readable documentation should help a model find and cite you; the data do not support that assumption, at least not for citation in search-adjacent systems. That tension is worth sitting with before committing to an implementation rationale.

Where the file does work is narrower but real. AI coding assistants, specifically Claude Code, Cursor, and Codex, explicitly fetch llms.txt to understand project documentation at inference time. This is the use case the format was designed for, and it delivers. MCP servers that use the file as a structured manifest for real-time knowledge retrieval are another reliable consumer. Profound's data indicates that Microsoft and OpenAI crawlers are actively fetching both llms.txt and llms-full.txt, suggesting the agentic pipeline, distinct from the search pipeline, does consume it.

Implement llms.txt for the developer using Cursor or Claude Code to integrate with your product; not for organic search visibility. These are different systems with different reading behaviors, and conflating them produces misplaced effort and, eventually, confusion about why nothing moved.

Connecting llms.txt to MCP Servers for Agent-Accessible Documentation

The Model Context Protocol, introduced by Anthropic in late 2024, reached over 1,000 community-built servers within its first six months. The relationship between that ecosystem and llms.txt is worth understanding before committing to a documentation architecture, because the two are not alternatives; they address different layers of the same problem.

The emerging pattern is to use llms-full.txt as a live knowledge source that an MCP server exposes, so AI assistants can query documentation content in real time rather than relying on a pre-fetched snapshot. In this configuration, llms.txt becomes a programmatic manifest that gives developers structured, auditable control over what an agent retrieves. That is a meaningfully different posture than passive hope that a crawler finds the file and interprets it sensibly.

LangChain's published implementation illustrates this concretely: llms.txt provides full documentation index discovery, while two complementary MCP servers let AI applications query LangChain content in real time. The files handle discovery; the servers handle retrieval. Mintlify now provisions MCP servers alongside automatic llms.txt and llms-full.txt generation. GitBook auto-generates MCP servers for published documentation alongside its llms.txt support. These are not isolated experiments; they reflect a consistent architectural direction across multiple tooling vendors.

The files and the server are not redundant. Files handle passive discovery, enabling an agent to understand what exists and where; an MCP server handles active, structured retrieval, enabling an agent to query specific content with precision at runtime. A team that ships both is matching the architecture to how the most capable AI agents actually work. The file alone represents only half of the infrastructure the ecosystem is converging on, and recognizing that early costs considerably less than retrofitting it later.