Making API Reference Consumable by AI Agents

The Token Cost of HTML: What Gets Consumed Before an Agent Reaches Useful Content

A typical API reference page loads as HTML: navigation menus, styling markup, interactive playground scaffolding, sidebar chrome, footer links. The agent processes all of it, including everything it cannot use. The gap is not subtle. A page consuming roughly 10,000 tokens as HTML requires approximately 1,000 tokens as Markdown, a compression ratio of around 90 percent per page. But what if that ratio is only half the problem?

That ratio reflects a publication format decision with real consequences at scale. More endpoints fit within context limits; more can be held in parallel. The difference between an agent that understands three endpoints and one that understands thirty can come down entirely to source format, not API quality.

Documentation drift makes this worse than it first appears. A 2025 survey of 1,200 developers found 67 percent reported their API docs were out of date within 30 days of a release. So even when an agent parses a page cleanly, it may be parsing a stale contract. It is also worth considering that both problems, token bloat and drift, trace back to documentation built for static human consumption and maintained separately from the live API. Teams typically treat them as independent failures and pay for each twice.

How llms.txt Gives Agents a Navigable Entry Point into a Documentation Site

A plain Markdown file at a site's root, listing the pages that actually matter: core concepts, API references, authentication docs, policy pages. That is llms.txt. The sitemap analogy is imperfect but usable: it makes the right content findable without replacing the content itself.

Jeremy Howard proposed the convention in 2024. It is not an IETF or W3C standard, but adoption among developer-facing companies has been real and fairly rapid. Anthropic, Stripe, Vercel, Cloudflare, Cursor, Supabase, LangGraph, and Mintlify have all implemented it. The companion file, llms-full.txt, goes further, collapsing an entire documentation site into a single file an AI tool can ingest in one pass. For developer tooling contexts, that artifact is often the more immediately useful one.

Precision about the audience matters here. llms.txt is most valuable when developers actively point tools like Cursor, Windsurf, or Claude Code at documentation. This is developer experience infrastructure. The major AI search providers, OpenAI, Anthropic, Google, and Perplexity, have not officially confirmed their crawlers consume the file normatively. The value lives in the tools developers deliberately use, not passive discoverability.

What llms.txt leaves unsolved is the layer beneath it. It indexes pages; it tells an agent nothing about what the API accepts, returns, or requires. That raises an important question: if an agent can find the right page but cannot interpret what's on it, how much has navigation actually solved?

The OpenAPI Spec as the Machine-Readable Contract Everything Else Depends On

Navigation gets an agent to the door. The OpenAPI spec tells it what's inside. A complete OpenAPI 3.0+ spec encodes every endpoint and HTTP method, all parameters with types and constraints, request body schemas, response formats for every status code, and description fields at the endpoint, parameter, and field level. Not just structural metadata, but intent.

Spec quality has measurable downstream consequences. AutoMCP tested 5,066 endpoints and found an initial tool call success rate of 76.5 percent, with every failure tracing back to spec inconsistencies. After correction, success reached 99.9 percent. That 76.5 percent baseline sounds passable until you consider what the remaining 23.5 percent means when failures compound across thousands of calls. Why exactly does this happen? The mechanism is a direct line between spec completeness and call success, not a diffuse quality problem.

Realistic example responses matter more than most teams acknowledge. Generic placeholder values like "string" or 123 produce less accurate code generation than examples reflecting actual API output, because coding assistants use those examples to generate response-parsing code. The example is the signal, not ornamental scaffolding.

OpenAPI 3.2, released in September 2025, added first-class streaming support for SSE and JSON Lines, along with structured tag navigation for large API catalogs. Tools still operating on 3.0 or Swagger 2.0 lack these capabilities, and that gap grows as streaming and catalog scale become routine rather than edge cases.

The spec's role extends well beyond documentation. Google ADK's OpenAPIToolset generates callable tools directly from an OpenAPI 3.x spec. Microsoft Azure AI Foundry and Semantic Kernel both consume OpenAPI specs to wire APIs as agent tools. An incomplete or outdated spec doesn't just produce bad documentation; it produces bad agents operating with confidence, at scale.

Content-Level Choices Inside the Spec and Docs That Determine Whether an Agent Reasons Correctly

Terminology consistency is probably the most underestimated problem in documentation meant for agent consumption. When "API key," "access token," and "auth credential" appear interchangeably across a documentation set, an agent builds probabilistic guesses about their relationships. Those guesses produce incorrect behavior not occasionally but systematically, because the inconsistency is present everywhere the agent looks.

Heading hierarchy shapes how agents build content maps. A skipped level, an H1 jumping directly to H3, breaks the model's understanding of structural relationships and causes it to retrieve irrelevant sections. This is a structural signal, not a formatting preference.

Code block discipline follows the same logic. A human skims paragraphs to find a function signature; an agent looks for exact structural matches. A signature buried in a sentence may be missed entirely; the same signature in a dedicated code block with clear arguments and return values is reliably parsed. The difference between content an agent can act on and content it must reconstruct from context often comes down to that single formatting decision.

Intent-rich descriptions are what allow an agent to choose correctly between multiple options or navigate conditional logic. Documenting what a parameter does is necessary. But how does this affect our original promise of enabling sound reasoning? Documenting why an endpoint exists, and when it should be preferred over an alternative, is what enables reasoning rather than pattern-matching on surface features.

Multi-step workflow documentation requires that chains be made explicit. A refund flow may require authenticating, retrieving an order, checking eligibility, and calling createRefund with the correct orderId, reason, and amount. Without explicit chain documentation, the agent reconstructs the flow from first principles, and first-principles guessing against underdocumented state machines fails in predictable ways.

Machine-readable rate limit and deprecation signals close the loop at the response layer. Agents retry blindly against limits when X-RateLimit-Remaining and X-RateLimit-Reset headers are absent. Deprecated endpoints should be marked in the OpenAPI spec with migration hints surfaced in error responses; otherwise, agents trained on old documentation generate calls to routes that no longer exist, with no mechanism to self-correct.

There is also a maintenance cost that rarely gets priced in. When an API change requires simultaneous updates across HTML docs, Markdown files, code samples, and machine-readable indexes, something falls out of sync. In practice, it happens more often than not. A single source of truth is not an architectural ideal; it is the only arrangement that structurally avoids eventual inconsistency.

What MCP Adds on Top of a Well-Documented Spec, and What It Doesn't Replace

A complete OpenAPI spec makes an API agent-readable. MCP makes it agent-connectable at runtime. They solve different problems at different layers, which is why treating them as alternatives tends to go badly.

What an MCP server provides is runtime tool discovery with typed schemas. The agent doesn't generate HTTP requests from documentation; it connects and discovers available tools directly. When a developer asks a coding assistant how to authenticate, the assistant queries the MCP server and returns the current answer from live documentation, not a cached or hallucinated one.

MCP adoption moved faster than most developer infrastructure standards manage. Anthropic open-sourced it in November 2024; by March 2026, the project reported 97 million monthly SDK downloads. The React npm package took approximately three years to reach 100 million monthly downloads. MCP reached comparable scale in 16 months. Stacklok's 2026 software report found 41 percent of surveyed organizations already running MCP servers in limited or broad production.

In December 2025, Anthropic donated MCP to the Linux Foundation's Agentic AI Foundation, with backing from AWS, Google, Microsoft, OpenAI, Bloomberg, and Cloudflare. The governance structure signals durability; this is no longer a vendor project with a single point of failure.

The sequencing error teams make most often is trying to shortcut the spec to reach MCP faster. One might argue that a live runtime connection compensates for a weak spec — but the spec should come first. MCP follows naturally from a well-structured spec; building a runtime interface on top of a weak one produces a connected agent operating on unreliable information. A live connection to bad information is not better than a cached connection to bad information.

Skill files sit alongside MCP as a complement: Markdown documents encoding authentication flows, known pitfalls, and endpoint quirks that an agent fetches at task start to self-configure. They address the exploratory failure that comes from learning an API's idiosyncrasies through failed calls rather than documented guidance.

RAG as a Retrieval Architecture When Docs Are Too Large to Load in Full Context

Even with Markdown compression and llms.txt indexing, large API catalogs cannot fit in a single context pass. Retrieval becomes necessary, and semantic search with RAG lets an agent find relevant documentation based on intent rather than keyword matching, which matters when the query and the documentation share no obvious vocabulary.

Organizations using RAG report meaningful reductions in hallucinations and better answer accuracy for domain-specific questions. The precise figures vary by implementation and measurement approach, but the directional consistency across studies is strong enough to treat as a reasonable prior.

RAG's primary failure mode is less understood than it deserves to be. When naive RAG pipelines fail, the failure occurs at retrieval far more often than at generation. Teams that instrument generation quality but not retrieval quality are measuring the wrong layer; they optimize the generator while the retriever surfaces irrelevant content. The instrumentation gap is, in practice, more common than the architectural gap.

Agentic RAG addresses the static workflow limitation by embedding autonomous agents into the retrieval pipeline itself. Rather than running a single retrieval pass, the agent retrieves, reasons, and re-retrieves, enabling multi-step reasoning and adaptive behavior. Singh et al. explore this architecture in detail in research published on arXiv (arXiv:2501.09136, updated April 2026).

The prerequisite for RAG to work at all is clean, structured source content. RAG retrieves from what exists; if the underlying documentation is HTML-heavy, terminology-inconsistent, or schema-incomplete, retrieval surfaces bad material more efficiently. Investing in RAG to compensate for weak documentation quality misunderstands the dependency. Retrieval is a multiplier on what's already there, not a corrective for what is missing.

Keeping Agent-Consumable Docs Accurate as the API Changes

All the structural work described above degrades the moment the API changes and documentation doesn't follow. The 67 percent figure, docs out of date within 30 days of a release, is not primarily a discipline problem. It is a structural one. When documentation is maintained separately from the API contract, drift is the default outcome, not the exception.

To understand why this works, we must first look at what living documentation actually requires in practice: treating the OpenAPI spec as the authoritative source rather than a downstream artifact of narrative docs. It means regenerating llms.txt and llms-full.txt on deploy rather than maintaining them manually. Deprecation markers and migration hints belong in the spec and in error responses, propagated automatically. MCP servers should read from the live spec, not a cached snapshot.

When these layers connect to a single source of truth, a product change propagates through the entire agent-consumable stack: spec, discovery index, runtime tools. When they don't, parallel edits across disconnected formats make it likely that at least one copy is wrong at any given time.

Structure and currency are both required. A well-structured, stale spec produces confident, wrong behavior from an agent, consistently. That is the failure mode hardest to catch and easiest to ship.