Measuring How Well LLMs Understand Your Documentation

The intuition that language models should be good at reading documentation is understandable. LLMs are, at their core, language systems, and documentation is language. The assumption collapses under examination, and I've watched it collapse repeatedly in production systems that looked fine until they didn't.

Training data cutoffs mean that recent documentation, niche library references, or proprietary internal specifications may be entirely absent from a model's parametric knowledge. The model has never seen this information. When you ask it about a feature added last quarter, it is not recalling something it once knew; it is generating something that sounds plausible given surrounding context. That is a fundamentally different operation, and conflating the two is how teams end up debugging hallucinations six months after deployment.

Technical documentation compounds the problem structurally. Code blocks, API tables, version-specific caveats, cross-references between sections: these are exactly the elements that retrieval pipelines struggle to chunk and embed coherently. A table comparing parameter types across three SDK versions is not a paragraph. Treating it like one produces retrieval artifacts, and when the model receives a malformed chunk as context, it does not pause to flag uncertainty. It fills the gap, often with text that sounds authoritative because it has absorbed the register of technical writing during training.

There is what the model knows from training, parametric knowledge, and what it retrieves from the context window, retrieved knowledge. These two sources are not labeled differently in the output. A hallucination and a grounded answer can look identical on the surface, which is why evaluating only the final text is insufficient. You need to measure each stage of the pipeline separately, and most teams skip that step until something breaks publicly.

A 2025 MDPI study testing nine models, including GPT-4o, LLaMA 3 8B, and Gemini 1.0 Pro, found overall factual hallucination rates ranging from 48% to 82% on a public dataset. Those figures represent the baseline any evaluation framework has to improve upon. Roughly half to four-fifths of confident-sounding outputs may contain factual errors. That is not a model deficiency you can fine-tune away; it is a measurement problem.

How Documentation Structure Shapes What an LLM Can Retrieve and Use

Structure is not cosmetic. It determines what the retriever surfaces and how well the model can use what it receives. This is a point most teams reach only after debugging low evaluation scores and tracing the failures back to the documents themselves, not the model, which is a humbling place to arrive after you've already blamed the embedding model twice.

Clear question-and-answer format is the structure LLMs parse most reliably. In systems where retrieval logs are inspected, technical FAQs consistently appear among the most frequently surfaced source types, which aligns with how that format maps onto the prompt-response pattern the model trained on. Explicit, concise section headings create clean chunk boundaries; vague or marketing-oriented titles produce ambiguous cuts that bleed unrelated content into the same retrieved chunk. Imperative, direct phrasing retrieves and generates better than passive or heavily hedged prose.

Chunking strategy functions as a ceiling. A peer-reviewed clinical decision support study published in MDPI Bioengineering in November 2025 found that adaptive chunking aligned to logical topic boundaries achieved 87% accuracy compared to 13% for fixed-size baselines, a difference that was statistically confirmed. No amount of model sophistication recovers from fragmented or incoherent input. If the chunks are broken, the context is broken, and generation is unreliable before the model produces a single token.

For dense technical content where topics shift unpredictably within a section, semantic chunking is appropriate. For explicitly structured documentation, anchoring chunks to Markdown headers, HTML tags, or function definitions produces more reliable boundaries. Contextual retrieval, which prepopulates each chunk with its heading path and a short summary before embedding, makes chunks self-contained and meaningfully improves retrieval precision in practice.

When evaluation scores are low, the first question should not be whether the model is deficient. It should be whether the documentation is well-structured enough to retrieve from at all. I've been in enough post-mortems to know that question gets asked last, not first.

The Retrieval Layer: What to Measure Before the Model Generates Anything

Most RAG failures are retrieval failures. The model hallucinates not because it lacks knowledge but because the context it received was wrong, incomplete, or polluted with noise. Fixing retrieval is the highest-leverage intervention available, and it requires measuring retrieval separately from generation. These are not the same measurement, and treating them as one is where most evaluation plans go sideways.

Context precision answers whether the top-ranked chunks are actually relevant to the query. A retriever returning ten chunks where only two bear on the question has a context precision of 0.2; the model is working against substantial noise on every query. Low context precision is the proximate cause of most RAG hallucinations, because the model attempts to synthesize an answer from material that does not support one. The practical diagnostic is manual: inspect the top-k results for a representative sample of queries and count how many belong. This takes an afternoon and routinely surfaces more than automated metrics catch.

Context recall measures whether the retriever surfaces everything needed to answer correctly. This is the only core RAGAS metric that requires ground truth labels, which is part of why the golden dataset matters so much. Low recall means correct answers are structurally impossible regardless of how capable the generator is.

Normalized Discounted Cumulative Gain, nDCG, accounts for both relevance and ranking position, weighting documents that appear earlier in the retrieved list more heavily. Research in information retrieval has found nDCG correlates more strongly with end-to-end RAG quality than binary relevance metrics, because position in the context window affects how much attention the model allocates to different chunks.

Top-k is a tunable parameter with real consequences. Empirical testing across embedding models suggests that four to eight chunks is optimal for most production RAG systems. Above eight, faithfulness scores typically degrade as the model's attention dilutes across an increasingly large context. That finding has held up in practice often enough that I now treat eight as a starting ceiling and work down from there.

The Generation Layer: Faithfulness, Relevance, and the Difference Between Them

A model can produce output that is faithful but irrelevant, or relevant but unfaithful. These are not the same failure and they do not have the same fix. Treating them as one dimension is how evaluation frameworks produce confident-looking scores that miss half the ways a system can go wrong.

Faithfulness, sometimes called groundedness, asks whether every claim in the output derives from the retrieved context. Operationally, the method is to decompose the answer into atomic claims and check each against the source chunks. A fluent, confident-sounding answer that contradicts the documentation is a faithfulness failure. It is also the most dangerous failure mode for documentation-grounded systems precisely because nothing in the surface text signals that the model drifted. It looks fine. The customer reads it as authoritative.

Answer relevance is a different question. Does the answer address what was actually asked? An answer can be perfectly grounded in context and still respond to a tangential reading of the question. That is a relevance failure rather than a faithfulness one, and it requires different remediation.

Context utilization functions as a diagnostic signal connecting the two layers. When a model produces a correct answer but shows low utilization of the retrieved chunks, it may be drawing on parametric training data rather than the documentation. The answer might be right today, while the documentation is current. When documentation updates and the parametric knowledge does not, the answer becomes wrong. This is the drift problem, and it does not announce itself.

RAGAS documentation from 2025 suggests that scores above 0.8 on context precision and faithfulness indicate production-ready performance, with the caveat that optimal thresholds vary by domain. Useful reference points, not universal standards.

Three Measurement Methods and When Each One Applies

No single method covers all four evaluation dimensions. Knowing which tool fits which job prevents both over-reliance on cheap automation and under-reliance on human judgment. These are not philosophical positions; they are practical boundaries I've hit often enough to have opinions about.

Reference-based metrics, including string matching and embedding-based similarity, are reproducible, cheap, and automatable. They fit constrained outputs where a correct answer can be specified in advance: extracting a version number, a specific API parameter value, a policy date. Their limitation is categorical. They cannot detect a fluent answer that contradicts the source unless the contradiction also produces a string mismatch, which it often does not.

LLM-as-a-judge has become the dominant scalable approach for open-ended documentation QA evaluation. Pearson correlation coefficients as high as 0.85 have been observed against human judgments on extractive QA tasks, substantially outperforming exact match and F1 as evaluation signals. The approach scales to large query volumes and generalizes across answer formats. The failure modes, however, are documented and non-trivial. Position bias produces inconsistent scores depending on the order answers are presented, with GPT-4 showing roughly 40% inconsistency in some evaluations. Verbosity bias inflates scores for longer outputs. Self-enhancement bias adds a score premium when a model evaluates its own outputs. RAND Corporation researchers found that no judge model was uniformly reliable across benchmarks, with frontier models exceeding 50% error rates on advanced bias tests in the JudgeBiasBench evaluation from 2025. Mitigation requires rubric-based scoring that constrains evaluation to specific factual criteria and, ideally, ensemble or multi-agent evaluators to reduce single-model systematic bias.

Human evaluation is the ground truth layer. It cannot scale as a primary method, but it is indispensable for building and validating the golden dataset and for safety-critical dimensions where automated judges have been insufficiently calibrated. Its role is to anchor the other methods.

In practice, most teams doing this well blend all three: reference-based metrics for retrieval measurement, LLM-as-a-judge for generation quality at scale, and human review to catch systematic judge errors and maintain the golden dataset.

Evaluation Frameworks and Tools Available Today

RAGAS covers faithfulness, answer relevance, context precision, and context recall within a single framework. It can generate test questions from a user-provided corpus, which is useful when no ground truth dataset yet exists. Validation against human judgments, documented in the original RAGAS paper (arxiv:2309.15217), showed close alignment particularly on faithfulness and answer relevance. Its limitations include incomplete support for context-window scaling and limited out-of-the-box customization for domain-specific metrics.

ARES (Automated RAG Evaluation System) uses prediction-powered inference with multi-dimensional scoring. It requires approximately 150 human-annotated samples for calibration. The cost of entry is real but bounded, and the resulting scores tend to be better calibrated than uncalibrated LLM-as-a-judge approaches.

The RAG Triad, implemented in TruLens and subsequently maintained under Snowflake, introduced three metrics in 2023: context relevance, groundedness, and answer relevance. Each metric is computed automatically via LLM-as-a-judge, and the framework is designed for enterprises where ground truth datasets are limited or unavailable.

LExT, released in April 2025, introduces the Question Answer Generation score, focused on counterfactual stability and contextual faithfulness. It detects unsupported claims by verifying that outputs are deducible from the given context, which targets the drift problem more directly than frameworks built around standard QA accuracy.

For teams working with very long contexts, NoLiMa and HELMET, both released in April 2025, measure a model's ability to extract and reason over information in ultra-long contexts exceeding 128,000 tokens. Domain-specific benchmarks worth knowing include RAGBench, CRAG, LegalBench-RAG, WixQA, and T²-RAGBench, depending on the documentation domain.

Building the Golden Dataset That Makes All Other Metrics Meaningful

Every framework, threshold, and automated judge is downstream of the golden dataset. A weak dataset means no metric gives reliable signal, regardless of how sophisticated the tooling is. This is consistently the most underinvested piece of the evaluation stack, and teams that skip it carefully are the ones I've seen get burned.

The dataset should contain real production failures: actual queries that broke or nearly broke the system. These are more valuable than synthetic adversarial examples because they reflect how real users phrase questions, which is often not how an engineer would phrase them. An engineer writes a query like documentation; a user writes a query like a frustrated text message. Alongside real failures, the dataset needs deliberate edge cases: version-specific behavior, conflicting sections where the documentation itself is ambiguous, information that appears in multiple places with different phrasing. These are the cases where the model is most likely to produce plausible-sounding errors.

Fifty to one hundred well-labeled cases is a more useful starting point than a thousand cases with inconsistent labels. Labeling quality compounds; labeling quantity does not.

The labeling approach should match the output type. Where the correct answer is unambiguous, label with exact answers. Where the answer space is open-ended, use rubric-based scoring that specifies what criteria a response must satisfy. That rubric also serves as the evaluation constraint when LLM-as-a-judge is applied, which closes the loop between human judgment and automated scaling.

Version the dataset in the repository alongside the code. It is an artifact, not a spreadsheet, and it should be subject to the same change management. Guard against leakage: if the evaluation corpus overlaps with fine-tuning data, the scores are meaningless because the model has effectively memorized the answers. Update the dataset when the documentation changes. A static golden dataset evaluating a live corpus will drift out of alignment and produce scores that are technically high and operationally misleading, which is arguably worse than having no scores at all.

Putting It Together: A CI-Compatible Evaluation Loop for Documentation Changes

One-time evaluation is not enough. Documentation changes constantly; a model that was grounded last month may drift this month when a new section is added, a feature is deprecated, or a policy is revised. The evaluation cadence has to match the change cadence, and in most organizations those are not the same calendar.

Retrieval layer checks, covering context precision, recall, and nDCG, should run on every documentation update. These metrics can degrade without any change to the model if a new document creates retrieval competition or a rewritten section produces worse chunk boundaries. Generation layer checks, covering faithfulness, answer relevance, and context utilization, should run against the golden dataset on every model or prompt change. Thresholds function as gates: scores below 0.8 on faithfulness or context precision should flag a review before the change deploys.

The triggers for re-evaluation follow the change surface: any update to the documentation corpus, any change to retrieval configuration including chunking strategy, top-k, or embedding model, any change to the generation model or system prompt. These are not separate systems. They interact, and changes to any one of them can shift performance on any of the evaluation dimensions in ways that are non-obvious until you've been surprised by them.

Without a continuous evaluation loop, a team is operating on the assumption that the system is working. That assumption becomes harder to justify the more the documentation changes, the more diverse the user queries become, and the higher the stakes of a wrong answer. Air Canada's chatbot situation in 2024 did not involve a catastrophic technical failure; it involved a system producing confident, wrong information with no mechanism to detect the divergence before a customer relied on it. The evaluation loop described here is no guarantee against such failures. It is, however, the only structural approach that creates visibility into whether they are occurring, and visibility is what makes correction possible before the failure reaches someone who trusted the answer.