AI Crawler Behavior on Documentation Sites
Different AI crawlers extract value differently, and blocking one doesn't block them all.
Renata Haddad
Section
llms.txt
6 stories in llms.txt.
What llms.txt Is and Why It Exists
A plain-text file helps AI systems navigate the web more efficiently than traditional crawling.
Renata Haddad
Measuring How Well LLMs Understand Your Documentation
LLMs hallucinate on unfamiliar documentation, and measurement reveals why.
Renata Haddad
Versioned Documentation and LLM Context Accuracy
Stale training data makes AI confident about outdated APIs with no internal warning signal.
Renata Haddad
Structuring Documentation Content for LLM Comprehension
Organize documentation by how LLMs will chunk and retrieve it, not just for human readers.
Renata Haddad
Implementing llms.txt for a Documentation Site
A minimal specification for helping AI agents navigate your documentation efficiently.
Renata Haddad