AI Crawler Behavior on Documentation Sites
Different AI crawlers serve different purposes, and blocking one doesn't block the others.
Renata Haddad
Contributing Editor
Renata Haddad
Renata Haddad covers llms.txt and docs as code for Docs As Code.
7 stories
What llms.txt Is and Why It Exists
A plain-text convention helps AI agents find what matters on websites, not just what exists.
Renata Haddad
Measuring How Well LLMs Understand Your Documentation
Most LLM documentation failures trace to retrieval problems, not model deficiency.
Renata Haddad
Versioned Documentation and LLM Context Accuracy
Renata Haddad
Structuring Documentation Content for LLM Comprehension
Renata Haddad
Implementing llms.txt for a Documentation Site
A guide to structuring documentation for AI agents using the minimal llms.txt specification.
Renata Haddad
Writing Technical Documentation for Dual Audiences
Structure, not tone, solves the dual-audience documentation problem.
Renata Haddad