Generate llms.txt files for AI agents and LLMs to index your site.
Just like robots.txt tells legacy web crawlers (like Googlebot) what pages they are allowed to index, llms.txt tells modern AI Agents and Large Language Models how to efficiently read and summarize your website.
When an AI coding agent or chatbot visits your documentation, it doesn't need your CSS, React navigation bars, or promotional banners. Processing that HTML bloat wastes tokens and context window space. The llms.txt file acts as a manifest, pointing the agent directly to simplified, Markdown-formatted versions of your content.
The llms.txt file is strictly written in Markdown and is conventionally placed at the root of your domain (e.g., https://example.com/llms.txt).
It typically includes:
llms-full.txt).A standard web page might contain 500 words of actual text, but the underlying HTML could be 20,000 tokens of DOM nodes and inline styles. If an AI agent fetches the HTML, it burns through its context window limit extremely quickly.
By providing markdown files via llms.txt, you guarantee a near 1:1 ratio of tokens to actual semantic information, drastically improving the intelligence and accuracy of the AI's responses about your product.
What is the primary purpose of an llms.txt file?