Should You Add an llms.txt File? What It Does and When It Pays Off

You publish a 300-page documentation site, and when someone asks ChatGPT how your API handles rate limits, it either guesses or cites a competitor. The problem is not always your content. It is that an LLM crawling raw HTML has to wade through navigation, cookie banners, and boilerplate to find the two paragraphs that matter. llms.txt is a proposed convention that hands the model a clean, curated map of your most important pages instead of making it dig.

The idea comes from Jeremy Howard of Answer.AI, who published the specification at llmstxt.org in September 2024. It is worth being clear up front: this is a community proposal, not a ratified standard, and no major AI engine has publicly committed to reading it. Treat it as a low-cost, forward-looking bet, not a lever that moves your rankings this quarter.

What an llms.txt file actually contains

An llms.txt file lives at the root of your domain, at yoursite.com/llms.txt, and it is written in plain Markdown so a language model can parse it without stripping HTML. That location and format are the whole convention. There is no XML schema, no validation service, and no ping endpoint like the ones you use for XML sitemaps.

The proposed structure is short. You start with an H1 that names the project or company, add a blockquote with a one-sentence summary, then group links under H2 headings. Each link points to a page and carries a brief description of what the reader, human or model, will find there.

Imagine a Markdown file that opens with a single H1 reading "Acme Analytics," a one-line blockquote describing what the product does, then an H2 called "Docs" listing the quickstart, the authentication guide, and the API reference, each with a sentence of context. A second H2 called "Optional" might hold your changelog and blog. That is a complete, spec-compliant file. The point is curation: you decide which 15 pages represent your site, rather than letting a crawler infer importance from your internal link graph.

llms.txt versus robots.txt and sitemap.xml

People confuse these three files because they all sit at the domain root. They do different jobs, and llms.txt does not replace either of the others.

A sitemap wants completeness: every URL you will let a search engine index. An llms.txt file wants the opposite. It is an editorial shortlist with human-written context attached. Robots.txt controls access; llms.txt assumes access is already allowed and tries to improve comprehension.

Who honors it right now, and who does not

Be skeptical of anyone who tells you the big engines read this file today. As of October 2026, Google's own guidance is that Google Search ignores llms.txt: having one neither helps nor hurts your rankings, AI Overviews, or AI Mode. Google's Gary Illyes has said the company does not support the file, and John Mueller compared it to the long-ignored keywords meta tag. OpenAI and Perplexity have not committed to reading it on public websites either.

Where you do see traction is in developer-facing tools and documentation platforms. Anthropic, for instance, publishes an llms.txt file for its own docs, and several documentation frameworks now generate one automatically. So the convention has a real foothold in the technical-docs corner of the web, even while the general-purpose answer engines stay quiet.

The one notable shift in 2026 came from a different part of Google. Chrome's Lighthouse audit tool added an experimental Agentic Browsing category that checks for an llms.txt file and describes it as an emerging convention. That check is about AI agents browsing a site on someone's behalf, not about search rankings or AI citations, but it is the first time a major vendor has treated the file as something worth auditing.

The honest read: adding the file will not get you cited in an AI Overview tomorrow. It helps the AI agents and developer tools that already look for it, positions you for a future where answer engines might, and costs you very little to be early.

How to create one without overthinking it

You can hand-write a useful file in under an hour for most sites. The steps are mechanical.

1. List the 10 to 20 pages you would show someone who asked "what does this site do and how do I use it." For a SaaS product that is usually your docs home, quickstart, core feature pages, pricing, and API reference.

2. Write one sentence per page describing its content in specific terms. "Authentication guide: how to generate API keys and pass bearer tokens" beats "learn about auth."

3. Group the links under H2 headings that reflect intent, such as Docs, Guides, and Optional. The spec treats an "Optional" section as content a model can skip when it needs a shorter context.

4. Save it as Markdown, name it llms.txt, and serve it from your domain root with a text/plain or text/markdown content type.

Some teams also publish llms-full.txt, a companion file containing the actual expanded content of those pages rather than just links, so a model can ingest the substance in one fetch. That is optional and heavier to maintain, so start with the link-based version.

The mistakes that make it useless

The most common failure is generating the file once and forgetting it. Your product changes, pages move, and six months later your llms.txt points a model at three 404s and a renamed feature. A stale curated map is worse than none, because it actively misrepresents your site.

The second mistake is dumping every URL into it. If you paste your whole sitemap in Markdown, you have thrown away the only thing this file offers: editorial judgment about what matters. Keep it short and opinionated.

A third trap is expecting measurable traffic from it. There is no reporting, no "llms.txt impressions" dashboard, and no confirmed crawler that fetches it at scale. Do not promise your team a traffic lift you cannot instrument.

Fitting llms.txt into a wider AI-readiness plan

This one file is a small piece of being usable by answer engines. The signals that carry more weight today are the ones the engines already parse: valid structured data (Schema.org markup for articles, products, and organizations), clean semantic HTML, fast pages, and content that answers questions directly. llms.txt sits alongside those as insurance for what comes next, not as a substitute for them.

Because the file can silently drift or disappear during a redeploy, check it whenever you check site health. CrawlTide looks for an llms.txt file on every crawl and includes the result in your AI-readiness score, next to structured data and AI crawler access, so a file that has gone missing shows up the next time your site is crawled.

Add the file. Keep it to a tight, honest shortlist. Update it when your site changes, and watch it the way you watch your sitemap. You are spending an hour on a convention that might matter a great deal in a year, and very little if it does not. That is a reasonable trade, as long as nobody sells it to you as a guaranteed path into AI answers.