llms.txt, and whether it's worth publishing
What llms.txt is, what the evidence says about whether anything reads it, and when it's still worth having.
You have probably been told to add an llms.txt file to your site. The advice usually arrives with some urgency, as though AI assistants are queuing at your door and this file is the doorbell. It’s a five-minute job, so plenty of teams do it and move on.
Then nothing measurable happens. That’s not because you did it wrong. It’s because the evidence for llms.txt is much thinner than the advice suggests, and it’s worth knowing that before you build a content strategy on top of it.
What llms.txt actually is
llms.txt is a plain markdown file that sits at the root of your site, at /llms.txt. It lists your important pages with a line of description each, so a language model arriving at your domain can see the map instead of working it out from your navigation, cookie banner, and footer.
The idea is sound. Web pages are cluttered with markup that means nothing to a model trying to answer a question, and a curated index in clean markdown is genuinely easier to consume. The proposal lives at llmstxt.org, and it’s a community suggestion rather than a specification from any search engine or AI company.
That last point is the one that gets skipped. Nobody agreed to read this file. It was offered, not requested.
What the evidence says
Two findings from 2026 changed the picture, and neither is ambiguous.
- Google ignores it. Google’s guidance on optimizing for generative AI features states plainly that you don’t need to create new files of any kind, and that Google Search ignores llms.txt. It won’t help your rankings and it won’t hurt them. Google’s framing is that it’s fine to keep one for other systems that use the file, which is a polite way of saying this is not for us.
- Almost nothing fetches it. Ahrefs checked all 137,210 domains in its Web Analytics that received traffic in May 2026, looked for an llms.txt on each, and counted the requests (Ahrefs).
The detail inside that study is the interesting part. Where files were fetched, GPTBot came first and Claude-Code second, ahead of every AI search product. Roughly 12% of all fetches came from the AEO industry checking on itself: llms.txt validators, GEO tools, and researchers.
| The claim | What the evidence shows |
|---|---|
| “AI needs llms.txt to find your content” | AI crawlers never even check for one on sites that don’t have it |
| “It improves your AI visibility” | Google Search ignores it; no major AI search product reports using it |
| “Everyone is doing it” | About 28% of domains have one, and 97% of those see no requests |
| “It’s read by the assistants my customers use” | The top two fetchers are a training crawler and a coding agent |
Why it hasn’t worked the way people hoped
Three reasons, and they compound.
- Nothing goes looking for it. A model answering a question doesn’t wander onto your domain hunting for a map. It’s handed pages by a search index, a retrieval system, or a direct integration, and in all three cases the content is what gets read, not your index of it.
- It’s self-declared. An llms.txt is your own summary of your own site, so anything reading it would still have to check the pages to confirm the claims. Once you’re reading the pages, the index has saved you very little. Search engines learned this with the keywords meta tag about twenty years ago.
- It drifts. A hand-written index is accurate the day you write it. Six months later it lists three articles you merged and misses the twelve you added.
When it’s still worth publishing
It comes down to who maintains the file:
- Publish one if it’s automatic. If your platform generates and updates the file as your content changes, the cost is genuinely zero and a couple of real crawlers do fetch it. GPTBot topping that list matters if you care about being in training data at all.
- Skip it, or deprioritise it, if it’s manual. A file you maintain by hand will drift, and the failure mode above is real. That time buys more elsewhere.
On eligible HelpDocs plans your knowledge base publishes an llms.txt index and clean markdown for you, at help.yoursite.com/llms.txt. We do it because it’s free to do and occasionally useful, not because we think it’s the thing standing between you and AI visibility. It isn’t.
What actually moves the needle
The unglamorous answer is the content. A retrieval system’s ceiling is set by three things, none of which live at your domain root:
- Whether the answer exists in your documentation at all.
- Whether it’s stated once, rather than three times with contradictions.
- Whether a machine can locate the right passage inside the right article.
That’s the work, and unlike llms.txt it’s testable. Our guide to building an AI knowledge base has nine specific checks with a way to test each, and how AI uses your knowledge base explains the retrieval mechanics underneath them. On the format question, clean semantic HTML and a real sitemap are still doing more for you than any new file you can add today.
