llms.txt, and whether it's worth publishing
What llms.txt is, what the evidence says about whether anything reads it, and when it's still worth having.
You have probably been told to add an llms.txt file to your site. The advice usually arrives with some urgency, as though AI assistants are queuing at your door and this file is the doorbell. It’s a five-minute job, so plenty of teams do it and move on.
Then nothing measurable happens. That’s not because you did it wrong. It’s because the evidence for llms.txt is much thinner than the advice suggests, and it’s worth knowing that before you build a content strategy on top of it.
What llms.txt actually is
llms.txt is a plain markdown file that sits at the root of your site, at /llms.txt. It lists your important pages with a line of description each, so a language model arriving at your domain can see the map instead of working it out from your navigation, cookie banner, and footer.
The idea is sound. Web pages are cluttered with markup that means nothing to a model trying to answer a question, and a curated index in clean markdown is genuinely easier to consume. The proposal lives at llmstxt.org, and it’s a community suggestion rather than a specification from any search engine or AI company.
That last point is the one that gets skipped. Nobody agreed to read this file. It was offered, not requested.
Info
A quick vocabulary check, because these get mixed up. robots.txt tells crawlers what they may fetch. A sitemap tells them what exists. llms.txt proposes what’s worth reading. Only the first two are honoured by anything at scale.
What the evidence says
Two findings from 2026 changed the picture, and neither is ambiguous. Neither means AI agents are ignoring your docs, though: they made 20.5% of knowledge base pageviews in our AI traffic statistics. They just don’t find your pages through llms.txt.
- Google ignores it. Google’s guidance on optimizing for generative AI features states plainly that you don’t need to create new files of any kind, and that Google Search ignores llms.txt. It won’t help your rankings and it won’t hurt them. Google’s framing is that it’s fine to keep one for other systems that use the file, which is a polite way of saying this is not for us.
- Almost nothing fetches it. Ahrefs checked all 137,210 domains in its Web Analytics that received traffic in May 2026, looked for an llms.txt on each, and counted the requests (Ahrefs).
The detail inside that study is the interesting part. Where files were fetched, GPTBot came first and Claude-Code second, ahead of every AI search product. Roughly 12% of all fetches came from the AEO industry checking on itself: llms.txt validators, GEO tools, and researchers.
| The claim | What the evidence shows |
|---|---|
| “AI needs llms.txt to find your content” | AI crawlers never even check for one on sites that don’t have it |
| “It improves your AI visibility” | Google Search ignores it; no major AI search product reports using it |
| “Everyone is doing it” | About 28% of domains have one, and 97% of those see no requests |
| “It’s read by the assistants my customers use” | The top two fetchers are a training crawler and a coding agent |
Note
None of this means llms.txt is a scam. It means it’s an unratified proposal with low adoption on the consuming side, which is a very different thing from a ranking factor.
Why it hasn’t worked the way people hoped
Three reasons, and they compound.
- Nothing goes looking for it. A model answering a question doesn’t wander onto your domain hunting for a map. It’s handed pages by a search index, a retrieval system, or a direct integration, and in all three cases the content is what gets read, not your index of it.
- It’s self-declared. An llms.txt is your own summary of your own site, so anything reading it would still have to check the pages to confirm the claims. Once you’re reading the pages, the index has saved you very little. Search engines learned this with the keywords meta tag about twenty years ago.
- It drifts. A hand-written index is accurate the day you write it. Six months later it lists three articles you merged and misses the twelve you added.
Warning
A stale llms.txt is worse than no llms.txt. The few crawlers that do fetch it will follow it to pages that have moved, which is a worse outcome than letting them read your sitemap.
When it’s still worth publishing
It comes down to who maintains the file:
- Publish one if it’s automatic. If your platform generates and updates the file as your content changes, the cost is genuinely zero and a couple of real crawlers do fetch it. GPTBot topping that list matters if you care about being in training data at all.
- Skip it, or deprioritise it, if it’s manual. A file you maintain by hand will drift, and the failure mode above is real. That time buys more elsewhere.
Every HelpDocs plan ships an llms.txt index and clean markdown for your knowledge base, at help.yoursite.com/llms.txt. We include it because it costs nothing to generate and occasionally helps. On its own, it won’t get you cited. Content that answers questions clearly does that.
Tip
If you want a five-minute job that reliably helps, make sure your articles answer their own titles in the first paragraph. Retrieval pulls passages, and the opening passage is the one most likely to get pulled.
What actually moves the needle
The unglamorous answer is the content. A retrieval system’s ceiling is set by three things, none of which live at your domain root:
- Whether the answer exists in your documentation at all.
- Whether it’s stated once, rather than three times with contradictions.
- Whether a machine can locate the right passage inside the right article.
That’s the work, and unlike llms.txt it’s testable. Our guide to building an AI knowledge base has nine specific checks with a way to test each, and how AI uses your knowledge base explains the retrieval mechanics underneath them. On the format question, clean semantic HTML and a real sitemap are still doing more for you than any new file you can add today.
Important
Be sceptical of AI-visibility tactics you can’t measure. If a change can’t be checked in your server logs or your analytics, treat the confidence around it as marketing rather than evidence.
Templates for this topic
