Does your site need an llms.txt? An honest read on what it does in 2026

A growing share of buying and research questions now start in an AI assistant rather than a search box, and that has produced a new piece of advice you will see everywhere: add an llms.txt file. The pitch is that it makes your site legible to large language models. Some of that is true, and some of it is hope dressed up as fact. This is what the file actually is, what it does in 2026, and whether it is worth your afternoon.

What llms.txt actually is

llms.txt is a plain Markdown file you serve at the root of your domain, at /llms.txt. It is a curated map of your site for a model with a limited context window: a title, a one-line description, and a short, labelled list of your most important pages, so an assistant does not have to guess which of your URLs matter. The idea was proposed in September 2024 by Jeremy Howard of Answer.AI and is documented at llmstxt.org. A companion file, llms-full.txt, goes further and inlines the actual content of those pages so an assistant can read your material without crawling and rendering each one.

So far so reasonable. The file is small, human-readable, and easy to keep current. The question is not whether it is tidy. It is whether anything reads it.

The honest state of adoption in 2026

Here the marketing and the evidence part ways. The big answer engines have not committed to the convention:

• Google has said its Search systems do not use llms.txt, so it is not a ranking signal and will not change your position in classic results.
• As of 2026, no major AI search provider (Google, OpenAI, Anthropic, or Perplexity) has documented that its crawlers consume llms.txt from external sites.
• Industry monitoring of AI bot traffic has found direct requests to llms.txt files to be a negligible share of all AI crawling, which is the blunt version of the point above.
• Adoption by sites is still early too. One large 2026 crawl of 300,000 domains found only around one in ten had an llms.txt file, after well over a year of industry conversation about it.

None of that makes the file harmful. It makes it a forward-looking bet rather than a present-day ranking lever, and the honest framing is to treat it that way.

The 2026 numbers: adoption up 8.8x, requests near zero

The clearest read yet arrived in mid-2026, and it sharpens the point rather than softening it. An Originality.ai tracking update published on 2 July 2026, drawn from more than three million sites, found the number of llms.txt files had grown about 8.8 times in a year, from roughly 4,000 in June 2025 to about 36,000 by May 2026. Across the broad web that is still only a sliver of all sites, which is why a curated crawl of large domains reports a higher share than a three-million-site sweep does, but the direction is real and accelerating: more sites are publishing the file every month.

The reading, though, has not followed. A parallel server-log study by Ahrefs across 137,000 domains found that in May 2026, 97 percent of llms.txt files were fetched zero times. Among the requests that did land, actual AI retrieval bots were only about 1 percent; SEO-audit tools, unidentified bots and general web crawlers made up most of the rest, with named AI crawlers like GPTBot and ClaudeBot each a low single-digit share or less. So the file is spreading quickly and being consumed almost not at all. That gap between publishing and reading is the whole story, and it is the honest reason to treat llms.txt as a cheap forward bet rather than a lever that changes how you are answered today.

Where it genuinely earns its place today

The real traction is not in web search at all. It is in developer tooling, where an assistant is pointed at a documentation site and benefits from a clean index of it:

• Coding agents and IDE assistants increasingly fetch a project or product documentation site on demand, and a tidy llms.txt or llms-full.txt is exactly the curated entry point they want.
• Several developer-facing companies now publish one for their docs for this reason, which is the clearest signal of where the file is actually useful.
• It doubles as a human-readable index of your best pages, which has modest value regardless of whether any model reads it.

So the pattern is clear: the heavier your documentation and the more developer-shaped your audience, the more an llms.txt earns its keep. For a small marketing site the upside is thinner, though the cost is close to zero.

The half that matters right now: can the crawlers even reach you

Before llms.txt can help anyone, an AI crawler has to be allowed to fetch your pages at all, and that is decided in robots.txt, not in llms.txt. This is the part with real consequences today, and it is where sites most often lock themselves out by accident. One distinction changes what you should do:

• Training crawlers such as GPTBot, Google-Extended, and CCBot gather content to train future models. Blocking them keeps your text out of training runs but does not, on its own, remove you from live answers.
• Answer and search crawlers such as OAI-SearchBot, PerplexityBot, and ClaudeBot fetch pages to ground real-time answers and cite sources. If you want to be quoted in AI answers, these are the ones to let in.

The right setting is a deliberate choice, not a default. Plenty of publishers block training while allowing the answer engines. Getting that right is worth more this year than any llms.txt file, because it decides whether you can be cited at all.

So should you add one?

Add an llms.txt if you have substantial documentation or a developer audience, where coding and doc assistants already read these files. For a small marketing site, add one only because it is cheap and tidy, not because you expect it to move rankings, because it will not. In every case, do the higher-value work first: confirm your robots.txt lets the answer engines in, then make your pages easy to parse and quote, which is what genuinely decides whether an assistant cites you. The llms.txt is a low-effort bet placed on top of those fundamentals, not a substitute for them.

Measure whether it worked

All of this stays guesswork until you check the outcome it is aimed at: whether assistants actually cite you. Our free AI search visibility auditor builds a prompt grid you can run by hand, the guide on how to get cited by ChatGPT and Perplexity covers what to fix when you are not showing up, and if AI visibility becomes a channel you report on, the comparison of Profound, Otterly and Peec weighs the paid trackers that run that grid daily. Publish the llms.txt if it is cheap to; spend the real effort on being reachable, quotable, and measured.