What llms.txt does not do

The short answer: llms.txt is a file you publish, not a file anyone fetches. In a measurement from May 2026, 97% of published llms.txt files received zero requests. Google's John Mueller has been equally blunt: it is "purely speculative … none of the AI systems use it".

We still detect the file when we scan, and we report it as information. We award no points for it and deduct none when it is absent. A score that rewards a file existing rewards the cult around the file rather than the work.

This is not only about llms.txt

The whole genre looks the same once you measure it instead of reading about it:

• /.well-known/api-catalog (RFC 9727): of 74 domains probed, 68 answered with HTML and status 200 — a soft 404, not a catalogue.
• A2A Agent Cards: 65 of 22,341 hosts served one (0.29%), and only 10 of those were conformant.
• /.well-known/mcp.json is folklore. No specification defines that path.
• In our own scan of 135 Norwegian sites we found zero valid bodies for ai-plugin.json and mcp.json. Every 200 was an SPA shell.

Of all the agentic-commerce protocols that are detectable from outside at all, we found one valid manifest in the entire corpus: boozt.com, version 2026-04-08. That is not an argument for hurrying.

The part that can actually hurt you

Here is the trap. If your site answers 200 with HTML to any path — which every single-page application does by default — then you are claiming endpoints you do not have.

This is not theoretical. In an earlier run we reported 28 of 135 sites as advertising agentic-commerce endpoints. They were not. They just answered 200 to everything. Among them were an insurer, a clinic and Ticketmaster. We fixed it by probing a nonsense path first and only believing an endpoint if the body actually parses as the type the path promises.

But tools that do not do that still exist. And an agent that tries an endpoint you "have" and gets HTML back does not come again.

How to check

Ask for a path that certainly does not exist:

curl -s -o /dev/null -w "%{http_code}\n" https://yourshop.no/.well-known/does-not-exist-12345

The answer should be 404. If it is 200, your site is lying about everything under /.well-known/, and that is worth fixing regardless of what you think about agents.

Then look at what you actually publish:

for p in llms.txt robots.txt .well-known/security.txt; do
  printf "%s " "$p"
  curl -s -o /dev/null -w "%{http_code} %{content_type} %{size_download}\n" "https://yourshop.no/$p"
done

What to do instead

The order is not arbitrary. Everything here is fetched, unlike the files above.

1. Make sure robots.txt does not shut out the assistants answering questions right now. Blocking crawlers that collect training data is a separate and legitimate decision. 8 of the 135 sites in our corpus blocked search and live assistants, most likely without meaning to. Remember that robots.txt groups replace each other: a dedicated group for one bot name overrides the whole * group for that bot (RFC 9309 §2.2.1).
2. Keep sitemap.xml real and current. It does get fetched. In 46 of 135 scans we had to cut the sitemap walk short — either because it was enormous or because it was throttled.
3. Render listings, prices and availability on the server. That is where the other three articles here point, and that is where the return is.
4. Publish llms.txt anyway if you like. It costs nothing and harms nothing. Just do not believe you have done something.

One last warning about tooling: Cloudflare's own ai-rules check currently demands robots blocks for Claude-Web and anthropic-ai — two retired tokens — while omitting every current Anthropic token and PerplexityBot. So a site can pass that check and still be invisible to Claude. Worth remembering every time a checklist goes green.

Where these numbers come from

The 97% figure for llms.txt, the api-catalog and A2A Agent Card numbers, and the Mueller quote are not our measurements. They come from external standards research from May and August 2026, summarised in docs/roadmap.md. Everything else is ours: zero valid bodies for ai-plugin.json and mcp.json across 135 sites, one valid UCP manifest, 28 sites misreported by our own earlier run, 8 sites blocking search and live assistants, and 46 truncated sitemap walks.

We measure whether an assistant that reaches your site can use it. We do not measure whether assistants reach you at all — that is largely decided off the website, and a tool claiming otherwise is guessing.