Your site answers differently depending on who asks

Most sites answer the same regardless of who asks. Some do not, and then the name in the request — the User-Agent — decides what comes back. Almost always it is a firewall, a CDN rule or a bot service making that decision rather than anyone at the company. That does not make it less real.

Two real cases, measured

dnb.no. A request claiming to be ChatGPT-User receives 60 bytes — a stub of a security.txt — with status 200. The same holds for ClaudeBot, GPTBot, PerplexityBot and Googlebot. It holds on every path we tried, including /robots.txt. Our own honest identity received the whole site: 1,171,241 bytes. We reproduced it 23 times out of 23, on two different IP stacks, hours apart. It is not noise.

Note the status code. It is not a 403. It is a 200 with a 60-byte body. Anything counting fetches and error rates on that site sees a healthy, working website.

platekompaniet.no. The opposite. Our own identity gets a 62,755-byte shell with no JSON-LD and no prices. A request claiming to be ChatGPT-User gets 589,229 bytes, with JSON-LD and 22 prices in it. ClaudeBot gets a 403.

Both sites have made a decision. In the first case it looks like a bot block that caught more than it meant to. In the second it looks like deliberate prerendering for one named fetcher. Neither is visible in ordinary analytics, and neither shows up in a browser.

What we can say, and what we cannot

We send the name ChatGPT-User from our own address, without the vendor's signature and not from the vendor's IP range. So the only sentence we can support is:

This is what your site answers a request claiming to be ChatGPT-User.

That is not a measurement of what the vendor's own fetcher actually received. A site that recognises the vendor by IP range or by signed requests may answer the real fetcher something quite different from what it answers us. In our reports that wording travels with the observation itself and cannot be shortened along the way — precisely because the short version would be a selling point we cannot back.

The same rule runs the other way. If our scanner is refused, we never state that as proof that a named assistant is refused.

How to check your own site

It is your site, so you can ask it anything you like. Compare the size of the answer:

curl -sL -o /dev/null -w "%{http_code} %{size_download}\n" https://yourshop.no/
curl -sL -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)" \
  https://yourshop.no/
curl -sL -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \
  https://yourshop.no/

Three lines with roughly the same status and byte count means you treat everyone alike. That is usually the answer you want.

The gaps that matter:

• A much smaller answer to one of the names. You are serving a stub. Check whether it happens on /robots.txt too — that is a good sign it is infrastructure rather than your application.
• A much larger answer to one of the names. You are prerendering for one fetcher and not the others. Everything you built for ChatGPT, you have not built for Claude, Perplexity or whatever comes next.
• 403 or 429 to any of them. You have a bot rule. It may be entirely correct — but it should be a decision somebody took, not a default nobody has looked at.

Remember that this only measures the name in the request. If you want to know what actually happens, your server logs are a better place to look: search for the same tokens and see what you answered them over the past day.

What to change

1. Find out who is deciding. Nine times out of ten it is the CDN or the WAF, not the application. Look for a bot rule that is on because it was on by default.
2. Separate training from lookup. Blocking a crawler that collects training data is a legitimate decision, and we never report it as a defect. Blocking the tokens used when somebody asks a question right now takes you out of the answer the customer gets. Those are two very different choices, and they are often made as one.
3. Do not build for one name. If you prerender only for ChatGPT-User, you have built something that breaks the moment the list of names changes — and it changes. Render on the server for everyone instead.
4. Make sure robots.txt answers normally. A stub at robots.txt is worse than a block, because it looks like an answer.

One more thing: if you do block by name, remember that robots.txt groups replace each other. A dedicated OAI-SearchBot group fully overrides the * group for that bot (RFC 9309 §2.2.1). It is a common and expensive misunderstanding.

How to verify the fix

Run the three curl lines again. The target is three answers with the same status and byte counts within a few percent of each other — or, if you have deliberately chosen to shut someone out, that it happens with an honest 403 rather than a 200 carrying 60 bytes.

Where these numbers come from

The figures for dnb.no and platekompaniet.no are ours, measured on 8 August 2026 with ordinary GET requests, reproduced twice hours apart and on two IP stacks. Both are pinned as ground truth in the scanner's validity harness. The scanner also sends its own Signature-Agent token on measurements like these so a site owner can find us in their log; we measured that it changed the answer on neither site.

We measure what your site answers. We do not measure what a vendor's own fetcher actually received.