Which AI crawlers actually visit a website? We built an instrument to find out.
Peter Benes ·
Should a business block the AI bots? Most answers to that question are guesses. So we built a site that measures.
Rattlesnakes By Mail is a field guide to the crawlers of the web. It covers 15 crawlers and related bots from six companies: Anthropic, OpenAI, Google, Microsoft, Apple and Perplexity. Every fact on it is a dated claim with a short quote from the vendor’s own documentation.
It also counts which of those crawlers visit its own pages, how often, and whether each one is who it says it is.
The challenge
A crawler announces itself with a name in its request, called a user agent. “GPTBot”. “ClaudeBot”. The trouble is that anyone can type any name. A scraper can call itself Googlebot in one line of code.
So counting names tells you very little. To know who is actually reading a site, you have to check each visit against something the visitor can’t fake.
The choice
Check identity in three steps. First, match the name. Then check the visit’s address against the IP ranges the vendor publishes, or against the vendor’s documented reverse-DNS check. If neither exists for that crawler, fall back to Cloudflare’s verified-bot category. The site publishes the verified share beside every count.
Don’t over-read a failed check. The site says it outright: an unverified request is not evidence that the user agent was spoofed. It only means we couldn’t confirm it.
Block nothing. Nothing on the site blocks, challenges or slows down any crawler. If you block, you stop measuring.
Make every page readable three ways. Each page exists as a normal web page, as plain-text Markdown, and as structured data. That lets us see which format each crawler prefers.
Protect visitors. Visitor addresses are scrambled with a one-way hash whose key changes daily, and raw addresses are never stored. Human visitors are counted once a day as a total, never per request.
Stay in scope. Every observation describes requests to this one site only. It is not a statement about how a crawler behaves anywhere else.
What we found
Between September 13 and October 2, 2026, the site logged 7,996 requests from AI crawlers. A typical day sees about 121.
The busiest day was September 24: 2,684 requests, about 22 times a normal day. ClaudeBot made 1,141 of them and GPTBot 1,101. We don’t know what set that day off.
Over the whole period:
| Crawler | Requests | Verified |
|---|---|---|
| ClaudeBot | 2,690 | 100% |
| GPTBot | 2,294 | 100% |
| Googlebot | 1,239 | 44% |
| Applebot | 1,078 | 100% |
| GoogleOther | 555 | 100% |
| OAI-SearchBot | 56 | 100% |
| Bingbot | 53 | 100% |
| Claude-User | 19 | 89% |
| ChatGPT-User | 10 | 100% |
| PerplexityBot | 2 | 100% |
Three things stand out.
Two crawlers do most of the reading. ClaudeBot and GPTBot together made 4,984 of the 7,996 requests, about 62%.
Nearly everything checked out, with one exception. Almost every crawler verified at or near 100%. Googlebot did not: 44% over the period, against 99.1% in the site’s first count. We don’t know why yet. Per the rule above, that’s not proof of impostors. It’s an open question.
Crawlers read differently. ClaudeBot fetched the Markdown and data versions as well as the web pages, starting from the sitemap. GoogleOther skipped robots.txt and requested the site’s MCP endpoint, the connection point built for AI tools. And in our observation, Googlebot does not read llms.txt, the summary file written for AI tools.
What the vendors say about blocking
The site also collects what each vendor documents about its own bots. Three points matter to anyone thinking about blocking:
- Blocking Google-Extended does not remove you from Google Search.
- Blocking GPTBot while allowing OAI-SearchBot keeps you eligible for ChatGPT search.
- Blocking GPTBot does not govern ChatGPT-User, the agent that fetches a page when a person asks ChatGPT about it.
In other words, “block the AI bots” isn’t one switch. It’s a set of separate decisions with different effects.
The most-asked question on the site, so far unresolved, is whether robots.txt can cleanly separate “allow for search and citation” from “block for training”. It’s been asked five times. We don’t have a clean answer yet.
What we don’t know yet
This is one small site over 20 days. It says nothing about what crawlers do on a large site, a shop, or yours. We don’t know what caused the September 24 spike, why Googlebot’s verified share dropped, or whether any of this reading turns into AI tools citing the site.
What it means for your site
If you’re thinking of blocking AI crawlers, decide bot by bot, not all at once. Blocking a training crawler and blocking the one that fetches your page when a customer asks an assistant about you are different choices. Blocking the second may mean an assistant can’t read your page when a customer asks about you.
And remember that robots.txt is a request, not a lock. It works for crawlers that choose to follow it. A scraper using a fake name doesn’t have to.
For our part, we block nothing and measure everything. That approach runs through all our builds; more on how it works.