Copy for Originals · Original research
Copy for Originals checked the websites of 100 Chicago personal injury law firms on October 13 and 14, 2026. Only 3 disallow an AI crawler in robots.txt. But 13 of the 78 sites that could be tested refused a request carrying the name GPTBot, ClaudeBot or PerplexityBot every time, and 11 of those 13 served the same request under Googlebot’s and Bingbot’s names.
Educational only, not legal advice. This study describes public files and server responses. Whether a firm should let AI crawlers read its site is the firm’s own decision.
3 of 100. Each of the 100 sites has a robots.txt file, and 89 of them don’t mention an AI crawler. Of the 11 that do, 8 name crawlers without disallowing any of them.
| Crawler name | Company and stated use | Sites naming it | Sites disallowing it |
|---|---|---|---|
| GPTBot | OpenAI, training | 9 | 2 |
| OAI-SearchBot | OpenAI, ChatGPT search | 8 | 1 |
| ChatGPT-User | OpenAI, fetches for a user’s question | 7 | 1 |
| ClaudeBot | Anthropic, training | 10 | 2 |
| Claude-SearchBot | Anthropic, search | 3 | 0 |
| Claude-User | Anthropic, fetches for a user’s question | 3 | 0 |
| PerplexityBot | Perplexity, search | 7 | 1 |
| Google-Extended | Google, a robots.txt token for Gemini training and grounding | 8 | 2 |
| CCBot | Common Crawl, an open web archive | 5 | 2 |
“Disallowing” means the group of rules that names the crawler contains Disallow: / with no Allow: /. No site disallowed every crawler with a wildcard rule.
The three firms that disallow an AI crawler did it three different ways:
5 of the 93 files read on October 13, 2026 carry a Content-Signal line, the format Cloudflare published on September 24, 2025. 4 say search=yes, ai-input=yes, ai-train=yes, and 1 says search=yes, ai-input=yes, ai-train=no. Three of the files that say yes to training are identical apart from the site’s address, which points to one website platform writing the line for its law firm customers.
Yes, more often than in robots.txt. 13 of 78 sites (17%) served the homepage to a browser-style request but refused the same request when it carried an AI crawler’s name. Each site was asked five times per name, at least three minutes apart and in a different order each time, and “refused” here means refused all five times.
| Name in the request | Kind | Served all 5 times | Refused all 5 times | Mixed |
|---|---|---|---|---|
| GPTBot | AI crawler | 61 | 8 (10%) | 9 |
| ClaudeBot | AI crawler | 58 | 10 (13%) | 10 |
| PerplexityBot | AI crawler | 74 | 2 (3%) | 2 |
| Googlebot | Search crawler (control) | 76 | 2 (3%) | 0 |
| Bingbot | Search crawler (control) | 74 | 4 (5%) | 0 |
51 of the 78 sites served all five names every time. 13 refused at least one AI crawler’s name every time. The refusals were 403 “Forbidden” responses at 11 sites and 429 “Too Many Requests” at 2. 6 of the 13 sites sit behind Cloudflare.
Mostly not. 11 of the 13 sites that refused an AI crawler’s name served the same request every time when it carried Googlebot’s or Bingbot’s name. 2 refused a search crawler’s name as well.
This matters for reading the result. A site that checks whether a visitor really is the crawler it claims to be would turn away a request falsely carrying Googlebot’s name. 11 sites did not, so an identity check is an unlikely explanation there. Those sites treat an AI crawler’s name as a reason to refuse.
Across all 78 sites, 2 refused Googlebot’s name every time and 4 refused Bingbot’s. 3 sites refused Bingbot’s name while serving the three AI names and Googlebot’s.
No. Only 1 of the 13 sites has a robots.txt rule against a crawler it refused. 10 have no AI crawler rule at all. 2 name the refused crawler in robots.txt without disallowing it, so the file reads as an invitation and the server turns the visitor away.
It also runs the other way. 2 sites disallow ClaudeBot in robots.txt and served the ClaudeBot-named request every time. RFC 9309, the robots.txt standard, says its rules “are not a form of access authorization.” The file states what the site asks of a crawler, and the crawler decides whether to comply.
Largely, yes. Three more runs on October 13, 2026 went out through a VPN exit in Chicago with the same six requests. 74 sites could be tested from there and 71 from both places. 68 of those 71 landed in the same place from both: refusing the same AI crawler names, or none. 12 refused at least one AI crawler’s name from both places. From Chicago, 13 of the 16 sites that refused an AI crawler’s name served Googlebot’s and Bingbot’s names every time.
It doesn’t prove the real crawler is refused. The study’s requests came from ordinary internet connections, not from OpenAI’s, Anthropic’s or Perplexity’s own addresses. A refusal in this table means the site treats the crawler’s name differently from a browser, and the Googlebot comparison shows that at most of these sites the name is what triggers it. Only the site’s own server logs show what the real crawler receives.
21 sites refused the browser-style request in every run, and 1 couldn’t be reached, so there was nothing to compare against. Most of the 21 run bot protection that turned away a script calling itself a browser: a Cloudflare challenge page at some, a plain 403 at others.
16 of those 21 served the page when the same request carried the name GPTBot. 11 served all three AI names. 5 served GPTBot’s name and refused ClaudeBot’s and PerplexityBot’s. On these sites a crawler’s name alone was enough to get past a block that stopped an ordinary request.
42 of 92, or 46%. An llms.txt file is a markdown file at the root of a site, proposed on September 3, 2024 to help AI tools use the site.
| Result | Sites |
|---|---|
| A markdown llms.txt file | 42 (46%) |
| A file written in robots.txt syntax | 8 (9%) |
| No file | 42 (46%) |
8 of the 100 sites didn’t give a readable answer for this address and are left out.
Who wrote the files. 19 of the 42 files name the software that generated them: Yoast SEO in 12, Rank Math in 4, All in One SEO in 1, Hostinger Tools in 1 and MileMark Schema Pro in 1. Yoast says its file is “generated and refreshed automatically” once the feature is switched on. The other 23 files don’t say how they were made.
How big they are. Among the 35 files read directly, the median is about 9.1 KB and the smallest is 1.3 KB. 11 files are over 50 KB, and the largest is about 2 MB.
The robots.txt-style files. 8 sites answer the llms.txt address with lines such as User-agent: * and Allow: /. Six of them serve the identical 65-byte file, which includes the line Disallow-Training: /. The llms.txt proposal describes a markdown file with a title and links. It doesn’t define access rules, so nothing in the proposal says a crawler should act on these lines.
yourfirm.com/robots.txt and search for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot. A User-agent line naming one of them, followed by Disallow: /, asks that crawler to stay out.yourfirm.com/llms.txt exists, it should be markdown that starts with a # title and lists your pages. If it starts with User-agent, it isn’t in the format the proposal describes.For what answer engines do with a page once they can read it, see how to write content that LLMs retrieve and cite. The firms in this study come from the Chicago personal injury lead generation study.
User-agent line as RFC 9309 describes. A crawler counts as disallowed when a group naming it contains Disallow: / or Disallow: /* and no Allow: /. Fifteen AI crawler names were checked; the table shows nine.GPTBot/1.4, ClaudeBot/1.0, PerplexityBot/1.0, Googlebot/2.1 and bingbot/2.0. There were five runs on October 14, 2026 from one home internet connection in Sweden, at least three minutes apart, with the six requests in a different order each run. The exit address was checked before and after every run. A site counts as testable when the browser-style request returned 200 in all five runs (78 sites). A name counts as refused when it got a 403 or 429 in all five runs, and as mixed when the runs disagreed.User-agent were counted separately.Every site and every response code is in the data file, and the collection scripts are in the method folder on GitHub.
Few do it in robots.txt, and more do it at the server. In a Copy for Originals study of 100 Chicago personal injury law firm websites on October 13 and 14, 2026, 3 disallowed at least one AI crawler in robots.txt. Of the 78 sites that could be tested at the server, 13 refused a request carrying the name GPTBot, ClaudeBot or PerplexityBot in all five runs: 10 refused ClaudeBot, 8 refused GPTBot and 2 refused PerplexityBot.
Mostly not. In the Copy for Originals study, 11 of the 13 law firm websites that refused a request carrying an AI crawler’s name served the same request every time when it carried Googlebot’s or Bingbot’s name. Across all 78 sites tested, 2 refused Googlebot’s name and 4 refused Bingbot’s name.
Yes. A firewall, a hosting company or a security plugin can refuse a crawler before the website itself answers, and robots.txt does not change. In the Copy for Originals study, 12 of the 13 law firm websites that refused a request carrying an AI crawler’s name had no robots.txt rule against that crawler, and 2 of them named the refused crawler in robots.txt without disallowing it.
42 of 92, or 46%, in the Copy for Originals study of Chicago personal injury law firm websites on October 13 and 14, 2026. Another 8 of the 92 answered the llms.txt address with a file written in robots.txt syntax, which the llms.txt proposal does not define, and 42 had no file. Of the 42 files, 12 said they were generated by Yoast SEO.
All read on October 13, 2026.
How to cite: Copy for Originals, “Do law firm websites block AI crawlers?,” version 2, October 14, 2026, https://tools.copyfororiginals.com/law-firm-websites-ai-crawlers-study/. Data licensed CC BY 4.0.
Change log: version 1, October 13, 2026, used three runs and no control names and reported 15 of 78 sites refusing an AI crawler’s name. Version 2, October 14, 2026, reran the server test with five spaced runs from one connection, added Googlebot and Bingbot as control names, added three runs from a Chicago address as a second location, and read 13 robots.txt files by a second route in place of older archive copies. Version 1’s data is kept as pilot-three-runs.csv.