353 government sites block ChatGPT search but allow Google, study finds

Government websites can stay open to Google while shutting out the crawler behind ChatGPT search. Josh Blackburn SEO’s AI Crawler Access Study, published October 29, identified 353 such sites. Its findings point largely to catch-all website rules and shared templates, rather than rules that explicitly single out OpenAI’s search crawler.

The consequence concerns which sources can appear in an answer. The study’s press release, citing OpenAI documentation, says sites opting out of OAI-SearchBot do not appear in ChatGPT search answers, although they can still appear as navigational links. That distinction matters for questions about trash pickup, permits or council meetings: a link to a city website is not the same as using it as a cited source.

Old approved-crawler lists leave newer bots outside

The study examined robots.txt, the file containing a website’s crawler rules. Across all 423 government sites it classified as blocking OAI-SearchBot—not just those allowing Google—only 11 named the OpenAI crawler. The other 412 used a catch-all restriction beneath a list of approved crawlers. A crawler missing from that list falls under the general restriction.

Two shared files accounted for 242 of the 353 sites that blocked OpenAI search while allowing Google. The larger file appeared on 183 sites across 27 states and seemed to be a platform default for small local governments. Another 59 sites across 26 states used an older template written for search engines of the 2000s.

Blackburn attributes the pattern to rules inherited from an earlier search era. His explanation is that these files exclude crawlers that did not exist when the approved lists were written. The distinction is important: naming an AI crawler in a restriction and leaving it off an old approved list produce similar access results, but do not demonstrate the same intent.

2.9%All responding government sites

The 353 sites represented 2.9% of the 12,298 government sites that responded.

5.2%Sites with valid robots.txt files

Among the 6,765 sites publishing a valid robots.txt file, the share was 5.2%.

The gap extends beyond OpenAI

For 283 of the 353 sites, the split covered several search services: Googlebot and Bingbot were allowed, while OAI-SearchBot, Claude-SearchBot and PerplexityBot were blocked. Minnesota had the highest reported state rate, at 72 of 329 sites with a robots.txt file, or 21.9%. Of those 72 cases, 63 used the platform template.

Search access and training access also diverged. The platform template allowed CCBot, the Common Crawl crawler used to collect AI training data, while excluding the AI search crawlers. The release recommends treating those permissions separately rather than assuming that opening a site to AI search requires opening it to training.

The study’s proposed changes

• Add explicit permissions for OAI-SearchBot, Claude-SearchBot and PerplexityBot if AI search access is wanted.
• Keep separate restrictions on training crawlers if desired; platform-hosted sites may need their vendor to change the file.

A snapshot of rules, not every government page

The October 24 crawl covered 15,331 state, county, city, special-district, school-district and interstate domains from the federal government’s official domain list. It checked only homepages and interpreted files under RFC 9309 using the same group-matching rules as Google’s parser. The results describe a single day, not a continuing audit.

Individual sites were not named, and state figures were suppressed where fewer than 10 sites published a robots.txt file. The findings were issued through a GlobeNewswire press release by Josh Blackburn SEO; Markets Insider and Business Insider’s editorial teams were not involved in producing it.