Full disclosure: This post contains affiliate links. If you make a purchase through these links, 99signals may earn a commission at no additional cost to you.
On August 15, 2026, the owner of zoundroom.com changed a single setting through Cloudflare’s Bot Management API. They set AI training to block, left search and agent crawling untouched, and moved on.
Over the following week, Google Search Console flagged 403 errors on 68 URLs, and 47 of them were canonical pages listed in the sitemap. Indexed pages fell from 114 to 72.
The intent was keeping content out of AI model training. The outcome was a site partially removed from Google.
Cases like this are why crawler access belongs at the very start of any AI visibility audit. Content strategy, entity work, and citation tracking all assume a crawler can read the page, and a growing number of sites have security layers making that call without telling anyone.
How Googlebot Ended Up Classified as an AI Training Bot
On July 1, 2026, Cloudflare replaced its single “Block AI bots” toggle with three categories based on what a crawler does: Search, Agent, and Training. Cloudflare sits in front of roughly a fifth of the web, so the change reached a large share of sites at once.
The complication was crawlers that do more than one job. Cloudflare classifies Googlebot, Bingbot, and Applebot as mixed-use, since one crawler feeds both a search index and AI products, and it judges mixed-use crawlers under whichever rule is strictest.
That category isn’t a niche. Mixed-use crawlers account for 36.6% of verified crawler traffic on Cloudflare’s network, the largest single category it tracks.
The practical effect was that blocking training could block Googlebot. On August 4, Search Engine Journal reported a Reddit thread where a site owner set AI Training to Block and watched Googlebot and Bingbot receive 403 errors on the sitemap. The errors stopped the moment the block came off.
That report landed six weeks before Cloudflare’s own September 15 enforcement date. The behavior clearly didn’t wait for the announced schedule.
What Cloudflare Changed on September 15
The deadline didn’t play out the way most summer coverage predicted. Cloudflare introduced an Accountable designation for crawler operators, and Apple, Google, and Microsoft were the first mixed-use operators to receive it.
To qualify, an operator must offer a training opt-out, an opt-out from AI summaries, URL-level visibility into how content is used, and a public assurance that declining training won’t affect search rankings. Operators can meet these now or commit to a timeline.
Alongside the label, Cloudflare launched a setting called Disallow AI Training. It publishes a no-training preference, keeps Accountable mixed-use crawlers crawling for search, and blocks training-only crawlers such as GPTBot and ClaudeBot.
That resolved the worst version of the problem, but only for sites that choose the right option. Selecting Block on the Training control still stops Googlebot, Bingbot, and Applebot entirely, and Block on pages with ads now applies to mixed-use crawlers too.
Defaults also changed for new domains, new sites on existing accounts, and free-tier zones that never saved a preference. Training and Agent crawlers are now blocked on pages displaying ads, and the Agent category covers fetchers like ChatGPT-User, which pull pages live when someone asks about your topic.
Firewalls Are Blocking Crawlers Nobody Chose to Block
Cloudflare’s AI controls get most of the attention. The more troubling cases involve settings the site owner never touched at all.
Hostinger users have been reporting challenge pages served to Googlebot since late 2025. According to a detailed write-up of the complaints, support eventually admitted that its own system engineers had enabled aggressive bot protection at the platform level, upstream of the customer’s site, CDN, and plugins.
One affected site had roughly 150 pages disrupted, with Search Console filling up with “Page with redirect” errors. At least one page fell from the first two pages of results to fully deindexed.
HostGator customers found a similar layer: a Cloudflare partner integration built into the shared hosting stack that issues managed challenges to Googlebot. There’s no dashboard toggle for it, so switching off Bot Fight Mode on their own account changes nothing.
Agency setups create the same risk from a different direction. In a case Jonathan Bird documented, a managed IT provider enabled a Cloudflare setting to block all bots in July 2026, which took out organic search, Google Ads, and every Merchant Center listing for two weeks.
Why Crawler Blocks Rarely Show Up in Your Reports
Most of these blocks pass every standard check. That’s why they tend to surface only after rankings or citations have already dropped.
Challenge pages often return 200 OK. Many security plugins serve their CAPTCHA screen with a success status, so the server reports nothing wrong while Googlebot indexes the challenge text. In the worst cases, Google consolidates real URLs under a canonical pointing at the challenge screen.
Robots.txt can be clean while the firewall refuses the crawler. The edge layer acts before a request reaches your server, so a robots.txt audit passes even when the crawler never gets through.
User-triggered fetchers bypass robots.txt entirely. OpenAI’s December 2025 documentation revision removed robots.txt compliance language for ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt because a person requested the page. For those agents, your CDN or firewall is effectively the only control in play.
No AI crawler has a Search Console. Search Console reports what Googlebot experienced. There’s no equivalent for OAI-SearchBot, Claude-SearchBot, or PerplexityBot, so a refused AI crawler triggers no alert anywhere.
Training Crawlers and Search Crawlers Need Different Rules
Part of the confusion comes from treating “AI crawlers” as one group. Each major AI company runs separate agents for training, for building a search index, and for live fetches a user requests.
| Company | Training | Search index | User-triggered fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None separate | PerplexityBot | Perplexity-User |
| Google-Extended (token) | Googlebot | None separate |
The distinction determines what you lose. OpenAI states that sites opted out of OAI-SearchBot won’t appear in ChatGPT search answers, while blocking GPTBot only affects training.
Google works differently. Google-Extended is a control token covering Gemini training and certain grounding uses, but Googlebot does the actual crawling for Search, including AI Overviews and AI Mode.
So blocking Google-Extended keeps you in Search, and blocking Googlebot removes you from both. For most publishers, the defensible setup is allowing the search and user agents, then making a separate, deliberate call on training.
How to Audit AI Crawler Access in Five Steps
Run these in order. Each one catches a layer the previous check can’t see.
Check your edge settings first. In Cloudflare, open the AI crawler controls and confirm Training is set to Disallow AI Training rather than Block if you want to stay in search. Then review Security Events for 403s or challenges served to verified bots, and check whether Bot Fight Mode is on, since it can’t be exempted with a Web Application Firewall (WAF) custom rule.
Ask your host what sits upstream. Both the Hostinger and HostGator cases involved protection the customer couldn’t see from their own dashboard. A support ticket asking specifically about platform-level bot challenges takes five minutes.
Run a live URL inspection on key pages. Search Console’s live test returns a screenshot of what Googlebot actually received. A challenge page or loading spinner in that screenshot means you’re blocked, whatever the status code says.
Test each AI agent’s response. Request key pages using each bot’s user agent string, then compare status codes and response size against a normal browser request. Treat this as a first pass only, because Cloudflare verifies bots by IP address and a spoofed user agent from your laptop may be handled differently from the real crawler.
Confirm against server logs. Logs are the only record of what genuine crawlers received. Look for 403s and challenge responses to Googlebot, OAI-SearchBot, Claude-SearchBot, and PerplexityBot, and watch for bots that have stopped appearing altogether.
How to Keep Crawler Access Monitored Over Time
A one-time audit tells you what’s true this week. Both the zoundroom.com and Hostinger cases involved settings that changed after everything had been working fine, so the ongoing layer matters more than the first check.
That ongoing layer is where the Semrush SEO Toolkit does most of the work in my own setup.
Site Audit runs scheduled crawls with more than 140 technical checks, so pages that start returning errors or become uncrawlable appear as new issues between runs. It also flags changes to your robots.txt file, which matters more now that Cloudflare’s managed controls can write directives into that file on your behalf.
Its AI Search Health and Blocked from AI Search widgets show which major AI search agents your robots.txt restricts, covering OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, and Claude-User. You also see how many pages each restriction affects.
One limit deserves a plain statement. Those widgets read robots.txt, so an IP-verified block at the edge won’t appear there, and Cloudflare’s Security Events remain the place to confirm what verified crawlers actually received.
Position Tracking covers the symptom side with daily rankings against competitors. An access failure tends to pull down many unrelated keywords at once, while algorithm updates and competitor gains usually concentrate in specific topics, so the shape of a drop is an early diagnostic signal.
Organic Traffic Insights connects Google Analytics and Search Console data with
Once access is confirmed, the AI Visibility Toolkit is the natural follow-up. It shows whether restored access is turning into mentions and citations across ChatGPT, Perplexity, Gemini, and Google’s AI surfaces.
SEO Toolkit plans start at $139 per month, or $117.33 per month billed annually, covering Site Audit, Position Tracking, and five monitored websites. The Starter tier at $199 per month adds an AI-ready Site Audit and 50 tracked AI prompts, and both tiers come with a free trial.
Start your free Semrush SEO Toolkit trial
The above link is an affiliate link, which means 99signals earns a commission if you sign up through it. This doesn’t cost you anything extra, and it doesn’t influence what I recommend. I’ve been covering
The Bottom Line
Most AI visibility advice starts with content, structure, and authority. All of it assumes the crawler can read the page, and for a growing number of sites that assumption is wrong because of a setting nobody consciously chose.
Cloudflare’s September changes fixed the most dangerous default for sites that select Disallow AI Training. They didn’t touch host-level protection, plugin challenge pages, legacy firewall rules, or the absence of any console showing what AI crawlers experience.
The order of operations matters here. Confirm that every crawler you care about can reach your key pages, then invest in content, because a team that skips the first step can spend a quarter optimizing pages no crawler was ever allowed to fetch.