In short: if your website goes through Cloudflare, two settings can stop crawlers from reading it, even when your robots.txt allows them: Bot Fight Mode, which challenges every bot it detects, and the AI settings (“AI Crawl Control”), which block some categories of AI crawlers. Both can be checked in a few minutes in the dashboard.
What I found
While testing my free audit on a website protected by Cloudflare, the analysis failed: every page returned an anti-bot challenge (“Managed Challenge”), except the robots.txt file. In a regular browser, the website loaded normally.
Cloudflare's security logs left no doubt: the Service column read “Bot fight mode”, for my crawler as well as for other automated visits coming from servers. The website had no custom rules: this mode, switched on with a single toggle, was filtering everything.
Bot Fight Mode: a blanket filter
Bot Fight Mode is Cloudflare's free anti-bot protection. It makes the bots it detects solve computationally expensive challenges. Its documentation is clear on two points:
- it can't be configured: it protects the whole domain, with no exceptions per page or per bot;
- a WAF rule can't get around it: “You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules”.
Bots verified by Cloudflare, such as those of the major search engines, are in principle excluded from the default configurations. Everything else gets the same treatment: SEO audit tools, crawlers and agents that aren't on that list.
The AI settings: the other switch
Since July 2026, Cloudflare groups AI traffic into three categories that can be set separately, including on the free plan:
- Search: crawlers that index your content to answer questions about it later;
- Agent: real-time automated activity on a person's behalf, such as an assistant opening a page on request;
- Training: crawlers that collect content to train a model.
For each one, you can block on all pages, block only on pages that show ads, or not block at all. For new domains added since 15 September 2026, Cloudflare blocks Training and Agent by default on pages with ads and keeps Search allowed. Existing domains keep their settings.
None of this shows up in your robots.txt. An audit that only reads that file can't detect it.
How to check your website
- Security events: in Cloudflare, Security → Analytics, Events tab. If the Service column shows “Bot fight mode” on legitimate requests, you are affected.
- Bot Fight Mode: Security → Settings, “Bot traffic” filter. Note whether it is on.
- AI settings: AI Crawl Control menu. Check what is allowed for Search, Agent and Training.
What to change, depending on your plan
Free plan. Bot Fight Mode is all or nothing. If your visibility in search engines and assistants matters, turn it off and replace it with targeted custom rules against the abuse you actually see (countries, sensitive paths, rate limiting). IP access rules, which take effect before Bot Fight Mode, also exist, but the new dashboard only offers them once a first rule has been created.
Paid plan (Pro and above). Move to Super Bot Fight Mode: a custom rule with the Skip action lets a specific bot or address through. You keep the protection while choosing what gets in.
In every case. In AI Crawl Control, keep at least Search allowed if you want to be mentioned, and Agent if you want assistants to be able to open your pages when a user asks them something. Training is an editorial choice that is yours to make.
What this changes for an audit
This case changed my tool: when a website shows this challenge, the free audit now produces a partial report instead of failing, and it flags websites served by Cloudflare so their owners check these settings. A poorly tuned anti-bot protection can make a website invisible to tools, and to some of the assistants it hopes will mention it.