Skip to content
Promptopus

Fix guide

Check Cloudflare’s AI crawler settings

Make sure Cloudflare isn’t blocking the AI search crawlers you want, separately from your robots.txt.

Time
10 minutes
Difficulty
Easy
Updated
28 October 2026

Cloudflare can block AI crawlers at its firewall, before they ever read your robots.txt. Many newer Cloudflare sites have AI crawler blocking switched on by default, and from outside it looks the same as the site being down. That’s why we can’t check it for you.

Check it

  1. Sign in to Cloudflare and open your domain.
  2. Open AI Crawl Control. It lists AI crawlers and whether each is allowed or blocked. (Older dashboards show a Block AI bots setting under Security.)
  3. Allow the search crawlers you want, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. You can keep training crawlers such as GPTBot blocked.
  4. If you use custom firewall rules, check none of them block these crawlers.

Names and menus vary by plan. If you’re unsure, ask whoever manages your Cloudflare account.

For your developer

Review AI Crawl Control (per-crawler allow/block) and any "Block AI bots" or custom WAF rules. Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot; training crawlers can stay blocked. Verified bots should not be challenged.

General guidance only. Take a backup before making changes, and test on a staging copy where you can. Nobody can guarantee what AI tools recommend.