Blocking GPTBot won’t take you out of ChatGPT search. Here’s what will.
Published 28 October 2026
AI companies use separate crawlers for training and for search. Here’s which is which, and how to set robots.txt so you’re found without being trained on.
Many sites added User-agent: GPTBot / Disallow: / to their robots.txt to keep their content out of AI training. That’s a reasonable choice, and it doesn’t remove them from ChatGPT search. What does is blocking a different crawler, often by accident.
Training crawlers and search crawlers are different
Most AI companies now run at least two crawlers:
| Company | Training | Search and answers |
|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot |
| Anthropic | ClaudeBot | Claude-SearchBot |
| Google-Extended (a control, not a crawler) | Googlebot | |
| Perplexity | – | PerplexityBot |
| Microsoft | – | Bingbot |
Blocking the training column keeps your content out of future model training. Blocking the search column keeps you out of the answers people see, and out of the links in them.
The accidental block
The most common problem is a blanket rule:
User-agent: *
Disallow: /That blocks every crawler you haven’t named separately, including the search crawlers. Sites sometimes add it while being built and never remove it. Firewalls can have the same effect: Cloudflare, for example, can block AI crawlers before they read robots.txt at all.
A sensible setup
If you want to be found in AI answers but not used for training:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yourbusiness.co.uk/sitemap.xmlNot sure what yours says? Run a free Promptopus check: it reads your robots.txt the way each of these crawlers does and tells you which can get in.
This guide is general information, not legal or professional advice. Check details with your developer or host before making changes.