Let AI search crawlers read your site

AI companies use separate crawlers for search and for training. The search crawlers decide whether your pages can appear, and be linked, in AI answers. The training crawlers only collect material for training future models. You can keep the second group out and still let the first group in.

1. Open your robots.txt

Visit yourdomain/robots.txt in a browser to see the current rules. Then find where to edit it:

• Shopify: Online Store › Themes › ⋯ › Edit code. If there’s no robots.txt.liquid template, add one (Add a new template › robots). Shopify’s default rules don’t block AI search crawlers, so a block usually comes from an edit or an app.
• WordPress: in Yoast SEO go to Tools › File editor; in Rank Math go to General Settings › Edit robots.txt. Otherwise it’s a robots.txt file in your site’s main folder.
• Other platforms: look for robots.txt in your SEO settings, or ask your developer.

2. Remove the blocks on search crawlers

Look for groups like these and remove them:

User-agent: OAI-SearchBot
Disallow: /

Do the same for Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. Also check there’s no Disallow: / under User-agent: *, which blocks every crawler that isn’t named elsewhere.

3. Keep training blocked, if you prefer

To opt out of AI training only, block the training crawlers by name:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

4. Check again

Changes can take a day or so to reach each AI company. Run Promptopus again to confirm the new rules read the way you expect. If you use Cloudflare, check its AI crawler settings too, because a firewall block overrides robots.txt.