Exactly what Promptopus looks at

Promptopus reads your site the way most AI crawlers do: it downloads the pages and files your site already serves to every visitor and looks at the HTML, without running JavaScript. It never logs in, fills in forms or adds anything to a basket, and it always identifies itself as PromptopusBot rather than pretending to be an AI company’s crawler.

What we request

For each check we make at most 10 ordinary web requests to your site:

• Your homepage, over HTTPS first (plain HTTP only if HTTPS doesn’t answer).
• /robots.txt, the file that tells crawlers what they may read.
• Your XML sitemap: the one robots.txt names, otherwise /sitemap.xml.
• /llms.txt, if you have one.
• For online shops: one product page, found from your homepage’s links or your product sitemap.

We follow at most three redirects, wait no more than five seconds for each request, and read no more than 1.5 MB of any page.

What we look for

Access: can AI crawlers get in?

• AI search crawlers in robots.txt: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude), PerplexityBot (Perplexity), Googlebot (Google Search, including AI Overviews and AI Mode) and Bingbot (Bing and Microsoft Copilot). Blocking any of these is a failure, because it keeps you out of those tools’ answers.
• AI training crawlers: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent. Blocking these is a business choice, so we only note it. It doesn’t stop you appearing in AI search answers.
• robots.txt errors: a server error on robots.txt makes Google treat the whole site as off-limits until it’s fixed.
• noindex and nosnippet on your homepage, in the page or the X-Robots-Tag header. Google uses these to keep a page out of its results, including AI Overviews and AI Mode.
• Cloudflare: if your site uses Cloudflare, we remind you to check its AI crawler settings, which we can’t see from outside.

Content: can they read it?

• Text in the HTML: how much of your homepage is readable without running JavaScript.
• Title, description and headings, the page’s own summary of itself.
• Language and canonical address: whether the page says what language it’s in, and doesn’t point search engines to another site.

Structured data: do they understand you?

• schema.org structured data (JSON-LD or microdata), and whether each block is valid JSON.
• Organization or LocalBusiness details: name, website, logo, contact details and social profiles.

Discovery: can they find every page?

• XML sitemap, and whether robots.txt points to it.
• llms.txt: a proposed plain-text guide for AI tools. No major AI company has confirmed it uses these files, so it never counts against you.

Online shops: will they recommend your products?

• Product structured data on a product page: name, image, price, currency and stock, plus description, brand, SKU and barcode (GTIN or MPN).
• Prices: whether the price in your data appears on the page. We only flag this when the page shows prices in the same currency.
• Reviews: whether your rating is in the product data, and whether a review app we recognise is showing reviews AI tools can’t see.
• Shipping and returns in your product or organisation data.
• What we can’t see: Google Merchant Center and, for Shopify, the AI shopping channels in your admin. We point you to both.

How the results are graded

• Failed: something that stops AI tools reaching or understanding your site. Fix these first.
• Warning: worth fixing, but AI tools can still work with your site.
• Note: information, a choice you’ve made, or something we can’t see from outside.
• Passed: checked and fine.

How sure the results are

Every check is based on what your site shows publicly and on what AI companies publish about their crawlers. Some things can’t be seen from outside, such as firewall rules that treat AI crawlers differently from us, and we say so where it matters. AI companies change their crawlers and the way they choose answers, and nobody outside those companies controls what they recommend. Promptopus removes the technical reasons to skip you; it can’t promise a recommendation.

How we avoid becoming a nuisance

• Results for each domain are reused for 24 hours, so a site is checked at most once a day however many people ask.
• Each visitor can run 3 checks a minute and 10 a day, and must pass a Cloudflare Turnstile check.
• We only check public websites on the standard web ports, never IP addresses or internal networks.

Our crawler

Requests come from Cloudflare’s network and identify themselves with this user agent:

PromptopusBot/1.0 (+https://promptopus.co.uk/bot)

TotemEngine

Promptopus runs on TotemEngine, the crawl engine Advent Digital is building for all of its free tools, alongside PatchTurtle.

Opting out

If you run a site and don’t want it checked, email hello@promptopus.co.uk from an address at that domain and we’ll add it to our exclusion list. You can also block the user agent above at your firewall.