Before an AI engine can cite your website, its crawler has to read your website. That sounds too obvious to write down — except that a meaningful share of the sites we audit are blocking AI crawlers somewhere, usually without anyone on the team knowing. To an answer engine, a blocked site doesn't look "penalized". It looks like nothing. It simply isn't there.
Meet the crawlers
The bots that matter for AI visibility, and what each one feeds:
- GPTBot — OpenAI's training crawler.
- OAI-SearchBot / ChatGPT-User — ChatGPT search indexing and live browsing. Blocking GPTBot alone does not block these, and vice versa — decide per bot.
- ClaudeBot / Claude-User — Anthropic's crawler and Claude's live browsing.
- PerplexityBot / Perplexity-User — Perplexity's index and live retrieval.
- Google-Extended — controls whether Google may use your content for Gemini training. It does not affect regular Google indexing or AI Overviews.
- Bingbot — ordinary search crawler, but it also feeds Microsoft Copilot.
Where the blocks hide
Robots.txt is the place everyone checks, and it's the place we least often find the problem. The usual culprits, roughly in order:
- CDN bot protection. Cloudflare, Akamai, and friends ship AI-bot blocking as a toggle — and some plans have enabled versions of it by default. Your robots.txt can say "welcome" while the CDN returns 403s.
- WAF and rate-limiting rules written years ago against "aggressive scrapers" that now match AI user agents.
- Copy-pasted robots.txt templates. A wave of 2023-era templates blocked every AI bot on principle. Plenty of sites shipped one and forgot.
- Hosting-level "security" plugins (common on WordPress) with a block-AI checkbox someone ticked.
Should you ever block them?
It's a real business decision for publishers whose content is the product. But if your website exists to win customers, blocking AI crawlers is unilateral disarmament: ChatGPT will still answer the customer's question — from your competitor's site. You can also split the decision: allow search/browsing bots (OAI-SearchBot, Perplexity-User) that produce cited answers, while blocking pure training bots, if that trade-off fits your business.
How to check your own site in five minutes
- Read
yourdomain.com/robots.txtand look for the user agents above. - Check your CDN dashboard for AI/bot-blocking toggles — this is the big one.
- Fetch a key page with an AI user-agent string (curl with
-A "GPTBot") and confirm you get a 200 and real HTML, not a challenge page. - Ask Perplexity a question your site answers, and see whether you're citable in practice.
Or let us do it — the free AEO audit checks crawler access for every major AI bot, at the robots.txt layer and the network layer, and tells you exactly what's blocked and where.
Frequently asked questions
Does blocking GPTBot stop ChatGPT citing me?
Not on its own. GPTBot is the training crawler; OAI-SearchBot and ChatGPT-User handle search indexing and live browsing, and they obey their own rules. Blocking one and allowing the others is a legitimate strategy — but doing it by accident, which is the common case, leaves you invisible in the surface that actually sends traffic.
My robots.txt allows everything. Can I still be blocked?
Yes, and this is where most blocks live. A CDN bot-protection toggle, a WAF rule written years ago against scrapers, or a hosting security plugin can all return 403s to AI user agents while robots.txt says welcome. Robots.txt is a request; the network layer is enforcement.
How do I test it myself?
Fetch a real page with an AI user agent — curl -A "GPTBot" https://yourdomain.com/ — and confirm you get a 200 with real HTML rather than a challenge page. Repeat for OAI-SearchBot, ClaudeBot and PerplexityBot, because they can be treated differently.
Should a publisher block AI crawlers?
If your content is the product you are selling, it is a real commercial decision and reasonable people land on both sides. If your website exists to win customers, blocking is unilateral disarmament — the engine still answers the question, using a competitor's page.