GPTBot, ClaudeBot, PerplexityBot: Is Your Site Blocking AI Crawlers?

Before an AI engine can cite your website, its crawler has to read your website. That sounds too obvious to write down — except that a meaningful share of the sites we audit are blocking AI crawlers somewhere, usually without anyone on the team knowing. To an answer engine, a blocked site doesn't look "penalized". It looks like nothing. It simply isn't there.

Meet the crawlers

The bots that matter for AI visibility, and what each one feeds:

• GPTBot — OpenAI's training crawler.
• OAI-SearchBot / ChatGPT-User — ChatGPT search indexing and live browsing. Blocking GPTBot alone does not block these, and vice versa — decide per bot.
• ClaudeBot / Claude-User — Anthropic's crawler and Claude's live browsing.
• PerplexityBot / Perplexity-User — Perplexity's index and live retrieval.
• Google-Extended — controls whether Google may use your content for Gemini training. It does not affect regular Google indexing or AI Overviews.
• Bingbot — ordinary search crawler, but it also feeds Microsoft Copilot.

Where the blocks hide

Robots.txt is the place everyone checks, and it's the place we least often find the problem. The usual culprits, roughly in order:

• CDN bot protection. Cloudflare, Akamai, and friends ship AI-bot blocking as a toggle — and some plans have enabled versions of it by default. Your robots.txt can say "welcome" while the CDN returns 403s.
• WAF and rate-limiting rules written years ago against "aggressive scrapers" that now match AI user agents.
• Copy-pasted robots.txt templates. A wave of 2023-era templates blocked every AI bot on principle. Plenty of sites shipped one and forgot.
• Hosting-level "security" plugins (common on WordPress) with a block-AI checkbox someone ticked.

Should you ever block them?

It's a real business decision for publishers whose content is the product. But if your website exists to win customers, blocking AI crawlers is unilateral disarmament: ChatGPT will still answer the customer's question — from your competitor's site. You can also split the decision: allow search/browsing bots (OAI-SearchBot, Perplexity-User) that produce cited answers, while blocking pure training bots, if that trade-off fits your business.

How to check your own site in five minutes

• Read yourdomain.com/robots.txt and look for the user agents above.
• Check your CDN dashboard for AI/bot-blocking toggles — this is the big one.
• Fetch a key page with an AI user-agent string (curl with -A "GPTBot") and confirm you get a 200 and real HTML, not a challenge page.
• Ask Perplexity a question your site answers, and see whether you're citable in practice.

Or let us do it — the free AEO audit checks crawler access for every major AI bot, at the robots.txt layer and the network layer, and tells you exactly what's blocked and where.