Your CDN Might Be Blocking ChatGPT From Your Docs (And robots.txt Won't Show It)

Short summary

Roughly 27% of sites unintentionally block AI crawlers at the CDN or WAF layer, even when robots.txt looks permissive. This post shows how to test for hidden blocks using curl with AI bot user agents, configure Next.js robots.txt for major AI crawlers, and check edge logs for 403s. Being crawlable is the precondition for AI citation; structured, extractable content is what wins citations.

• •~27% of sites block AI crawlers at CDN/WAF layer despite permissive robots.txt
• •Test with curl using AI bot user agents (GPTBot, ClaudeBot, PerplexityBot) and check edge logs for 403s
• •robots.txt is necessary but not sufficient — CDN rules are what actually block crawlers

Generated with AI, which can make mistakes.