Your HTML is fine. The CDN still blocks the bot.

Short summary

CDN and WAF rules are silently blocking AI crawlers from accessing content that site owners believe is open. Cloudflare Radar data shows 5.2% of AI bot requests get 403s and 1.3% get 429s at the edge, often from default settings rather than intentional policy. The fix is to grep access logs for AI crawler User-Agents and check actual status codes returned.

• •Over 6% of AI crawler requests are refused at the CDN edge (403/429), often by default WAF rules
• •robots.txt and browser testing can't detect CDN-level blocks — only access logs reveal the truth
• •Grep access logs for GPTBot, ClaudeBot, etc. to verify what status codes AI crawlers actually receive

Generated with AI, which can make mistakes.