Serving Agent Traffic Without Breaking Your Origin

Short summary

AI agent traffic is growing explosively — Cloudflare recorded 15x growth in user-action crawling across 2025, with retail and software sectors absorbing 40% of all AI crawler activity. The article argues for caching near-static machine surfaces (llms.txt, feeds, product pages) at the edge while keeping dynamic offer data fresh, rather than blocking agents. Real-world costs are significant: iFixit was hit nearly a million times daily by ClaudeBot, and Read the Docs faced $5,000/month in bandwidth from a single abusive crawler.

• •AI user-action crawling grew 15x in 2025; retail and software draw 40% of all AI crawler traffic
• •Cache static machine surfaces at edge, keep price/availability fresh — don't block verified agents
• •Anthropic's crawl-to-refer ratio hit 500,000:1; iFixit and Read the Docs both faced costly unmanaged crawler traffic

Generated with AI, which can make mistakes.