What AI Crawlers Actually Do to a Small Blog: 9 Days of Logs

Short summary

Analysis of 9 days of nginx logs reveals AI crawlers now vastly outnumber human visitors on small blogs, with ChatGPT-User being the dominant source via live fetches for human queries. Author distinguishes training crawlers (GPTBot, ClaudeBot—respect robots.txt) from answer-engine fetchers (ChatGPT-User, Perplexity-User—live user-initiated requests), showing that blocking crawlers costs referrals without preventing training. Critical finding: AI crawlers don't execute JavaScript, so client-side structured data is invisible; metadata must live in server-rendered HTML to be indexed by AI systems.

• •AI bots now dominate traffic on small blogs—5.2% of total requests, significantly more than human visitors
• •Distinguish training crawlers (GPTBot, ClaudeBot, respect robots.txt) from live answer-fetchers (ChatGPT-User, Perplexity-User, user-initiated requests)
• •AI crawlers don't execute JavaScript—client-side JSON-LD and meta tags are invisible; must be in server-rendered HTML for AI indexing

Generated with AI, which can make mistakes.