Scraping Indian government open data in 2026: what actually works

Short summary

A maintainer of an open-source Indian village mapping project shares five hard-learned lessons from daily scraping of government data feeds. Key issues include WAFs that fingerprint and block python-requests User-Agent strings (returning misleading 502s), using Unix exit code 75 for clean upstream-failure handling in CI, and reverse-engineering client-rendered government portals like myScheme whose internal APIs require matching Origin/Referer headers and rotate access keys with frontend releases.

• •data.gov.in WAF blocks python-requests by User-Agent fingerprint, returning fake 502s instead of 403s
• •Use Unix exit code 75 in CI pipelines to distinguish clean upstream failures from real bugs
• •myScheme's internal API requires exact Origin/Referer headers and rotates keys with frontend deploys

Generated with AI, which can make mistakes.