9,964 Readings a Second. Then I Pulled the Plug. Hazshield AI - Part 2

Short summary

A developer details building Hazshield, a Rust-based industrial sensor gateway handling 10,000 readings/second with safety-critical prioritization where alarms always win over bulk storage. The post walks through chaos engineering tests including killing Postgres mid-cascade during a simulated gas leak, demonstrating graceful degradation where alarm evaluation continued untouched while storage buffered and recovered. The architecture uses atomic config snapshots, rate limiting that yields to safety checks, and file-based fallback for Redis outages.

• •Rust gateway processes 10k sensor readings/sec with p99 latency of 3ms, prioritizing alarm delivery over bulk storage
• •Chaos test: killing Postgres at full load mid-gas-leak caused zero request errors; system degraded storage to 1-in-10 sampling while alarms continued uninterrupted
• •Redis outage fallback spills violations to append-only local files that auto-replay on recovery, proving resilience through absence of data loss

Generated with AI, which can make mistakes.