How We Scaled to 1 Million Connections — and the Silent Kernel Bug That Almost Broke It

Short summary

An engineering deep-dive on scaling a Java-based chat cluster to 1 million concurrent connections, covering kernel-level tuning that application monitoring can't see. The key incident: nf_conntrack table silently filled at 800K connections, dropping packets with no app-level error. Covers file descriptor limits, TCP buffer sizing via bandwidth-delay product math, interface queue length, and connection tracking table sizing, with concrete sysctl values and a capacity planning formula for buffer memory.

• •nf_conntrack table silently drops packets when full — invisible to app-level monitoring, only visible in dmesg
• •TCP buffer sizing must follow bandwidth-delay product math: RTT × interface_speed / 8, with send buffer ≥ receive window
• •Capacity planning: 4KB per socket × 2M connections ≈ 46GB in TCP buffer memory alone

Generated with AI, which can make mistakes.