When Your Homelab Grows Up: How SQLite Took Down My k3s Control Plane

Short summary

A Hetzner k3s homelab quietly grew into a production-grade platform running dozens of operators, hammering the default SQLite/kine datastore until compaction entered a death-spiral: 1.36M rows, a 13.8 GB WAL that wouldn't checkpoint, CPU pinned at 99%, load average 79 on 8 cores. The author stopped the bleeding by truncating the WAL, then migrated the control plane to embedded etcd, shrinking 7.5 GB SQLite to 313 MB etcd and dropping load from 79 to 5.

• •Overgrown k3s homelab hit SQLite/kine compaction death-spiral: 1.36M rows, 13.8 GB WAL, load average 79 on 8 cores
• •Leader-election leases from dozens of operators generated ~55K dead revisions each with no compaction
• •Fix: truncate WAL for immediate relief, then migrate control plane to embedded etcd (7.5 GB → 313 MB, load 79 → 5)

Generated with AI, which can make mistakes.