Postmortem: Silent Data Loss in an In-Process Vector Store, and the Case for Durable Execution

Short summary

A postmortem of a silent data-loss bug in ZVec, an in-process vector database, where calling optimize() on a crash-recovered collection discarded WAL-replayed documents without any error signal. The root cause was a WAL handle lifecycle bug where segment recovery failed to reattach the handle, causing optimize() to skip persisting recovered data. The author argues that fixing individual bugs is insufficient and advocates for durable execution layers like Temporal where state ownership and recovery are explicit.

• •ZVec optimize() silently discarded crash-recovered documents, reporting false success signals
• •Root cause: WAL handle lifecycle bug — recovered segment not reattached, so optimize() skipped persisting data
• •Author argues for durable execution/event-sourcing layers like Temporal to eliminate implicit durability guarantees

Generated with AI, which can make mistakes.