The most expensive outages return HTTP 200

Short summary

The author argues that the most expensive bugs return HTTP 200—everything works correctly but costs silently spike. Using a LiteLLM prompt cache invalidation incident as an example, they show how reordering messages can break prefix caching, tripling token costs with no visible failure. The solution is treating cost like latency: assert on it in CI, monitor cache hit rates, and make cost regressions fail builds just like broken tests do.

• •Silent cost bugs return HTTP 200—output is correct but spend triples unnoticed
• •LiteLLM prompt cache invalidation from message reordering caused costs to spike with no alerts
• •Fix: treat cost as an engineering indicator—assert cache hit rates and cost per call in CI

Generated with AI, which can make mistakes.