My Engineering Passion: Why I built a sub-15ms proxy to stop LLM agent crashes

Short summary

The author built Selixes, an edge proxy that sits between LLM agents and upstream providers like OpenAI, providing sub-15ms mid-stream failover to a backup model (Google Gemini) when the primary provider drops or spikes. It uses atomic Redis operations for hard spend caps to prevent runaway API costs and exposes a standard OpenAI-compatible endpoint requiring only a base URL change. The architecture relies on optimistic TCP pre-warming and sliding-window health checks to minimize user-visible latency during failover events.

• •Selixes is an edge proxy providing sub-15ms failover for LLM API calls
• •Uses Redis INCRBYFLOAT for atomic spend caps and sliding-window health checks for pre-warming fallback connections
• •Google Gemini serves as the default fallback due to its global availability and speed

Generated with AI, which can make mistakes.