What running an LLM in production actually costs you

Short summary

A practitioner's deep-dive into the real infrastructure required to run LLMs in production at scale. Covers semantic caching with vector similarity (0.95 cosine threshold) to cut costs 40-50%, perceptual hashing for vision call deduplication, a subtle normalization bug with Gemini's truncated embeddings, and circuit breaker patterns with tiered degradation that prioritizes paying users during outages. Concrete code examples throughout.

• •Semantic caching with 0.95 cosine similarity cuts LLM costs 40-50% with no quality drop
• •Perceptual hashing deduplicates vision calls before expensive embeddings
• •Gemini Matryoshka truncation requires manual renormalization — silent bug if skipped
• •Circuit breaker with tiered degradation prioritizes paying users during model outages

Generated with AI, which can make mistakes.