You're Not Paying for Compute. You're Paying for Memory Bandwidth

Short summary

For production LLM inference, the real bottleneck isn't compute (FLOPs) but memory bandwidth — specifically the cost of moving the KV cache on every decode step. Decode-heavy workloads are bandwidth-bound, not compute-bound, making FLOPs-per-dollar buying decisions systematically wrong. The fix is architectural: disaggregate prefill from decode, right-size batch and context, and treat bandwidth as the scarce resource.

• •LLM decode is memory-bandwidth-bound, not compute-bound — FLOPs-per-dollar is the wrong metric
• •KV cache traffic grows nonlinearly with context length, making long context a bandwidth liability
• •Architectural fixes include prefill/decode disaggregation and treating bandwidth as the scarce resource

Generated with AI, which can make mistakes.