How much VRAM do you actually need to run Llama 3 or Gemma locally?

Short summary

GPU VRAM for local LLMs has three components: model weights, KV cache (growing with context length), and overhead. KV cache is often the OOM culprit, not weights—two models with similar parameters can need vastly different memory depending on architecture. Use the provided formula: weight size plus ~1GB per 8K context for 7-8B models, plus 10% overhead.

• •KV cache is the hidden bottleneck; context length matters more than parameter count
• •Identical-looking models (e.g., Gemma vs Llama) can have 2-3GB memory differences due to architecture
• •Calculate actual VRAM needs per model, quantization, and context length before downloading

Generated with AI, which can make mistakes.