The headline needs to capture: KV cache quantization, Qwen 35B, 8x context, 12GB VRAM, and the key technique.

Short summary

By quantizing the KV cache to q8_0 (halving its VRAM footprint) and offloading MoE experts to CPU, you can run Qwen 35B with 8× longer context (32k vs 4k) on a 12GB RTX 4070 with negligible speed or quality loss. The author benchmarks f16 vs q8_0 KV cache, showing identical decode speed and imperceptible perplexity delta. Three flags (--cpu-moe -ctk q8_0 -ctv q8_0) unlock agent-grade context windows on consumer hardware.

• •q8_0 KV cache halves VRAM usage with no measurable quality or speed regression
• •Three flags enable 32k context on 12GB VRAM for agent workloads
• •Context size should be tiered by workload: 4k for chat, 8k for light agents, 32k for heavy multi-tool agents

Generated with AI, which can make mistakes.