VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs

Short summary

VarRate is a training-free KV cache codec that assigns each token a variable low-rank budget based on query salience, avoiding the irreversible eviction problem of token-selection methods. At a 20% budget on LongBench, it stays within 0.8 points of uncompressed models on Llama-3.1-8B and Qwen2.5-7B, outperforming uniform-rank ablations and matching KVzip at one-eighth the prefill overhead. The key insight is that rank should be allocated rather than evicted.

• •VarRate allocates variable low-rank budgets per token instead of evicting tokens, avoiding accuracy collapse
• •Stays within 0.8 points of uncompressed model at 20% KV budget on LongBench across Llama-3.1-8B and Qwen2.5-7B
• •Matches KVzip accuracy at ~1/8th the prefill overhead, with no training required

Generated with AI, which can make mistakes.