Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference

Short summary

This paper identifies a structural-role bias in attention-based KV cache eviction methods like H2O: on schema-dense inputs (e.g., nested JSON), non-content tokens such as delimiters and KEY tokens are over-retained while answer-carrying VALUE tokens are evicted, collapsing exact-match accuracy from 88% to 0% at a 5% budget. The authors propose a retraining-free, role-conditional allocation over SnapKV's windowed score that closes 63-98% of the H2O gap at sub-20% budgets using a single tuned hyperparameter and a lightweight 15 MB linear role probe.

• •KV cache eviction methods like H2O disproportionately retain structural tokens (KEYs, delimiters) over content tokens (VALUEs) on schema-dense inputs
• •Proposed role-conditional filter closes 63-98% of the H2O accuracy gap at sub-20% cache budgets without retraining
• •A 15 MB linear role probe supplies token-role labels at negligible inference cost

Generated with AI, which can make mistakes.