How LLMLingua compresses prompts: perplexity, deletion, and trust

Short summary

LLMLingua from Microsoft Research compresses prompts by using a small language model to identify and delete tokens the target LLM could predict anyway, measured via perplexity. The first version uses a coarse-to-fine pipeline with budget control and iterative token-level deletion; the second generation distills GPT-4's compression choices into a bidirectional encoder for faster, context-aware decisions. Results show up to 20x compression with minimal task performance loss, though output text becomes telegraphic and unreadable for humans.

• •LLMLingua deletes high-predictability tokens using a small LM's perplexity scores
• •V2 distills GPT-4 compression decisions into a bidirectional encoder for speed and accuracy
• •Up to 20x compression on benchmarks; compressed text is unreadable for humans but effective for LLMs

Generated with AI, which can make mistakes.