EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

Short summary

EntMTP dynamically adjusts LLM inference speculation depth based on local entropy, achieving 1.15–1.36x speedups over Hydra and Medusa baselines. Training-free scheduler toggles tree-based attention topologies to match context predictability, maximizing accepted-token throughput without sacrificing generation quality.

• •Entropy-guided approach dynamically adapts speculation depth during inference generation
• •Achieves 1.15–1.36x speedup improvements over existing multi-token prediction methods
• •Training-free, no model retraining required; applicable to existing foundation models

Generated with AI, which can make mistakes.