CausalGate: Causal Importance Distillation for Transformer Module Pruning

Short summary

CausalGate is a framework for compute-efficient transformer inference that uses causal interventions—zeroing out individual attention and MLP sub-layer outputs—to measure true semantic damage, then distills this importance hierarchy into static scalar gates with zero runtime overhead. Evaluated on TinyLlama-1.1B, Qwen2.5-3B, and Llama-3.1-8B, it consistently outperforms dynamic routing and layer-skipping baselines across language modeling and commonsense reasoning benchmarks, translating theoretical compute savings into real hardware latency reductions.

• •Uses causal interventions to measure sub-layer importance via KL divergence
• •Distills importance into static scalar gates with zero runtime overhead
• •Outperforms dynamic routing baselines on TinyLlama, Qwen2.5, and Llama-3.1

Generated with AI, which can make mistakes.