Head-level attention fusion trims compute

Short summary

HydraHead splits transformer attention heads into full (25%) and linear (75%) variants, reducing attention compute by ~40% while maintaining benchmark performance. The technique works even at extreme mixing ratios (7:1 linear-to-full) and outperforms traditional layer-wise hybrid approaches. Potential impact: larger models and context windows on edge-class hardware.

• •Head-level attention mixing reduces FLOPs by ~40% vs traditional transformers
• •Maintains benchmark performance even at 7:1 linear-to-full attention ratio
• •Enables larger models and context windows on edge-class hardware

Generated with AI, which can make mistakes.