Depth-Staggered Fibonacci Spacing for Sparse Attention: Static Schedules Beat Learned Dilation and Extrapolate Where Dense Attention Fails

Short summary

Static Fibonacci-spaced attention with per-layer scaling outperforms learned approaches and extrapolates to 4x training length with minimal loss. Trade-off: 26% higher perplexity at training length. Fixed-offset design enables better length extrapolation than dense attention.

• •Static stagger beats learned per-layer alpha and fixed baselines on perplexity
• •Sparse attention maintains performance at 4x training length; dense attention perplexity rises 201%
• •Cost: 26% perplexity hit at training length versus dense baseline

Generated with AI, which can make mistakes.