Testing PyTorch 2.13 MPS FlexAttention on M1 Max: Up to 7.83x Faster for Sparse Attention

Short summary

Benchmarks PyTorch 2.13's new MPS FlexAttention against standard SDPA on an M1 Max, finding a 7.83x speedup for sparse local-window attention at 32,768 tokens. For standard causal attention, SDPA remains ~19x faster. FlexAttention on Apple Silicon is promising for long, sparse patterns but not a universal win, and the API remains unstable in 2.13.

• •FlexAttention achieves 7.83x speedup over SDPA for 256-token local window at 32K sequence length on M1 Max
• •SDPA is ~19x faster for standard causal attention — FlexAttention only wins on sparse patterns
• •Includes reproducible benchmark code, methodology, and hardware specs

Generated with AI, which can make mistakes.