Back to feed
arXiv cs.LG
arXiv cs.LG
7/15/2026
Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels

Semidirect Fourier Delta Attention: Phase-Controlled Delta Memory with Constructive Chunk-WY Kernels

Short summary

SFDA generalizes Kimi Delta Attention by replacing real diagonal decay with block-rotational Fourier control, enabling phase-controlled memory in linear attention. The authors prove a constructive chunk-WY factorization with bounded rank growth, yielding exact affine chunk transfer and formal stability bounds. Toy experiments show SFDA learns cyclic memory where the phase-disabled KDA baseline stays near chance, though large-scale LLM comparisons are deferred.

  • •Phase-controlled generalization of Kimi Delta Attention using Fourier block-rotational decay
  • •Proves constructive chunk-WY factorization with bounded rank growth and exact chunk transfer
  • •Toy experiments demonstrate cyclic memory learning where KDA baseline fails

Generated with AI, which can make mistakes.

#research-breakthrough
Read full article at arXiv cs.LG

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

Depth-Staggered Fibonacci Spacing for Sparse Attention: Static Schedules Beat Learned Dilation and Extrapolate Where Dense Attention Fails
arXiv cs.CL
SpecLA: Efficient Speculative Decoding for Linear-Attention Models
arXiv cs.CL
Gaussian Mixture Attention: Linear-Time Sequence Mixing via Probabilistic Latent Routing
arXiv cs.LG
Training transformers where every layer W = V·Uᵀ from initialization reveals a corpus-determined optimal rank - looking for arXiv endorser (cs.LG) [D]
r/MachineLearning