Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE

Short summary

Jet-Long introduces a tuning-free method to extend LLM context windows beyond training limits using dynamic bifocal RoPE, achieving 2–5pp accuracy gains. It recovers short-context performance while scaling to 128K tokens with ≤4% inference overhead. Compatible with existing architectures, enabling practical long-context deployment without model retraining.

• •Zero-shot context extension method using dynamic bifocal RoPE that adapts rescaling factor based on sequence length
• •Achieves 2–5pp accuracy improvements on long-context benchmarks while maintaining short-context fidelity
• •Scales to 128K context with minimal inference overhead (≤4%), compatible with hybrid attention architectures

Generated with AI, which can make mistakes.