ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning

Short summary

ReCoLoRA addresses catastrophic forgetting in LoRA-based continual fine-tuning through spectrum-aware recursive consolidation. Instead of stacking adapters on frozen weights, it re-decomposes the current effective weight before each new task, preserving learned knowledge while opening fresh capacity. Experimental results on GLUE benchmarks show comparable or better performance than existing LoRA variants with fewer parameters.

• •Novel method: Recursive consolidation re-decomposes weights before each new task, not frozen originals
• •Addresses LoRA limitation: Prevents task interference and catastrophic forgetting in multi-task sequences
• •Validated results: Outperforms LoRA, PiSSA, AdaLoRA, DoRA on GLUE benchmarks with fewer parameters

Generated with AI, which can make mistakes.