CARE-LoRA: Compressed Activation REconstruction for Memory-Efficient LoRA

Short summary

CARE-LoRA is a new PEFT method that tackles the activation memory bottleneck in LoRA fine-tuning by replacing full input activations with low-rank compressed activations from the LoRA branch. A lightweight reconstruction matrix computed during the forward pass restores gradient signals during backpropagation at negligible cost. Experiments across diverse models show competitive or superior performance versus standard LoRA while substantially reducing memory footprint. Code is publicly available on GitHub.

• •Proposes CARE-LoRA to compress activation memory in LoRA fine-tuning
• •Uses low-rank LoRA branch activations plus a lightweight reconstruction matrix for backprop
• •Achieves competitive performance with reduced memory; code open-sourced

Generated with AI, which can make mistakes.