BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models

Short summary

BaLoRA extends LoRA with Bayesian parameterization for uncertainty quantification and adaptive noise injection, improving accuracy and confidence calibration with minimal compute overhead. Testing shows it narrows the full fine-tuning gap on natural language reasoning and vision tasks. For materials science, BaLoRA produces zero-shot uncertainty estimates that correlate more strongly with model errors than ensembles.

• •Bayesian extension of LoRA adds uncertainty quantification with minimal parameter overhead
• •Improves prediction accuracy and narrows full fine-tuning gap on NLP and vision tasks
• •Zero-shot uncertainty estimates better predict model error than trained ensembles

Generated with AI, which can make mistakes.