LoRA Doesn't Approximate Fine-Tuning. It Changes What You Can Teach

Short summary

LoRA isn't a cheaper version of fine-tuning—it's a structurally different optimization problem that reshapes which solutions are learnable. For preference alignment, rank constraints force models to capture surface correlates (politeness, length) rather than deeper distribution shifts alignment requires. Use LoRA only for genuinely local updates; for preference optimization, prefer full-parameter tuning or higher ranks combined with behavioral tests.

• •LoRA solves a different optimization problem than full fine-tuning, not a cheaper approximation
• •Rank constraints bias preference optimization toward surface features over deep alignment
• •Evaluate LoRA preference models with adversarial tests; surface-feature shortcuts collapse on reparaphrased prompts

Generated with AI, which can make mistakes.