RAG Retrieves. Fine-Tuning Forgets. HyperNetworks Inject - and Now We Have the Scaling Laws.

Short summary

A paper from Nace AI and Purdue University introduces HyperNetwork-based knowledge injection as a third alternative to RAG and fine-tuning. A secondary network generates LoRA adapters from fact batches at inference time while the base model stays frozen. The team derived the first systematic scaling laws, showing power-law scaling across loss, accuracy, and OOD generalization, with steeper OOD scaling than LoRA fine-tuning at larger target model sizes.

• •HyperNetworks generate LoRA adapters from facts at inference time, keeping the base model frozen
• •First systematic scaling laws show power-law relationships for loss, accuracy, and OOD generalization
• •Scaling the target LLM beats scaling the HyperNetwork; OOD generalization outperforms LoRA fine-tuning at scale

Generated with AI, which can make mistakes.