Quantized LoRA Adapters for On-Device LLMs: Hot-Swapping Task-Specific Behaviors on Android Without Reloading the Base Model

Short summary

Load a 4-bit quantized base model once on Android via mmap, then hot-swap small LoRA adapter files (~2MB each) in 50–90ms to switch between tasks without reloading. This avoids the 12GB RAM penalty of running separate models per task, enabling multi-adapter workflows on mobile with sub-100ms latency. ARM NEON-optimized fusion folds adapter weights into base layers with zero overhead, making multi-adapter on-device agents viable in production.

• •One quantized base model (4.2GB) + swappable LoRA adapters (2MB each) beats three separate models (12GB) in both RAM and task-switch latency
• •mmap-based loading keeps base model in page cache across adapter swaps, achieving 50–90ms switches versus 8–12s full reloads
• •NEON-optimized matrix fusion integrates LoRA deltas with zero inference overhead, enabling mobile agentic workflows with task specialization

Generated with AI, which can make mistakes.