Compile-Time Memory Layout Optimization for On-Device ML Models

Short summary

On-device ML inference on Android stalls due to GC pauses from uncontrolled allocations in the 12KB-256KB danger zone, not model performance. Three concrete strategies—baseline profile hints (30-40% pause reduction), direct ByteBuffer I/O (50-60%), and JNI-boundary isolation (80-90%)—progressively eliminate frame drops. Start with profiles; advance to native isolation only when real-time inference alongside UI rendering is required.

• •GC stalls during ML inference come from allocation patterns in the 12KB-256KB range, not model speed
• •Three progressive strategies: baseline profiles (30-40% reduction), ByteBuffer I/O (50-60%), full JNI isolation (80-90%)
• •Implement incrementally—profiles are low effort; reserve full native isolation for real-time UI-sync use cases

Generated with AI, which can make mistakes.