Heterogeneous AI Parallelism on Android: Orchestrating NPU, GPU, and DSP for On-Device Inference

Short summary

A deep technical guide on heterogeneous AI parallelism for Android, explaining how to orchestrate inference across NPU, GPU, and DSP instead of relying on a single processor. It covers the strengths and trade-offs of each accelerator, the memory-wall problem, zero-copy buffers, and Google's AICore system service that abstracts model management. Essential reading for engineers building on-device LLM or diffusion model apps.

• •NPU excels at quantized tensor ops but fails on unsupported layers; GPU offers flexibility but thermally throttles; DSP handles real-time sensor preprocessing
• •Zero-copy buffers and shared memory are critical to avoid the memory wall when moving tensors between accelerators
• •Google's AICore turns AI models into system services, reducing redundant model weights and abstracting hardware complexity

Generated with AI, which can make mistakes.