Speculative Decoding on Mobile GPUs

Short summary

Implement speculative decoding on Android using Vulkan compute shaders (draft) and NNAPI (verify) to achieve 2-3x latency reduction and 22+ tokens/second. Thermal-adaptive batch scheduling prevents throttling while maximizing speculation depth (K=1 to K=6). Zero-copy KV-cache sharing via Vulkan external memory exports is critical—a naive copy would erase 15-20% of the speculation benefit.

• •Split-compute architecture: Vulkan for 150M draft model, NNAPI for 3-7B verify model
• •Thermal-aware scheduler adapts speculation depth from K=1 (hot) to K=6 (cool) using PowerManager.getThermalHeadroom()
• •Zero-copy KV-cache via VK_KHR_external_memory_fd saves ~8ms vs copy, critical for 2-3x speedup

Generated with AI, which can make mistakes.