Beyond the Prompt: Mastering On-Device GenAI Performance and Thermal Management on Android

Short summary

On-device Generative AI on Android delivers fast, private inference directly on phones, but thermal constraints create fundamental engineering challenges. Google's AICore system service handles memory deduplication and hardware abstraction, while developers must monitor Time To First Token (TTFT), Tokens Per Second (TPS), and memory pressure. Using Kotlin coroutines, you can build thermal-aware systems that gracefully degrade under load rather than freezing or crashing.

• •AICore deduplicates LLM weights across apps and abstracts NPU/GPU complexity via hardware abstraction layer
• •Critical metrics: TTFT (first token latency), TPS (steady-state generation speed), RSS (memory footprint)
• •Implement thermal-aware orchestration with Kotlin Flow to respond to dynamic frequency scaling

Generated with AI, which can make mistakes.