Building a Local Voice AI Assistant on 8GB VRAM with Sub-3s Latency

Short summary

A developer built Mana, a fully local voice AI assistant running on an 8GB VRAM GPU that achieves 2-3 second response latency. The key architectural decision was using a single 4B model (Qwen) for reasoning, code generation, and explanation instead of chaining multiple specialized models that would require costly VRAM swaps. Whisper handles transcription and Kokoro/Chatterbox handle TTS, with a Live2D avatar for visual presence.

• •Single 4B model replaces multi-model pipeline to avoid VRAM swap overhead
• •2-3s total latency from voice input to audio output on 8GB VRAM
• •XML-structured output separates silent code execution from spoken explanation

Generated with AI, which can make mistakes.