My Home AI's First Reply Took Four Minutes. Now It Takes Eleven Seconds.

Short summary

A home AI system running a 32B model on an AMD integrated GPU reduced first-reply latency from 242 seconds to 11 seconds through four non-model optimizations: disabling unnecessary reasoning tokens, pruning unused tool schemas, caching static prompt portions, and fixing a guardrail that only logged instead of stopped loops. The key insight is that latency is a system property, not a model property — configuration, tool bloat, and caching caused all the delays. Cold starts are amortized via scheduled warm-ups rather than eliminated.

• •Reduced local 32B model latency from 242s to 11s by fixing tool bloat, prompt caching, reasoning flags, and guardrails — none of the issues were the model itself
• •Unused tool schemas cost ~8,700 tokens per call and caused inference hangs; pruning them was the single biggest win
• •Cold starts are scheduled to 6 AM warm-ups since they can't be eliminated, only hidden from user-facing moments

Generated with AI, which can make mistakes.