The original title is "LLMs on Consumer Hardware — Part 1: The Stack and First Benchmarks"

Short summary

A build-log documenting a local LLM project on consumer hardware, comparing throughput and quality across four models using Ollama with ROCm and CUDA. Gemma 4 26B (Q4_K_M) was selected as the primary model after scoring 99/100 on a ten-task quality suite while sustaining ~17 tokens/sec on an AMD RX 6900XT. The key finding: the fastest model is rarely the right default for substantive work.

• •Gemma 4 26B quantized to Q4_K_M achieves 99/100 quality score at ~17 tokens/sec on consumer AMD hardware
• •Smaller 7B models are 2-3x faster but fail logic and instruction-following tasks
• •Flash attention and 8-bit KV cache recover enough VRAM headroom for GPU/CPU layer splitting

Generated with AI, which can make mistakes.