The original title is: "Running Gemma-4 26B MoE on a single AWS inf2.xlarge via int8 quantization"

Short summary

Detailed engineering log of squeezing Gemma-4's 26B-parameter 128-expert MoE onto a single AWS inf2.xlarge (2 NeuronCores, 32 GB HBM, ~$0.76/hr) — an 8.6× cost reduction. After hitting walls with fp4 quantization (broken SDK paths, CPU-only dequant, mispathed NKI kernels), the author found the real win was int8-quantizing the tied lm_head and shared dense MLP, not the experts. Final result: ~12–13 GB per-rank resident, compiler PASS, 232 ms prefill.

• •Goal: run 26B MoE on inf2.xlarge at $0.76/hr (8.6× cheaper than inf2.24xlarge)
• •fp4 quantization path is broken in current Neuron SDK — multiple dead ends
• •Key fix: int8-quantize lm_head (1.48→0.37 GB/rank) and shared dense MLP, not just experts

Generated with AI, which can make mistakes.