Quantizing MedGemma to INT4 (GPTQ/W4A16): Everything That Broke Along the Way

Short summary

A detailed technical walkthrough of quantizing Google's MedGemma-1.5-4B (a medical vision-language model) to INT4 using GPTQ via llm-compressor. Key challenges include AutoAWQ deprecation and lack of Gemma3 support, vision dataset label masking not being implemented, and the need to exclude the vision tower from quantization. Includes exact package versions, calibration dataset setup, and inference verification steps.

• •AutoAWQ deprecated and lacks Gemma3 support; GPTQ via llm-compressor is the path forward
• •Vision dataset label masking unimplemented — use text-only calibration data and exclude vision tower from quantization
• •Exact environment: torch 2.11.0+cu128, llmcompressor 0.12.0, transformers 5.10.1 on RunPod RTX A5000

Generated with AI, which can make mistakes.