Colibri: Running a 744B AI Model on Your Laptop

Short summary

Colibri is a pure C inference engine that runs GLM-5.2 (744B parameter MoE model) on a laptop with just 25 GB RAM by streaming experts from NVMe SSD instead of loading the full model. It keeps dense layers (~10 GB) resident and fetches only the ~11 GB of routed experts needed per token. Performance is 0.3-2 tok/s — 10-100x slower than cloud H100 inference but completely free and offline.

• •Colibri runs 744B GLM-5.2 on a laptop with 25GB RAM by streaming MoE experts from disk
• •Dense layers stay in RAM (~10GB), routed experts (~11GB/token) fetched from NVMe on demand
• •0.3-2 tok/s performance — 10-100x slower than cloud but free, offline, and privacy-preserving

Generated with AI, which can make mistakes.