Colibri: Running MoE Models on CPU+SSD and a Proposed Speculative Loading Architecture

Short summary

Colibri is a project that runs massive Mixture-of-Experts models on ordinary CPUs and SSDs by loading only the active experts into memory at each step, discarding them after use. The author identifies the real bottleneck as SSD-to-RAM data transfer latency and proposes a speculative loading architecture that predicts which experts will be needed next, preloading them in parallel. This would transform storage from a reactive to a proactive component, nearly eliminating idle wait time during inference.

• •Colibri runs MoE models on CPU+SSD by loading only needed experts into memory
• •Bottleneck is SSD→RAM transfer latency, not compute power
• •Author proposes speculative expert preloading with predictive scheduling and async I/O

Generated with AI, which can make mistakes.