MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference

Short summary

MawForge tests whether local MoE inference can work on constrained unified-memory machines by storing the full model on disk, keeping common tensors resident, and materializing routed expert tensors into a bounded execution cache on demand. It proves effective as a bounded execution mechanism but not as a cache-maximization policy. Performance depends on balancing expert reuse, resident footprint, KV-cache size, quantization, route locality, and macOS memory pressure.

• •MawForge enables local MoE inference by storing full model on disk and materializing experts on demand into bounded cache
• •Effective as execution mechanism but not as cache-maximization policy
• •Performance depends on expert reuse, resident footprint, KV-cache size, quantization, and route locality

Generated with AI, which can make mistakes.