Multi-level context Modeling for consistent expert selection in Mixture-of-Experts

Short summary

MCF-MOE is a framework that improves expert routing in Mixture-of-Experts models by fusing cross-layer semantic aggregation with local token-level interactions, addressing context incompleteness in traditional routers. Experiments on language modeling and understanding benchmarks show consistent gains over strong MoE baselines. Code is publicly available.

• •Proposes Multi-level Context Fusion MOE (MCF-MOE) to address context incompleteness in expert routing
• •Integrates cross-layer semantic aggregation and local token-level interactions for more consistent expert selection
• •Demonstrates consistent improvements over MoE baselines on language modeling and understanding benchmarks

Generated with AI, which can make mistakes.