LongCat 2.0: How Meituan Trained a 1.6T MoE Model Without NVIDIA or TPU Hardware

Short summary

Meituan open-sourced LongCat 2.0, a 1.6T-parameter MoE model trained on 50,000+ chips without NVIDIA GPUs or TPUs, using n-gram embeddings and sparse attention to reduce computational costs. The architecture includes speculative decoding and custom ASICs optimized for prefill vs decode phases. The achievement demonstrates feasibility of training large models independently of Western hardware stacks.

• •1.6T MoE model trained without NVIDIA/TPU dependency using distributed chips
• •Novel optimizations: n-gram embeddings, sparse attention, speculative decoding
• •Custom ASICs and training on 35T tokens proves alternative to Western hardware viability

Generated with AI, which can make mistakes.