Index SLM Technical Report

Short summary

Bilibili releases Index-1.9B, a family of four open small language models (1.9B parameters) trained on 2.8T Chinese-English tokens. The base model scores 64.92 average on standard benchmarks, outperforming larger open models. The release includes controlled studies on depth, learning-rate scheduling, data quality, and instruction data during pre-training, plus an unexplained benchmark surge mid-training.

• •Four open models released: base, pure (no instruction data), chat (SFT+DPO), and character (RAG role-playing)
• •1.9B parameter model achieves 64.92 avg on standard benchmarks, competitive with much larger models
• •Controlled studies on learning-rate decay, data quality, and instruction data effects during pre-training

Generated with AI, which can make mistakes.