I'm trying to implement CALM paper, and I have some questions. [P]

Short summary

An ML engineer details their struggle implementing Kyutai's Pocket TTS from paper, facing poor inference quality despite low training losses on smaller datasets. Key challenges include gradient explosion, exposure bias, and a fundamental trade-off: placing voice tokens near target latents produces better voice cloning but worse text generation. Seeking guidance on data scaling, model architecture, and resource commitment before larger training runs.

• •Implementing Pocket TTS on LJSpeech/LibriSpeech yields poor inference quality despite low training losses
• •Gradient explosion and conflicting trade-offs between text generation and voice cloning fidelity are primary blockers
• •Seeking recommendations on data scaling strategy, architecture fixes, and resource commitment before scaling to larger clusters

Generated with AI, which can make mistakes.