Exploring More to Solve More: Boosting Diversity in Text Diffusion Models via Entropy-Based Guidance

Short summary

SAKE is a training-free guidance method for text diffusion models that computes Rényi entropy over a semantic kernel matrix to dynamically adjust sampling distributions. It flattens distributions during redundant generation and sharpens them when diversity is needed, achieving a superior Pareto frontier between fidelity and diversity. The method improves multi-sample performance on reasoning-intensive tasks like code and math generation compared to temperature scaling baselines.

• •Training-free Semantic-Aware Kernel Entropy (SAKE) guidance for text diffusion models
• •Dynamically adjusts sampling to balance fidelity and diversity using Rényi entropy
• •Improves multi-sample performance on code and math generation tasks
• •Outperforms temperature scaling and discrete guidance baselines

Generated with AI, which can make mistakes.