KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling

Short summary

KV-PRM is an efficient process reward model that reads the KV cache directly during LLM generation instead of re-encoding trajectory text, reducing scoring cost from O(L²) to O(L). Across MATH, GSM8K, and AIME benchmarks, it matches or outperforms text-based PRMs while achieving up to 5,000x reduction in FLOPs, 37x lower latency, and 34x lower memory footprint. This makes PRM-guided test-time scaling practical for long multi-agent rollouts.

• •KV-PRM eliminates text re-encoding by reading KV cache directly, cutting scoring cost from O(L²) to O(L)
• •Matches or outperforms text-PRMs on MATH, GSM8K, AIME with up to 5,000x fewer FLOPs
• •Enables practical PRM-guided test-time scaling for long multi-agent trajectories

Generated with AI, which can make mistakes.