Beyond Scaling Laws: Why "Thinking Longer" Is a Systems Problem, Not a Prompting Trick

Short summary

Test-time compute shifts the optimization question from pretraining scale to per-query compute allocation, breaking standard inference infrastructure assumptions. Three strategies—extended generation, parallel sampling with voting, and tree search with pruning—have different cost-efficiency profiles and infrastructure requirements. Existing serving stacks built for independent, similar-length, single-pass requests fail when search spawns child requests sharing parent caches, creating bimodal length distributions and multi-model pipelines that demand new architectural thinking.

• •Test-time compute breaks inference stack assumptions about independent, uniform-length, single-pass requests
• •Three compute strategies (rambling, voting, tree search) have different efficiency and infrastructure costs
• •Adaptive escalation and alignment of verifiers are architectural problems, not prompting tricks

Generated with AI, which can make mistakes.