Back to feed
arXiv cs.LG
arXiv cs.LG
7/17/2026
Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning

Branching Policy Optimization: Sandbox-Native Language Agent Reinforcement Learning

Short summary

BPO is a new RL algorithm that exploits the deterministic, snapshottable nature of agent sandboxes by building a single branching tree instead of N independent rollouts. It snapshots at high-entropy decision points, forks alternative actions, and computes advantages from sibling returns. On WebShop, ALFWorld, and SWE-bench, BPO improves success by 3.6–6.1 points over GRPO/RLOO at matched compute and matches the best baseline with 38% fewer updates.

  • •BPO replaces independent-rollout RL with a single branching tree leveraging sandbox snapshot/resume capabilities
  • •Proven unbiased with strictly lower variance than trajectory-level baselines
  • •3.6–6.1 point success gains over GRPO/RLOO on WebShop, ALFWorld, SWE-bench at matched compute

Generated with AI, which can make mistakes.

#research-breakthrough#ai-agents#open-source
Read full article at arXiv cs.LG

Is this a good recommendation for you?

Comments

Failed to load comments. Please try again.

Explore more

It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches
arXiv CS.AI
Process Reward Informed Tree Rollout for Effective Multi-Turn RL
arXiv cs.CL
Recursive Self-Evolving Agents via Held-Out Selection
arXiv CS.AI
The original title is "Predictable GRPO: A Closed-Form Model of Training Dynamics"
arXiv cs.LG