LLM builds better Dota 2 solver than it plays the game

Short summary

An engineer replaced an LLM's card-by-card judgements in a Dota 2 drafting game with a deterministic Python solver, then built a rigorous benchmarking framework using common random numbers to compare policies fairly. The approach freezes game randomness into indexed tapes, separates train/validation/test episodes, and gates policy changes behind reproducible criteria. The result is a methodology for evaluating AI agent policies without confusing improvement with luck.

• •LLM underperformed a human in a Dota 2 drafting game, prompting a deterministic solver approach
• •Common-random-numbers technique pairs policies on identical game tapes to remove luck noise
• •Train/validation/test separation with frozen parameters prevents evaluation contamination

Generated with AI, which can make mistakes.