AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

Short summary

AgentLens is an open-source benchmark that evaluates coding agents by reviewing their full execution trajectories rather than just pass/fail outcomes. It combines formal verification with LLM-written trajectory reviews and side-by-side comparisons to produce readable explanations of each score. The authors use it to diagnose model behavior, compare agent versions, and catch product regressions in nightly evaluation pipelines.

• •AgentLens evaluates full agent trajectories, not just task pass/fail
• •Combines formal verification with LLM-written trajectory reviews for readable score explanations
• •Released as open source; used for diagnosing behavior, comparing versions, and catching regressions

Generated with AI, which can make mistakes.