Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utility

Short summary

This paper introduces two metrics for evaluating LLM agent tool use: tool efficiency (rate of useful tool calls in a trajectory) and marginal tool utility (per-call metric indicating whether a tool is useful or safely removable). The authors use LLM-as-a-Judge to determine marginal tool utility signs. The work aims to shift evaluation from accuracy-only proxies toward direct efficiency measurement, providing a foundation for leaner agent tool suites and future benchmark designs.

• •Introduces tool efficiency metric measuring rate of useful tool calls in LLM agent trajectories
• •Marginal tool utility identifies which tool calls can be safely removed without accuracy loss
• •Uses LLM-as-a-Judge to classify tool utility, enabling leaner agent tool suites

Generated with AI, which can make mistakes.