Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads

Short summary

This paper introduces a latency-aware LLM query router that jointly optimizes latency, accuracy, and cost when assigning queries to model instances. It uses a lightweight latency estimator simulating autoregressive token batch processing to predict time-to-first-token. Experiments show up to 40% improvement in accuracy-cost utility while maintaining the same latencies as standard load-balancing approaches like round-robin or join-the-shortest-queue.

• •Latency-aware router jointly optimizes latency, accuracy, and cost for LLM queries
• •Lightweight estimator simulates token batch processing to predict time-to-first-token
• •40% improvement in accuracy-cost utility vs standard load-balancing at same latency

Generated with AI, which can make mistakes.