How to Review an Edge LLM Benchmark Without Fooling Yourself

Short summary

A practical guide to reviewing edge LLM benchmarks without overgeneralizing results. The author provides a structured test envelope (device, power, cooling, model, quantization) and a set of critical test cases including cold start, repeated requests, long context, and offline behavior. The key insight: report median, p95, worst-case metrics and define an acceptance envelope tied to real product requirements rather than chasing peak tokens-per-second.

• •Define a full test envelope: device, power mode, cooling, OS, runtime, model tag, quantization, context length
• •Run cold-start, repeated-request, long-context, offline, and cancellation cases — not just a single throughput number
• •Report p95 and worst-case time-to-first-token, peak memory, thermal throttling, and task success rate

Generated with AI, which can make mistakes.