Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy

Short summary

This study evaluates LLM reliability by testing whether 13 models give consistent answers to meaning-preserving paraphrases across four benchmarks. Instance-level mismatch rates exceed 23%, meaning models often flip between correct and incorrect answers depending on phrasing despite stable aggregate accuracy. A simple self-paraphrasing strategy can partially recover latent knowledge and improve performance at inference time, suggesting standard accuracy metrics mask substantial instability.

• •13 LLMs tested across 4 benchmarks show 23%+ instance-level answer flips under paraphrasing
• •Aggregate accuracy stays stable but per-question reliability is far worse than benchmarks suggest
• •Self-paraphrasing at inference time partially recovers latent knowledge and improves performance

Generated with AI, which can make mistakes.