Evaluating Communicative Belief Updates in Large Language Models via Implicature Recognition and Cancellation

Short summary

This paper evaluates whether LLMs can recognize unspoken beliefs conveyed through implicatures and track belief updates when implicatures are cancelled. The authors create the first expert-annotated implicature cancellation dataset and find that LLMs lag behind humans, especially in natural scenarios. Control experiments suggest LLM successes may rely on prior beliefs rather than genuine pragmatic understanding, and failures depend on belief-update type and form.

• •First expert-annotated implicature cancellation dataset for evaluating LLM pragmatic understanding
• •LLMs lag behind humans in recognizing unspoken beliefs and belief updates
• •LLM successes may stem from reliance on prior beliefs rather than genuine pragmatic reasoning

Generated with AI, which can make mistakes.