Quantifying Aleatoric Uncertainty of In-Context Learning for Robust Measure of LLM Prediction Confidence

Short summary

Researchers propose a Bayesian method using self-function vectors to measure aleatoric uncertainty in LLM in-context learning, separating it from epistemic uncertainty. They introduce the first rigorous evaluation protocol for quantifying ICL uncertainty with real-world dataset validation. Applications include hallucination detection and trustworthy AI systems.

• •New method separates aleatoric from epistemic uncertainty in in-context learning
• •Proposes rigorous evaluation framework validated on synthetic and real-world tasks
• •Practical tool for hallucination detection and trustworthy AI applications

Generated with AI, which can make mistakes.