Why LLMs Still Struggle With Tabular Prediction

Short summary

A new paper shows that LLMs fail at tabular prediction primarily due to input dimensionality, not formatting, tokenization, or context length. Across 31 datasets and 252 classical model configurations, LLM accuracy declines as feature count grows while gradient-boosted trees remain stable. At low dimensions LLMs behave like k-NN, but this resemblance vanishes at higher dimensions where no classical model matches their behavior.

• •Dimensionality is the central failure mode for LLMs on tabular data, not serialization or tokenization
• •LLM accuracy drops as feature count grows; classical baselines stay stable or improve
• •At low dimensions LLMs resemble local methods (k-NN, short-scale GPs); at high dimensions this breaks down

Generated with AI, which can make mistakes.