Riemannian Geometry for Pre-trained Language Model Embeddings

Short summary

This paper probes whether sentence-level classification signal lives in the Riemannian geometry of contextual token embeddings by extracting per-token pullback metrics and aggregating them via Fréchet mean on the SPD manifold (Riemannian Mean Pooling). RMP outperforms Euclidean mean pooling on CoLA, CREAK, and RTE, but correctly stays at chance on FEVER-Symmetric. Ablations show the geometric aggregation itself drives most gains, with trained encoders adding signal only on knowledge-heavy tasks.

• •Riemannian Mean Pooling outperforms Euclidean pooling on three linguistic structure datasets
• •Method correctly stays at chance on FEVER-Symmetric, which removes lexical artifacts
• •Geometric aggregation drives gains; learned manifold structure contributes only on knowledge-heavy tasks

Generated with AI, which can make mistakes.