Belief or Circuitry? Causal Evidence for In-Context Graph Learning

Short summary

Researchers used graph-based tasks and causal interventions to show LLMs employ dual mechanisms for in-context learning: both local pattern-matching and global structure inference operate simultaneously. Internal representation analysis via PCA and activation patching revealed orthogonal encoding of competing graph topologies, incompatible with pure local copying. Findings challenge single-mechanism accounts and suggest LLMs develop parallel inference circuits.

• •Dual mechanisms for in-context learning: pattern-matching and structure inference both active simultaneously
• •PCA analysis shows competing topologies encoded orthogonally; activation patching confirms causal role
• •Challenges simplified models of LLM learning; suggests parallel circuit operation

Generated with AI, which can make mistakes.