A Three-Phase Factual Recall Circuit in Gemma-2B and Gemma-12B-IT

Short summary

Researchers used activation patching to map how Gemma-2B and Gemma-12B store and retrieve facts across transformer layers. The analysis reveals a three-phase circuit spanning layers, with the residual stream handling most information flow. Findings improve understanding of mechanistic interpretability in smaller language models.

• •Activation patching traces fact recall through three distinct phases across Gemma model layers
• •Residual stream is the primary channel for information flow during factual recall
• •Research advances mechanistic interpretability for smaller language models (2B–12B parameter scale)

Generated with AI, which can make mistakes.