Dynamic Parameterization Is Not Dynamic Inference

Short summary

This paper introduces Frozen-Controller Auditing (FCA) to disentangle dynamic parameterization from dynamic inference in transformer models. Across FeatureGate and MUDDPythia checkpoints, static layerwise profiles retain ~99% of performance, layer identity explains 87-96% of coefficient variance, and FeatureGate is actually 30.8% slower than dense baselines. The authors argue that claims about dynamic models should separately report coefficient variation, functional dependence, and actual execution cost.

• •FCA caches coefficient tensors and replays with cross-input reassignment to audit dynamic parameterization claims
• •Static profiles retain 98.7-99.4% of performance gap, showing coefficients are largely layer-identity-dependent not content-dependent
• •Dynamic parameterization alone does not establish dynamic inference or computational savings

Generated with AI, which can make mistakes.