PoPE: Placebo-Controlled Evaluation Challenges Error-Conditioned Self-Repair in Small Code LLMs

Short summary

Researchers introduced PoPE, a placebo-controlled evaluation methodology for measuring error-conditioned self-repair in frozen small code LLMs (0.5–1.5B parameters). In both prompt and weight channels, live error content did not outperform placebo controls, suggesting that raw error feedback may not drive repair as expected. The findings call for more rigorous benchmarking standards and rethinking how error signals are fed back to small models.

• •PoPE uses channel-specific placebos to isolate whether error content or feedback structure drives self-repair in small code LLMs
• •Results were mechanism-null: placebos matched or outperformed live error content in both prompt and adapter channels
• •Developers should reconsider simply feeding raw error messages back to small frozen models for repair tasks

Generated with AI, which can make mistakes.