Optimization Is Not All You Need

Short summary

This paper critiques the optimization culture underlying modern LLM development, arguing that alignment is not merely engineering but reflects a deeper conviction that measurable improvement along predefined axes exhausts value. The authors trace this through pretraining, decoding, preference tuning, and benchmarking, concluding that optimization cannot distinguish error from invention. They warn that loss functions and reward models now hold authority over legitimate language without genuine capacity for judgment.

• •Critiques optimization culture in LLM development as the latest expression of audit-society thinking
• •Argues loss functions and benchmarks cannot distinguish error from invention in generated text
• •Frames alignment as a transfer of linguistic authority from human institutions to automated procedures

Generated with AI, which can make mistakes.