Switching to GPT-5.6 in production: it's a schema, cache, and harness problem — not a model swap

Short summary

Ploy's experience switching from Claude Opus 4.8 to GPT-5.6 Sol in production reveals that model swaps are really infrastructure problems. A third of initial eval failures came from harness assumptions, not model behavior. Key fixes included a nullable schema transform for tool parameters (dropping empty reads from 52% to 0%), per-workspace cache keys (raising hit rates from ~0% to 83.7%), and disabling server-side reasoning replay to fix intermittent context errors.

• •~33% of initial eval failures were harness assumptions, not model behavior
• •Nullable schema transform at provider boundary fixed 52% empty tool reads to 0%
• •Cache key restructuring raised hit rates from ~0% to 83.7%, flipping cost advantage

Generated with AI, which can make mistakes.