My Auto-Publish Pipeline Shipped a Two-Year-Old News Story. Here's the Fix — All Three Layers of It.

Short summary

An automated content pipeline using an LLM judge published a two-year-old news story as current. The postmortem reveals two guards both failed as designed: a recency check scoped only to product items (not news), and an LLM judge never asked about currency. The fix is three layers: widen deterministic checks to all content types, add recency to the judge's rubric, and extract event dates from content to compare against feed timestamps. Key lesson: LLM judges have no instincts — every implicit human check must be written explicitly.

• •LLM judge scored a 2024 news story as current because currency was never in the rubric
• •Three-layer fix: widen deterministic checks, add recency to judge rubric, extract event dates from content
• •LLM judges lack human instincts — every implicit editorial check must be explicitly encoded

Generated with AI, which can make mistakes.