The Watermark Is the Real Boundary Between Training and Serving

Short summary

Streaming AI pipelines fail silently when watermark semantics in serving diverge from batch assumptions in training. Features like 'transactions in the last 5 minutes' mean different things when computed against settled data vs. a Flink watermark-bounded stream, creating distribution skew no dashboard catches. The fix: treat watermark lag and replay parity as first-class evaluation metrics owned by model quality teams, not just infrastructure tuning.

• •Watermark policy creates a hidden feature definition gap between training and serving
• •Infra health (no backpressure, no lag) does not equal feature correctness
• •Watermark configuration is a statement about acceptable feature error, not just throughput tuning

Generated with AI, which can make mistakes.