I measured every millisecond of my real-time AI pipeline. The LLM was the fast part.

Short summary

A real-time meeting assistant builder discovered that LLM token latency (1.2s p50) was not the primary bottleneck — speech-to-text tail latency and gating logic dominated the pipeline. Production metrics over 7 days showed gpt-4o-mini and Claude Haiku 4.5 perform similarly, while smarter models like gpt-5-mini are disqualified by 3.7s+ TTFT. The key insight: endpointing configuration, not model speed, is the real latency lever for real-time AI.

• •LLM was the fast part at 1.2s p50 TTFT; STT tail latency and gating were bigger bottlenecks
• •Smarter models (gpt-5-mini at 3.7s TTFT) lose to fast models when output has an expiry date
• •Endpointing configuration is the real latency knob — a tradeoff, not a bug

Generated with AI, which can make mistakes.