How I built a 3-provider LLM fallback system in production (and what actually broke)

Short summary

A student-built SaaS implements a 3-provider LLM fallback chain (Anthropic Claude → Google Gemini → Groq) to handle rate limits and cost across a 5-agent pipeline. Production issues included Groq's 6K TPM ceiling causing 429s, unreliable JSON parsing with smaller models requiring two separate calls, and invisible newlines in Railway config that broke authentication. The author shares exact routing logic and production patterns, including OpenAI-compatible endpoints and defensive config stripping.

• •3-provider fallback routing solves rate limits and cost for multi-agent SaaS deployments
• •Smaller models need separate streaming + JSON calls to avoid parsing failures across providers
• •Defensive config management: strip() API keys to catch invisible whitespace from clipboard pastes

Generated with AI, which can make mistakes.