The Chatbot Was Easy. The Engineering Wasn't.

Short summary

A Node.js developer recounts building a production banking AI chatbot, discovering that the real engineering challenges emerge after the basic WebSocket-to-LLM pipeline works. Intent detection became necessary when the bot confidently answered off-domain questions, but adding it introduced 200-300ms latency on every request. Caching strategies evolved from naive exact-match to intent-based keying, each revealing new edge cases around semantic equivalence versus intent overlap.

• •Simple chatbot architecture breaks when users ask off-domain questions, requiring an intent detection gatekeeper
• •Intent detection adds 200-300ms latency per request, prompting caching exploration
• •Exact-match and intent-based caching both fail on semantic variation and intent overloading

Generated with AI, which can make mistakes.