Put the LLM last: I replaced a 7B model with a tiny Go classifier

Short summary

A developer replaced a 7B-parameter LLM with a tiny TF-IDF + logistic regression classifier in Go for email triage, using a three-layer routing strategy: deterministic rules first, small ML model next, LLM only as last resort. The classifier weighs 2.4 MB, runs sub-millisecond on CPU, and achieves 81% accuracy in 5-fold cross-validation. The approach eliminates GPU costs while reserving expensive LLM calls for genuinely uncertain cases.

• •Three-layer routing: deterministic rules → small ML model → LLM as last resort
• •TF-IDF + logistic regression classifier in Go: 2.4 MB, sub-ms inference, 81% accuracy
• •Eliminates GPU dependency and reduces cloud LLM calls to only uncertain cases

Generated with AI, which can make mistakes.