How I Built a Local AI Orchestrator and City AI: My Journey as a Developer

Short summary

Raj Patil showcases two production projects: Local AI Orchestrator, a privacy-first system running Llama 3 and Mistral locally via LangChain and Ollama achieving sub-100ms inference on consumer GPUs, and City AI, which automates municipal complaint routing using semantic analysis with Google Gemini. Both demonstrate full-stack patterns for building intelligent systems without expensive cloud infrastructure. The work emphasizes that proper quantization and orchestration enable enterprise-grade AI performance on local hardware.

• •Local AI Orchestrator: offline LLM system using LangChain, Ollama, NVIDIA CUDA achieving <100ms latency on RTX 3050
• •City AI: automated municipal feedback routing with semantic categorization via Google Gemini API and Express backend
• •Key insight: Robust, responsive AI applications can run on consumer hardware with proper quantization—no expensive cloud required

Generated with AI, which can make mistakes.