Benchmarking Mitii AI Agent: 78% Success Rate on 500+ Tasks Using a Local Qwen3-Coder (30B)

Short summary

The author benchmarked Mitii, an open-source AI coding agent, on 515 tasks using a locally-run Qwen3-Coder 30B model via Ollama, achieving a 78% success rate. Mitii's three modes (Agent, Plan, Ask) handle different complexity levels, with Ask Mode showing 87% win rate on hard tasks due to its impact-analysis confirmation step. Security tasks passed at 87% and corner cases at 84%, though semantic retrieval remains weak at 63%.

• •Mitii AI agent achieved 78% (400/515) success rate on adversarial coding tasks with local Qwen3-Coder 30B via Ollama
• •Ask Mode's impact-analysis confirmation step yielded 87% win rate on hard tasks including prompt-injection attacks
• •Security tasks passed at 87%, corner cases at 84%, but semantic retrieval lagged at 63%

Generated with AI, which can make mistakes.