I Tested Gemma 4 and GPT-4o-mini on Indian Language Tasks — The Results Surprised Me

Short summary

Gemma 4 substantially outperformed GPT-4o-mini on Hindi professional writing (correct Devanagari script, cultural context) and linguistic nuance, while both handled casual Hinglish code-switching naturally. GPT-4o-mini's failure to select appropriate script (Roman vs. Devanagari) for formal contexts reveals critical gaps. The test exposes how most AI benchmarks miss regional language quality despite serving 1.4B Indian users across 22 languages.

• •Gemma 4 wrote professional Hindi emails in Devanagari script with proper honorifics; GPT-4o-mini defaulted to Roman transliteration, failing corporate context
• •Both models handled casual Hinglish code-switching naturally—neither sounded stilted or formal
• •Test reveals systemic gaps in regional language support: most AI benchmarks prioritize English, missing critical use cases for India's 1.4B users

Generated with AI, which can make mistakes.