I Built an llms.txt Generator, Showed It to the Creator of the Standard, and Had to Rewrite Everything

Short summary

The author built a llms.txt generator (a standard for AI agents to navigate sites) using a flat one-URL-to-one-summary approach. After feedback from the standard's creator, they redesigned it to use semantic clustering and hierarchical markdown, solving five engineering problems: context windows, token costs, pipeline speed, LLM reliability, and language handling. The rewrite teaches patterns for scalable AI-native products: embeddings, context caching, independent-layer queuing, exception hierarchies, and state recovery.

• •Initial flat llms.txt design was fundamentally flawed; the standard requires curated hierarchical documents, not auto-generated summaries.
• •Rewrote v2 to cluster pages by meaning, use Gemini context caching for token efficiency, decouple pipeline layers, and handle LLM failures gracefully.
• •Real product engineering exposes problems iteratively; shares specific architectural patterns applicable to AI-native systems at scale.

Generated with AI, which can make mistakes.