How Does an LLM Request and Response Cycle Work? A Full Walkthrough

Short summary

A beginner-friendly walkthrough tracing a single LLM prompt from client app through API gateway, tokenization, transformer forward pass, sampling, and streaming response. The author built iamspeed.dev, a browser-based LLM benchmarking tool, and wrote this to explain the steps between pressing Enter and seeing words appear. Covers JSON request structure, rate limiting, BPE tokenization, and why providers bill per token.

• •LLM requests pass through client app, API gateway, tokenizer, transformer layers, and streaming pipeline
• •Tokenization converts text to numeric IDs using byte pair encoding, determining billing units
• •System prompts and conversation history are bundled into every request since models are stateless

Generated with AI, which can make mistakes.