Day 3: Watch your grammar with AI, it may cost you — Understanding BPE Tokenizers 🍓🔡

Short summary

BPE tokenizers split text into sub-word units before LLMs process them, which is why models struggle with character-level reasoning (counting r's in 'strawberry'). Capitalization and spacing affect token count by 3.5x or more, directly impacting API costs. An interactive simulator explores tokenizer mechanics and demonstrates how adversarial inputs can bypass token-level safety filters.

• •BPE tokenizers split text into sub-word units, causing LLMs to struggle with character-level reasoning
• •Prompt formatting (capitalization, spacing) can increase token costs by 3.5x or more
• •Token-level safety filters can be bypassed with adversarial inputs like capitalization and Unicode tricks

Generated with AI, which can make mistakes.