🏷️ tokenizer

1 articles about 'tokenizer'

The 'Chinese Tax' on AI Tokens: Why Non-English Costs More

2026-05-03 llm 👁 11

Chinese text consumes up to 2x more tokens than English in most LLMs. Here's why tokenizers create an invisible language…