topics — Tokenization: how LLMs break text into pieces their neural networks can process

TOPIC AI

Tokenization how LLMs break text into pieces their neural networks can process

IN 10 SECONDS

Tokenization is the first step of any LLM pipeline. Text is split into tokens — subwords, not whole words. 'unbelievable' becomes `['un', 'believe', 'able']`. Each token maps to an integer ID in the model's vocabulary (typically 32K-200K tokens). Tokenization determines how the model 'sees' your text and directly affects cost, latency, and even comprehension.

GOTCHA Tokenization is language-dependent and can be unpredictable. Some languages (Chinese, Japanese) have very high token-to-character ratios, increasing cost. A single character change can completely change the tokenization. Always check your prompt's token count with a tokenizer before sending it to the API — use OpenAI's `tiktoken` or Anthropic's tokenizer.

HOW TOKENIZATION BREAKS DOWN TEXT

01 Raw text "I'm running Kubernetes on AWS" enters the tokenizer.

02 BPE tokenizer Byte-Pair Encoding scans the text for common subword patterns: `['I', "'", 'm', ' run', 'ning', ' Ku', 'ber', 'net', 'es', ' on', ' AWS']`.

03 Token IDs each token is mapped to an integer: `[40, 466, 12, 1285, 596, 17283, 29768, 4793, 659, 1649, 21371]`.

04 Model input the token IDs are passed through the embedding layer — the model never sees raw text, only these integer sequences.

POKE IT YOURSELF

python -c "import tiktoken; enc = tiktoken.get_encoding('cl100k_base'); print(len(enc.encode('Hello world')))" — count tokens in a string with tiktoken

node -e "const {getEncoding} = require('js-tiktoken'); const enc = getEncoding('cl100k_base'); console.log(enc.encode('Hello world').length)" — count tokens with js-tiktoken

Drill this topic → ~75 sec read