LAB · token-labRuns in your browser

Token Lab

Type or paste text and see how a model splits it into tokens. Chunk sizes, model limits and prices are all counted in these pieces.

Used on RAG-03 and RAG-15

Characters: 103
Words: 16
Tokens: 19
Tokens per word: 1.19

The router restarts itself when the firmware update finishes. Wait two minutes before you call support.

Tokenizer: o200k_base via gpt-tokenizer. Hover a token to see its id. Error codes, numbers, code and non-English text use more tokens per word than plain English prose.

What to notice

Common English words are one token each. Error codes and numbers break into several.

Surprise

The English sample uses about 1.2 tokens per word, the Dutch one 1.3 and the error code line 2.0. The same chunk size holds less text.

Try next

Paste a table row from a manual and compare tokens per word with plain prose.