Tokens
The units a language model actually reads and writes, why they are not words, and why every limit and price you meet is denominated in them.
Karl-Gustav Kallasmaa, Founder & CEOLast updated Tokens are the units a language model actually operates on. Text is encoded into a sequence of tokens before the model sees it, and the model emits tokens which are decoded back into text. Anthropic's glossary describes them as the smallest individual units of a language model, which can correspond to words, subwords, characters or even bytes, and notes that for Claude a token approximately represents 3.5 English characters, though the exact number varies with the language used.
That approximation is the only conversion worth carrying, and it should always be treated as approximate. Tokenisers differ between vendors and between model generations, so a token count is a property of a specific model and not of the text.
Why tokenisation is not word-splitting
A fixed vocabulary of whole words would fail immediately on names, typos, URLs, source code and every word coined after the vocabulary was frozen. Subword tokenisation solves that by letting a common word be one token while a rare or novel one is assembled from several pieces. As the Anthropic documentation puts it, larger tokens enable data efficiency during inference and pretraining and are used when possible, while smaller tokens allow a model to handle uncommon or never-before-seen words.
Two practical effects follow.
Cost and capacity are unevenly distributed across languages. Tokenisers are fit to the distribution they were trained on, so text in less represented scripts fragments into more tokens for the same meaning. The same paragraph can be materially more expensive, and occupy materially more of a context window, in one language than another.
Structure is not free. Markup, deeply nested formatting and verbose boilerplate are all tokens. A page that surrounds its answer with a large amount of chrome spends budget without adding information.
How tokens differ from characters, words and embeddings
Three units get muddled, and they are not interchangeable:
- A character is a unit of the text. Fixed, model-independent, and not what anything is billed or limited in.
- A token is a unit of the model's input and output. It is what a context window is measured in and what pricing is denominated in.
- An [embedding](/glossary/embeddings) is a vector of floating point numbers representing meaning. It has a dimension count, not a token count, and the two numbers describe entirely different things.
The most common mix-up is between a token count and an embedding dimension count. A 3,072-dimension embedding is not three thousand tokens of anything; it is one vector describing one piece of text.
Why any of this matters outside an API
Every operational limit you meet in AI systems is denominated in tokens. The context window is a token budget, and Anthropic's documentation states that everything in a request counts toward it: the system prompt, every message including tool results, images and documents, the tool definitions, and the output generated for the turn.
That budget is exactly why retrieval systems are selective. A retrieval-augmented answer has to choose which passages to load, because it cannot load everything. A page whose relevant claim is stated compactly, in a self-contained passage with its own heading, is cheaper to select than one where the same claim is distributed across two thousand words of build-up. This is a mechanical argument for tight, well-sectioned writing, not a stylistic one.
How to act on it
- Measure, do not estimate, when a limit is close. Vendors publish token-counting tools for exactly this.
- Assume a different rate per language rather than converting once and reusing the figure everywhere.
- Make each section stand alone. A passage that carries its own subject and its own qualifier survives being extracted without the surrounding article.
- Put the answer near the heading that promises it, so a retriever selecting on the heading gets the substance in the same span.
- Do not pad. Length that adds no facts adds tokens, and tokens are the scarce resource on the other side of the request.
Frequently asked questions
How much text is one token?
For Claude, approximately 3.5 English characters, varying by language.
Why are tokens not words?
Because subword units cover names, typos, code and unseen words that a word vocabulary cannot.
Do other languages cost more?
Usually yes for the same meaning, because the text fragments into more tokens.
What counts toward a request's total?
Everything: system prompt, all messages, tool definitions and the generated output.
Terms related to Tokens
The token budget a model can reference in one request, what counts against it, and why a bigger window does not remove the need to retrieve selectively.
Vectors of floating point numbers whose distance measures relatedness, the representation underneath semantic search and retrieval.
The architecture that retrieves documents at query time and has a model write from them, and the reason your page can be quoted without ever being trained on.
Search where a model composes the answer and fetches pages through named crawlers, rather than returning a ranked list of links for you to read.