Tokens

Language models do not read letters or words. They read tokens: numbered pieces of text from a fixed vocabulary. Every limit you meet when working with a model, how long a prompt can be, how much of a page fits, how long the answer may run, how much an API call costs, is counted in them.

How does text become tokens?

A tokenizer splits text using a vocabulary learned from a large body of text. Common words, usually with their leading space, get one token each. Less common words are split into pieces the vocabulary does have. Numbers are split into short groups of digits. Punctuation gets its own tokens or is folded into its neighbours.

The model then works only with the token numbers. That is why it can be oddly bad at counting the letters in a word: it never saw the letters.

What does tokenisation look like on real text?

We ran OpenAI's open-source tokenizer library, tiktoken, on 4 October 2026, over six names, codes and dates and one sentence. o200k_base is the tokenizer of OpenAI's newer models; cl100k_base is the previous one.

Our own name, Attensira, is three fragments: Att, ens, ira. A crawler name such as OAI-SearchBot is five, and a date written 2026-10-04 is six. Ordinary English is far cheaper: the sentence "The service of an SEO agency at the price of a tool." took 13 tokens for 12 words, one per word plus the full stop.

Three things follow. A new brand name is several tokens of fragments until a model has seen it often. Dates, codes and identifiers are expensive. And English is the cheapest language to process on most tokenizers, which matters when you track answers in several markets: Anthropic notes that Claude's token count varies with the language.

How do tokens differ from words, characters and embeddings?

• Character

  What it is

  One letter, digit or symbol

  Where you meet it

  Meta description limits

• Word

  What it is

  Space-separated text

  Where you meet it

  Word counts, readability

• Token

  What it is

  One piece from the model's vocabulary

  Where you meet it

  Context windows, output limits, API prices

• Embedding

  What it is

  A vector of numbers for a whole passage

  Where you meet it

  Retrieval, vector search

Anthropic documents that a Claude token is roughly 3.5 English characters, and that it varies by language. Treat any characters-per-token figure as an average, not a conversion.

Why do tokens matter outside an API?

Because AI engines spend a token budget on your page. When an engine fetches a page to answer a question, the text that reaches the model is limited: by the context window, by how many pages it read, and by how long a passage it keeps from each. A page that opens with thousands of tokens of navigation, cookie text and preamble spends that budget before it reaches the answer.

What helps:

• Put the answer first. The first lines under each heading should answer that heading's question.
• Server-render the text. A crawler that does not run JavaScript sees only what the HTML contains.
• Keep sections self-contained. A passage that is cut out alone should still make sense.

What do people get wrong about tokens?

• Counting words and calling them tokens. The ratio changes with the language, the names and the numbers in the text.
• Using one model's count for another. Each family has its own tokenizer.
• Thinking a big context window means the whole page is read. Engines can keep only part of what they fetch, whatever the window size.
• Ignoring markup. Raw HTML costs far more tokens than the visible text it carries, which is one reason engines extract text first.

How can you check a page?

The free page token inspector estimates a page's tokens section by section, the way a retriever sees it once navigation and scripts are gone, and shows where a cut at the first N tokens would land. It estimates rather than running a specific tokenizer, because every model counts differently. In a workspace, Attensira reads which of your passages engines actually cite, and the crew moves answers that sit too deep in a page up to where they are read.