# Tokens

> For the complete documentation index, see [llms.txt](https://attensira.com/llms.txt).
> Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.

> A token is the unit of text a language model reads and writes. How text becomes tokens, a real tokenizer test on brand names, and why limits use them.

- Canonical page: https://attensira.com/glossary/tokens
- Family: glossary
- Last updated: 2026-10-04
- Author: Karl-Gustav Kallasmaa (https://attensira.com/about)
- Claims as data: https://attensira.com/claims.json?slug=tokens

**In short:** A token is the unit of text a language model reads and writes: often a whole word, sometimes part of one, a number or a punctuation mark. Prompt limits, context windows and API prices are all counted in tokens. English averages a little over one token per word; names, numbers and other languages take more.

**How OpenAI's o200k_base tokenizer splits names, codes and dates**

| Text | Tokens | How it was split |
| --- | --- | --- |
| Attensira | 3 | Att · ens · ira |
| attensira.com | 4 | att · ens · ira · .com |
| GPTBot | 2 | GPT · Bot |
| OAI-SearchBot | 5 | O · AI · - · Search · Bot |
| robots.txt | 2 | robots · .txt |
| 2026-10-04 | 6 | 202 · 6 · - · 10 · - · 04 |

_Source: Attensira tokenizer test, tiktoken 0.14.0, as of 2026-10-04 (6 strings, o200k_base)._

## Key facts

- What it is: One piece of text from a model's fixed vocabulary
- Rule of thumb: [About 3.5 English characters per Claude token](https://platform.claude.com/docs/en/about-claude/glossary)
- Counted in: Context windows, output limits, API prices
- Varies by: Tokenizer and language: one model's count is an estimate for another
- Related term: [Context window, the token budget per request](/glossary/context-window)
- Check a page: [Page token inspector, a free Attensira tool](/tools/page-token-inspector)

Language models do not read letters or words. They read tokens: numbered pieces of text from a fixed vocabulary. Every limit you meet when working with a model, how long a prompt can be, how much of a page fits, how long the answer may run, how much an API call costs, is counted in them.

## How does text become tokens?

A tokenizer splits text using a vocabulary learned from a large body of text. Common words, usually with their leading space, get one token each. Less common words are split into pieces the vocabulary does have. Numbers are split into short groups of digits. Punctuation gets its own tokens or is folded into its neighbours.

The model then works only with the token numbers. That is why it can be oddly bad at counting the letters in a word: it never saw the letters.

## What does tokenisation look like on real text?

We ran OpenAI's open-source tokenizer library, tiktoken, on 4 October 2026, over six names, codes and dates and one sentence. `o200k_base` is the tokenizer of OpenAI's newer models; `cl100k_base` is the previous one.

Our own name, Attensira, is three fragments: Att, ens, ira. A crawler name such as OAI-SearchBot is five, and a date written 2026-10-04 is six. Ordinary English is far cheaper: the sentence "The service of an SEO agency at the price of a tool." took 13 tokens for 12 words, one per word plus the full stop.

Three things follow. A new brand name is several tokens of fragments until a model has seen it often. Dates, codes and identifiers are expensive. And English is the cheapest language to process on most tokenizers, which matters when you track answers in several markets: Anthropic notes that Claude's token count varies with the language.

## How many tokens does plain English take?

About one per word. Our canon line, "The service of an SEO agency at the price of a tool.", is 12 words and 52 characters, and took 13 tokens on OpenAI's o200k_base tokenizer: one per word plus the full stop. A 58-word English paragraph took 66. Names, codes and dates cost far more per character.

| Figure | What it counts | n |
| --- | --- | --- |
| 13 | tokens for the 12-word canon line | 52 characters, o200k_base |
| 66 | tokens for a 58-word English paragraph | 58 words, o200k_base |

Window 2026-10-04 to 2026-10-04. Attensira tokenizer test, tiktoken 0.14.0 (o200k_base), run 2026-10-04.

_Source: Attensira tokenizer test, tiktoken 0.14.0, as of 2026-10-04 (1 sentence and 1 paragraph)._

## How do tokens differ from words, characters and embeddings?

| Unit | What it is | Where you meet it |
| --- | --- | --- |
| Character | One letter, digit or symbol | Meta description limits |
| Word | Space-separated text | Word counts, readability |
| Token | One piece from the model's vocabulary | Context windows, output limits, API prices |
| Embedding | A vector of numbers for a whole passage | Retrieval, vector search |

Anthropic documents that a Claude token is roughly 3.5 English characters, and that it varies by language. Treat any characters-per-token figure as an average, not a conversion.

## Is a token in robots.txt the same thing?

No, and AI answers use the word both ways. In robots.txt a token is the name a crawler answers to, such as GPTBot, and nothing is split. ChatGPT's answer below uses it in that sense. A model's tokenizer cuts that same name into two pieces, GPT and Bot.

**ChatGPT, asked 2026-10-04:** "list of AI crawler user agents"

> Google-Extended and Applebot-Extended are robots.txt control tokens, not independent HTTP crawlers. You generally won't see requests with those exact User-Agent strings in your server logs.

Sources cited, in order: github.com, cloute.ai, quanticdata.io.

Attensira: not named.

_Source: attensira.com workspace, read over the Attensira MCP, as of 2026-10-04 (1 stored answer, ChatGPT, 2026-10-04)._

## Why do tokens matter outside an API?

Because AI engines spend a token budget on your page. When an engine fetches a page to answer a question, the text that reaches the model is limited: by the [context window](/glossary/context-window), by how many pages it read, and by how long a passage it keeps from each. A page that opens with thousands of tokens of navigation, cookie text and preamble spends that budget before it reaches the answer.

What helps:

- **Put the answer first.** The first lines under each heading should answer that heading's question.

- **Server-render the text.** A crawler that does not run JavaScript sees only what the HTML contains.

- **Keep sections self-contained.** A passage that is cut out alone should still make sense.

## What do people get wrong about tokens?

- **Counting words and calling them tokens.** The ratio changes with the language, the names and the numbers in the text.

- **Using one model's count for another.** Each family has its own tokenizer.

- **Thinking a big context window means the whole page is read.** Engines can keep only part of what they fetch, whatever the window size.

- **Ignoring markup.** Raw HTML costs far more tokens than the visible text it carries, which is one reason engines extract text first.

## How can you check a page?

The free [page token inspector](/tools/page-token-inspector) estimates a page's tokens section by section, the way a retriever sees it once navigation and scripts are gone, and shows where a cut at the first N tokens would land. It estimates rather than running a specific tokenizer, because every model counts differently. In a workspace, Attensira reads which of your passages engines actually cite, and the crew moves answers that sit too deep in a page up to where they are read.

## FAQ

### What is a token in AI?

The piece of text a language model actually processes. A tokenizer splits text into tokens from a fixed vocabulary: common words become one token, rarer words are split into pieces, and spaces and punctuation are folded in. The model never sees letters or words, only token numbers.

### How much text is one token?

It depends on the text and the tokenizer. In our test with OpenAI's o200k_base tokenizer, a 58-word English paragraph came to 66 tokens. A name such as Attensira takes 3 tokens, and a date such as 2026-10-04 takes 6. Anthropic documents that a Claude token is roughly 3.5 English characters.

### Why are tokens not just words?

Because a fixed vocabulary cannot hold every word in every language. Splitting rare words into common pieces lets one vocabulary cover any text, at the cost of using more tokens for names, codes and languages the tokenizer saw less of.

### Why do tokens matter for SEO and AI visibility?

Because every limit an AI engine works within is counted in tokens: how much of a fetched page fits in the model's context, how long a passage the engine keeps, how long the answer is. A page whose answer sits after thousands of tokens of navigation and preamble is more likely to be cut.

### Do different models count tokens differently?

Yes. Each model family has its own tokenizer, and even one vendor changes them: OpenAI's newer models use o200k_base, the older ones cl100k_base. Anthropic says the count for Claude also varies with the language. Token counts from one model are an estimate for another.