Page token inspector

The page is fetched once, server-side. Everything inside script, style, noscript, svg, template, head, iframe and canvas is discarded, HTML entities are decoded, and runs of whitespace are collapsed. What survives is the visible text, in document order.

That text is split at every h1 to h6. Everything before the first heading becomes the preamble. On our own pages that is the navigation and whatever sits above the H1, whether or not any of it is inside a nav element. Text that is inside nav, header, footer or aside is counted separately so you can see how much of the budget is spent on furniture that repeats on every page.

Tokens are then estimated at about 4 characters per token, applied to each block of text and summed. That ratio is the common rule of thumb for English prose in byte-pair vocabularies. It is not a measurement of any specific tokenizer, and this tool does not run one.

So these are approximations, and we will not dress them up as anything else. A real tokenizer would give you a precise number for one model and the wrong number for the next. What does not change between tokenizers is the shape: which sections are expensive, how much of the page is furniture, and how far down your answer sits. That shape is what this tool is for. Where the ratio is least fair (text outside the Latin alphabet, which it under-counts) the result says so instead of quietly reporting a low number.

The cut line is the number you typed. It is a chosen reference point, not a measured constant, and it is worth being blunt about why: no answer engine publishes how many tokens of a fetched page it keeps before answering. Retrieval budgets differ by product, by model, by query and by how many other documents are competing for the same context. No engine publishes one.

What the tool does with your number is arithmetic, and the arithmetic is honest: it walks the sections in document order, accumulates the estimated tokens, and reports where your number falls. Move the line and everything moves with it. Testing the same page at 800, 2,000 and 8,000 tokens tells you how sensitive the page is to a budget you cannot control, which is more useful than pretending to know the budget.

It also does not know which sentence answers your reader. It reports where the page's content starts and which sections fall below the line. Whether the section that got cut was the one that mattered is a judgement only you can make, so the tool declines to make it for you.

A model does not read your page the way a person does. Something fetches it, strips it, and hands a bounded slice of the text to the model along with the question. What is in that slice is what can be quoted, summarised or cited. What is outside it may as well not exist for that answer.

That is why the ordering of a page turns out to matter more than its length. A 12,000-token page whose answer is in the first 300 tokens is in better shape than a 2,000-token page that spends 900 tokens on navigation before it says anything. Length is not the problem; spending the beginning of the budget on furniture is.