Context window
The token budget a model can reference in one request, what counts against it, and why a bigger window does not remove the need to retrieve selectively.
Karl-Gustav Kallasmaa, Founder & CEOLast updated A context window is the amount of text a model can reference when generating a response, including the response itself. Anthropic's documentation describes it as working memory, explicitly distinct from the large corpus of data the model was trained on. It is measured in tokens, not words or characters, and it is the budget every other limit is expressed against.
Sizes are a property of a specific model, not of AI in general. Anthropic documents several current Claude models with a 1M-token context window, while others including Claude Sonnet 4.5 carry a 200k-token window, and notes a related cap: a single request can include up to 600 images or PDF pages, or 100 on the 200k-token models.
What actually counts against it
More than people expect, which is why requests fail at sizes that seem safe. Anthropic states that everything in the request counts toward the context window: the system prompt, every message in the conversation including tool results, images and documents, and the tool definitions themselves. The output generated for the turn counts too, including any thinking the model does.
That last point is the one that surprises application builders. Tool definitions are input tokens on every single request, so an agent with a large tool catalogue pays that cost continuously whether or not it uses any of them.
Bigger is not simply better
The intuitive model — fill the window, get better answers — is contradicted by the vendor's own documentation. Anthropic writes that more context is not automatically better, and that as token count grows, accuracy and recall degrade, a phenomenon it names context rot. Its conclusion is that curating what is in context is just as important as how much space is available.
This is the crucial point for anyone reasoning about AI search, because it disposes of the argument that growing windows will make retrieval obsolete. Two things stop that happening. The window is finite and the web is not, so something still has to choose. And even well inside the limit, indiscriminate filling degrades the answer, so choosing well is not merely an efficiency concern. Retrieval-augmented generation is the name for doing that choosing deliberately.
How it differs from memory, and from training
Three storage-shaped ideas are routinely conflated:
- Training fixed the model's weights before you arrived. It is not queryable, not editable and not attributable. See AI training data.
- The context window is per request. It holds only what was sent this time, and it is discarded afterwards. Nothing carries over unless the application sends it again.
- Application-level memory is a product feature built on top: the application stores something and re-sends it later. From the model's side it is still just tokens in the window.
The practical consequence is that "the assistant remembers our conversation" describes an application resending history, not a model retaining it.
How to act on it
For anyone whose interest is being read and cited rather than building on the API, the useful implications are about page structure:
- Write self-contained passages. A retriever selects spans, not documents. A paragraph whose subject and qualifier are both present survives extraction; one that depends on three paragraphs above it does not.
- Put the claim under the heading that promises it, so the span selected on a heading match contains the substance.
- Front-load the specifics. Numbers, definitions and comparisons early in a section make the first selected span the informative one.
- Keep pages focused. A page covering one question well is easier to select from than a page covering six, which is a real argument against merging thin pages into one long one just to clear a word count.
- Do not pad. Padding costs budget on the retrieval side and adds nothing that could be quoted.
Frequently asked questions
How large is a context window?
Model-specific: Anthropic documents 1M tokens on several current Claude models and 200k on others.
Is it the same as memory?
No. It is per-request working memory, distinct from training and from application-level memory.
What counts against it?
System prompt, all messages, tool results, documents, tool definitions and the generated output.
Do larger windows remove the need for retrieval?
No. The window is finite, and accuracy and recall degrade as it fills.
Terms related to Context window
The units a language model actually reads and writes, why they are not words, and why every limit and price you meet is denominated in them.
The architecture that retrieves documents at query time and has a model write from them, and the reason your page can be quoted without ever being trained on.
Vectors of floating point numbers whose distance measures relatedness, the representation underneath semantic search and retrieval.
Search where a model composes the answer and fetches pages through named crawlers, rather than returning a ranked list of links for you to read.