Real-time search

A language model on its own knows only what it learned before its training ended. Real-time search is how an assistant gets past that: during the conversation it searches the web, reads current pages and writes its answer from them, usually with links. It turned AI answers from a snapshot of the past into something a page published this morning can change.

How does real-time search work?

Four steps, each one a place a page can drop out:

1. Decide. The model judges whether the question needs a search. Prices, products, comparisons and anything recent usually do. A stable definition may not.
2. Query. It writes one or more search queries, often several for one question, and sends them to an index: the operator's own, or a partner's.
3. Fetch and read. It takes some of the results, fetches the pages, and keeps the passages that look relevant.
4. Write. It answers from those passages and attaches the pages it used as citations.

This is retrieval-augmented generation applied to the open web. The model still brings what it learned in training; search adds what is true now.

Which crawlers make it possible?

Two kinds, both separate from the crawlers that collect training data:

• Search crawler

  What it does

  Builds the index the assistant searches, ahead of time

  Examples

  OAI-SearchBot, PerplexityBot, Claude-SearchBot

• User-triggered fetcher

  What it does

  Fetches one page because a person's question needed it

  Examples

  ChatGPT-User, Claude-User, Perplexity-User

On our own site, counted at the edge in the 30 days to 4 October 2026, OpenAI's agents made 7,536 requests: 86% classed as OAI-SearchBot building the index, 11% as GPTBot collecting training data, and 3% as ChatGPT-User fetching a page because somebody asked. Anthropic's were fewer, 104 requests, but 35% of them were Claude-User fetches triggered by a person's question. Perplexity's 621 requests were all PerplexityBot indexing, with no live fetches recorded. Each live fetch is a real-time answer reading one of our pages. ChatGPT-User or Claude-User reached 21 of our 30 most-fetched pages, led by /pricing, /audit and /compare: the pages a buyer asks about.

Note the asymmetry in control. OpenAI says robots.txt may not apply to ChatGPT-User, because a person initiated the fetch. The search crawlers do follow robots.txt, and blocking one removes you from that engine's search answers.

Why does it decide whether you are quotable?

Because it is the only path for anything new. A model trained before your launch, your price change or your rebrand cannot know about it. If the engine searches and can fetch your page, the answer can be correct within days. If your page blocks the search crawler, renders only in JavaScript, or sits behind a login, the engine falls back on memory or on other people's pages about you.

The speed is documented on the access side: OpenAI says a robots.txt change reaches its search systems in about 24 hours. Training-based knowledge waits for the next model.

What goes wrong with it?

• The engine does not search. If it judges the question answerable from memory, your current page is never read.
• The engine reads a different page. It searches its index with its own queries, and an older or better-ranked page about you can win.
• The page cannot be read without JavaScript. Crawlers fetch HTML; content assembled in the browser may not exist for them.
• Facts are mixed. An answer can combine a fetched price with a remembered feature list.
• Surfaces differ. The consumer app and the developer API can search differently, so testing through the API does not show what a buyer sees.

How does Attensira measure real-time answers?

By asking the way a buyer does. Attensira reads chatgpt.com, perplexity.ai, Google's AI Overview and Gemini as a person would wherever those surfaces can be read, and uses the model APIs only as a labelled fallback; every reading records which it came from, and the two are never averaged. Each answer is stored with its links, so you can see which of your pages a real-time answer used and which page beat it. The crawler logs view shows which search crawlers and user-triggered fetchers reached your pages, and when.