# Real-time search

> For the complete documentation index, see [llms.txt](https://attensira.com/llms.txt).
> Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.

> Real-time search is an AI assistant fetching current web pages as it answers. How it works, which crawlers do it, and why a new page can be cited within days.

- Canonical page: https://attensira.com/glossary/real-time-search
- Family: glossary
- Last updated: 2026-10-04
- Author: Karl-Gustav Kallasmaa (https://attensira.com/about)
- Claims as data: https://attensira.com/claims.json?slug=real-time-search

**In short:** Real-time search is an AI assistant looking up current web pages while it answers, instead of relying only on what it learned in training. The assistant decides to search, sends queries to an index, fetches some results and writes from them. It is why a page published or corrected today can appear in answers within days.

| Figure | What it counts | n |
| --- | --- | --- |
| 3% | of OpenAI requests were live fetches | ChatGPT-User, of 7,536 |
| 35% | of Anthropic requests were live fetches | Claude-User, of 104 |
| 0% | of Perplexity requests were live fetches | of 621, all PerplexityBot |

Window 2026-09-05 to 2026-10-04. attensira.com AI traffic, read over the Attensira MCP on 2026-10-04; labelled by the user agent each request claimed, none IP-verified.

_Source: attensira.com AI traffic, read over the Attensira MCP, as of 2026-10-04 (143,601 AI-crawler requests, 30 days)._

## Key facts

- Also known as: Live web search; AI web search
- Steps: Decide to search, query an index, fetch and read, write with citations
- Crawlers involved: Search crawlers and user-triggered fetchers such as ChatGPT-User
- Robots.txt change: [About 24 hours to reach OpenAI's search systems](https://developers.openai.com/api/docs/bots)
- Related term: [RAG, the same mechanism over any document set](/glossary/rag)
- In Attensira: [AI traffic: crawler requests per page, at the edge](https://docs.attensira.com/traffic/ai-traffic)

A language model on its own knows only what it learned before its training ended. Real-time search is how an assistant gets past that: during the conversation it searches the web, reads current pages and writes its answer from them, usually with links. It turned AI answers from a snapshot of the past into something a page published this morning can change.

## How does real-time search work?

Four steps, each one a place a page can drop out:

1. **Decide.** The model judges whether the question needs a search. Prices, products, comparisons and anything recent usually do. A stable definition may not.

1. **Query.** It writes one or more search queries, often several for one question, and sends them to an index: the operator's own, or a partner's.

1. **Fetch and read.** It takes some of the results, fetches the pages, and keeps the passages that look relevant.

1. **Write.** It answers from those passages and attaches the pages it used as [citations](/glossary/source-citation).

This is [retrieval-augmented generation](/glossary/rag) applied to the open web. The model still brings what it learned in training; search adds what is true now.

## Which crawlers make it possible?

Two kinds, both separate from the crawlers that collect training data:

| Kind | What it does | Examples |
| --- | --- | --- |
| Search crawler | Builds the index the assistant searches, ahead of time | OAI-SearchBot, PerplexityBot, Claude-SearchBot |
| User-triggered fetcher | Fetches one page because a person's question needed it | ChatGPT-User, Claude-User, Perplexity-User |

On our own site, counted at the edge in the 30 days to 4 October 2026, OpenAI's agents made 7,536 requests: 86% classed as OAI-SearchBot building the index, 11% as GPTBot collecting training data, and 3% as ChatGPT-User fetching a page because somebody asked. Anthropic's were fewer, 104 requests, but 35% of them were Claude-User fetches triggered by a person's question. Perplexity's 621 requests were all PerplexityBot indexing, with no live fetches recorded. Each live fetch is a real-time answer reading one of our pages. ChatGPT-User or Claude-User reached 21 of our 30 most-fetched pages, led by /pricing, /audit and /compare: the pages a buyer asks about.

Note the asymmetry in control. OpenAI says robots.txt may not apply to ChatGPT-User, because a person initiated the fetch. The search crawlers do follow robots.txt, and blocking one removes you from that engine's search answers.

## Which of our pages did a person's question fetch?

The pages buyers ask about: pricing, the audit, comparisons, the sales contact. A user-triggered fetcher reached 21 of our 30 most-fetched pages in the 30 days; six are shown. The log gives agents per page, not individual requests or status codes, so that is all this readout claims.

**attensira.com pages a user-triggered fetcher reached, 5 Sep to 4 Oct 2026**

| Page | User-triggered fetchers seen | Search crawlers seen |
| --- | --- | --- |
| /pricing | ChatGPT-User, Claude-User | OAI-SearchBot, PerplexityBot, Googlebot |
| /audit | ChatGPT-User, Claude-User | OAI-SearchBot, PerplexityBot, bingbot |
| /compare | ChatGPT-User, Claude-User | OAI-SearchBot, PerplexityBot |
| /contact-sales | Claude-User | OAI-SearchBot, PerplexityBot, bingbot |
| /features/crawler-logs | Claude-User | OAI-SearchBot, PerplexityBot, Googlebot |
| /generative-engine-optimization | ChatGPT-User | OAI-SearchBot, PerplexityBot, Googlebot |

_Source: attensira.com AI traffic, read over the Attensira MCP, as of 2026-10-04 (6 of the 30 most-fetched pages; 143,601 AI-crawler requests in 30 days)._

## Why does it decide whether you are quotable?

Because it is the only path for anything new. A model trained before your launch, your price change or your rebrand cannot know about it. If the engine searches and can fetch your page, the answer can be correct within days. If your page blocks the search crawler, renders only in JavaScript, or sits behind a login, the engine falls back on memory or on other people's pages about you.

The speed is documented on the access side: OpenAI says a robots.txt change reaches its search systems in about 24 hours. Training-based knowledge waits for the next model.

## What goes wrong with it?

- **The engine does not search.** If it judges the question answerable from memory, your current page is never read.

- **The engine reads a different page.** It searches its index with its own queries, and an older or better-ranked page about you can win.

- **The page cannot be read without JavaScript.** Crawlers fetch HTML; content assembled in the browser may not exist for them.

- **Facts are mixed.** An answer can combine a fetched price with a remembered feature list.

- **Surfaces differ.** The consumer app and the developer API can search differently, so testing through the API does not show what a buyer sees.

## How does Attensira measure real-time answers?

By asking the way a buyer does. Attensira reads chatgpt.com, perplexity.ai, Google's AI Overview and Gemini as a person would wherever those surfaces can be read, and uses the model APIs only as a labelled fallback; every reading records which it came from, and the two are never averaged. Each answer is stored with its links, so you can see which of your pages a real-time answer used and which page beat it. The [crawler logs](/features/crawler-logs) view shows which search crawlers and user-triggered fetchers reached your pages, and when.

## FAQ

### What is real-time search in AI assistants?

The assistant runs a web search during the conversation and reads current pages before answering, so it can use information newer than its training data. ChatGPT search, Perplexity, Google's AI Overviews and Claude with web search all work this way, with links back to the pages they used.

### How does an assistant decide whether to search?

The model judges whether the question needs current or specific facts it does not reliably know. Questions about prices, recent events, products and comparisons usually trigger a search; definitions of stable concepts often do not. The decision varies by engine, by surface and from run to run.

### Which crawlers fetch pages for real-time answers?

Two kinds. Search crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot build the index the assistant searches. User-triggered fetchers such as ChatGPT-User and Claude-User fetch a specific page because a person's question needed it. Both are separate from training crawlers.

### How fast can a new page appear in AI answers?

As fast as the engine's index picks it up, often within days, provided its search crawler is allowed and the page is server-rendered. OpenAI says a robots.txt change takes about 24 hours to reach its search systems. Training-based knowledge, by contrast, waits for a new model.

### Does real-time search mean the answer is always current?

No. The engine may not search for every question, may retrieve an older page that ranks better, or may mix fetched facts with remembered ones. A correct, current page improves the odds; it does not guarantee the answer uses it.