For the complete documentation index, see llms.txt. Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.
Skip to content
Attensira LogoAttensira
Book a demo

Blog · AI Search Visibility

How ChatGPT finds, reads and cites your content

ChatGPT reaches web pages through three crawlers OpenAI documents. OAI-SearchBot fetches pages for ChatGPT search, GPTBot fetches pages that may be used for training, and ChatGPT-User fetches a page when a person's question needs it. Each can be allowed or blocked separately in robots.txt, and only pages it can reach can be cited.

Start freeBook a demo

7 days free · card required

OpenAI crawler requests
7,536
428 pages
Search indexing (OAI-SearchBot)
86%
of 7,536
Training (GPTBot)
11%
of 7,536
Live fetches (ChatGPT-User)
3%
of 7,536
Analyst agentAnalyst · reads this workspace's crawler logsattensira.com, Attensira MCP · by claimed user agent, rounded purpose mix · n=7,536 requests · to
Karl-Gustav KallasmaaKarl-Gustav Kallasmaa, Founder & CEO

Last updated

Key facts

"Indexing" is a Google word, and ChatGPT does not work quite like Google. There is no Search Console for ChatGPT, no index coverage report, no published ranking documentation. What there is: three documented crawlers, a robots.txt you control, and answers you can read.

This post explains what OpenAI documents about how ChatGPT reaches web content, what you control, what you can observe, and what our own logs and answers show over a month.

How does ChatGPT get to your pages?

OpenAI documents three user agents that fetch web pages, each with its own job:

  • OAI-SearchBot

    What OpenAI says it is for
    "used to surface websites in search results in ChatGPT's search features"
    When it fetches
    On its own schedule, to keep pages available for search

  • GPTBot

    What OpenAI says it is for
    crawls content "that may be used in training our generative AI foundation models"
    When it fetches
    On its own schedule, for training

  • ChatGPT-User

    What OpenAI says it is for
    "for certain user actions in ChatGPT and Custom GPTs"
    When it fetches
    When a person's question leads ChatGPT to visit a page

So there are really three paths from your page to an answer:

  1. Search. OAI-SearchBot has fetched your page before. When someone asks a question that needs current information, ChatGPT can retrieve it and cite it.
  2. Live fetch. A person's question leads ChatGPT to visit your page right then, as ChatGPT-User.
  3. Memory. The model learned something about you during training, from pages GPTBot or other sources gathered, and answers without fetching anything. These answers usually carry no link to you.

The first two produce citations. The third produces mentions without links, which is why your brand can be named in an answer that cites nobody.

What does this look like on a real site?

Here is our own site for the 30 days to 2026-10-04, read from our workspace that day. The frame at the top of this page has the crawling side.

Crawling. OpenAI's crawlers made 7,536 requests across 428 distinct pages. About 86% were search indexing, 11% training and 3% live fetches triggered by someone's question. Those shares are OpenAI's rounded purpose mix, so a per-crawler count is an approximation. These are counts by claimed user agent; none were verified, because OpenAI's crawlers cannot be verified the way Googlebot can.

Citing. On the 61 questions we track, ChatGPT answered 2,349 times in the window and linked our domain in 154 of them, 91 on the three questions that contain our name. ChatGPT adds utm_source=chatgpt.com to many of the links it shows, so clicks arrive in analytics labelled as coming from ChatGPT.

The gap. The OpenAI crawlers fetched 428 of our pages. The pages the engines cited most were a small set: the home page, the compare hub, pricing and a few comparison pages. This post was cited in 5 of 4,524 answers, every one to "what is Attensira". Being fetched is a condition for being cited, not a cause. Our own documentation puts it plainly: a model answers a prompt, a crawler fetches pages, and "a high crawl rate does not produce mentions".

What do you control?

Access, per crawler

robots.txt can allow or block each crawler separately, because each has its own user agent. Common choices:

  • Allow search, block training: allow OAI-SearchBot, disallow GPTBot. Your pages can be cited in ChatGPT search; they are not offered for training.
  • Allow everything: the default if you do nothing.
  • Block everything: your pages cannot be fetched for search or training. ChatGPT can still mention you from what it learned before, without a link.

OpenAI notes that a robots.txt change can take about 24 hours to take effect in its systems.

Two other things block crawlers more often than robots.txt does: bot protection at your CDN or host, which may challenge or drop crawler requests, and pages that only render their text with JavaScript, which a crawler fetching HTML will see as empty.

What the page says

Once a page can be fetched, what decides whether it is cited is its content: whether it answers the question, in a passage that stands alone. See AI search chunking and becoming the sentence AI search writes.

Where else you appear

ChatGPT often cites pages about you rather than pages by you: roundups, reviews, comparisons. For category questions those decide more answers than your own site does.

What can you observe?

Three things, and you need all three:

  • Crawler fetches

    Where it comes from
    Server, CDN or edge logs
    What it answers
    Can ChatGPT reach and read my pages?

  • Answers and citations

    Where it comes from
    Asking ChatGPT your buyers' questions, repeatedly
    What it answers
    Does ChatGPT name me, and which pages does it cite?

  • Referral visits

    Where it comes from
    Analytics, via utm_source=chatgpt.com
    What it answers
    Do people click through from ChatGPT?

None of them alone is enough. Fetches without citations means your pages are read but not used. Citations without fetches means ChatGPT is using pages about you, not yours. Referrals are the smallest number and the latest to move.

For the crawler side, see analyzing log files. For the answer side, see how to audit your AI mentions.

What do the three signals say about our own site?

The same month, read three ways. OpenAI's crawlers reached hundreds of our pages; ChatGPT linked us in a small share of its answers, most of them on questions that already contain our name; and this post was cited only when someone asked what Attensira is.

attensira.com, 30 days to 4 October 2026

  • Pages OpenAI's crawlers fetched

    Reading
    428
    Out of
    7,536 requests

  • ChatGPT answers linking attensira.com

    Reading
    154
    Out of
    2,349 answers

  • …on the 3 questions with our name

    Reading
    91
    Out of
    154 linking answers

  • Answers citing this post

    Reading
    5
    Out of
    4,524 answers

  • Referral visits from ChatGPT

    Reading
    Not read here
    Out of
    Analytics
Analyst agentAnalyst · reads this workspace's answers and crawler logsattensira.com workspace, read over the Attensira MCP · · n=4,524 answers, 7,536 requests

What should you check first if ChatGPT never cites you?

Work through this in order. Each step is cheaper than the next.

  1. robots.txt. Is OAI-SearchBot allowed? Look for a broad Disallow under User-agent: * that you forgot.
  2. Bot protection. Does your CDN or host challenge unknown bots? Check its logs for blocked requests from OpenAI's user agents.
  3. Rendering. Fetch your key pages with a plain HTTP client. If the text is missing, crawlers see the same.
  4. Fetches. In your logs, has OAI-SearchBot fetched your key pages in the last month?
  5. Content. Does the page answer the question someone would ask ChatGPT, in the first lines?
  6. Off-site. For category questions, are you on the pages ChatGPT cites? Usually that is where the answer is decided.

Most sites that are never cited fail at step 5 or 6, not at 1 to 4. But steps 1 to 4 take an hour, so do them first.

What does ChatGPT say when you ask why you are missing?

Asked on 3 October 2026, ChatGPT first guessed the question was about profiles or custom GPTs. For websites it gave the same first check as this post: allow OAI-SearchBot. It cited only OpenAI's help centre, which carried 797 of 1,395 citations on this question in 30 days.

Answer · ChatGPTasked

Question: why am I not showing up in ChatGPT

ChatGPT answered: If you mean “why can’t people find me or my profile on ChatGPT?”, there are a few possible reasons. … If you mean you’re not showing up in ChatGPT Search results—for example, you have a website/business/persona that you expected ChatGPT to mention—that’s different. ChatGPT says search placement isn’t guaranteed; websites need to allow OAI-Searchbot to crawl them to be eligible.

Sources cited

  1. 1help.openai.com

Not namedAttensira is missing from this ChatGPT answer.

Analyst agentAnalyst · reads this workspace's answersattensira.com workspace, read over the Attensira MCP · · 1 stored answer
How we read these numbers

Do llms.txt files or special markup help ChatGPT read you?

Not in any way OpenAI documents. OpenAI's crawler documentation describes user agents and robots.txt, and says nothing about llms.txt or AI-specific markup. Google is explicit for its own AI features: you "don't need to create new machine readable files, AI text files, or markup" to appear in them. An llms.txt file is harmless and can be useful to developers pointing their own tools at your docs, but there is no documented evidence that it changes what ChatGPT cites.

What does help is ordinary and verifiable: pages that return their text in the HTML, a sitemap that lists them, internal links that reach them, fast responses, and no bot challenge in front of them. Those are the same things that let any crawler read a site, and they are the part of "indexing" you can check yourself.

How should a docs or help site be set up for ChatGPT?

Docs and help centres are among the most cited pages on the web for "how do I" questions. In our own workspace, OpenAI's help centre and Google's developer documentation were the two most cited domains across every question we track. If you have docs, they are likely your best chance of being cited for implementation questions. Four things matter most:

  • One page per task, with the task as the title, so a question maps to a page.
  • The answer at the top of the page, before prerequisites and background.
  • Text in the HTML, not loaded by a client-side search widget or a single-page app shell.
  • Docs on a path crawlers can reach, with no login wall and no bot challenge.

A docs site built as a static site from Markdown, as most modern docs tools produce, usually gets all four right by default.

Does one engine name us more than the other?

The same 61 questions, read in ChatGPT and in Perplexity, counted apart. The two name attensira.com at about the same rate; a gap this small is inside the noise, so it is not a finding.

of ChatGPT answers name attensira.com
4.2%
n=2,346 answers
of Perplexity answers name attensira.com
4.5%
n=2,172 answers
link to it, ChatGPT / Perplexity
6.6% / 3.9%
n=4,518 answers
Analyst agentAnalystattensira.com workspace, Attensira MCP, ChatGPT and Perplexity · n=4,518 answers · to

What does OpenAI not document?

It is worth being clear about the limits. OpenAI does not publish how often OAI-SearchBot re-crawls a page, how search results are ranked, how many sources an answer considers, or how it chooses between them. Anyone who tells you the exact rules is guessing. What you can do is measure: read the answers, count the citations, and compare before and after each change you make.

How does ChatGPT treat freshness?

ChatGPT search can fetch recent pages, and live fetches by ChatGPT-User read the page as it is at that moment. But answers that come from memory reflect whatever the model learned, which can be months old. That is why an out-of-date fact can survive in ChatGPT answers after you have fixed the page: some answers come from the fixed page, others from memory. Read the citations to tell which is which; we wrote up how to handle that in freshness conflicts.

How does Attensira track ChatGPT?

Attensira reads ChatGPT where your buyers do: on the consumer surface, every day, for each question you track, with the other engines your plan includes beside it (ChatGPT, Perplexity, Claude and Google AI, with Gemini from Growth). It stores each answer and its citations, so you can see which of your pages ChatGPT used and which other pages decided the answers you lost. On Growth and above, crawler logs at your edge show which of your pages OpenAI's crawlers fetched. Then the crew does the work: fixing the page, drafting the pitch to the page that decided the answer, with your approval at each step.

That is the service of an SEO agency at the price of a tool. See how Attensira tracks ChatGPT, or run the free audit to see what ChatGPT says about your category today.

What else do people ask?

Does ChatGPT have its own search index?
OpenAI documents a crawler, OAI-SearchBot, used to surface websites in ChatGPT's search features, which means it keeps pages it can return in search. OpenAI does not publish how that index is built or ranked. What you can observe is whether its crawlers fetch your pages and whether answers cite them.
If I block GPTBot, will my site disappear from ChatGPT search?
Not according to OpenAI's documentation. GPTBot is the crawler used for training, and OAI-SearchBot is the one used for search features. They are separate user agents controlled separately in robots.txt, so you can block training and still allow search, or the reverse.
How long does a robots.txt change take to apply in ChatGPT?
OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust. That is about access. It does not mean a page's content changes in answers within 24 hours.
How can I see visitors who came from ChatGPT?
ChatGPT adds a utm_source=chatgpt.com parameter to many of the links it shows, and those visits appear in your analytics as referrals. The crawler visits themselves never appear in browser analytics, because crawlers do not run JavaScript; they appear only in server or edge logs.

Where do these facts come from?

2 sources, 4 statements: each statement with the page it came from and the day it was checked.

  1. OpenAI uses OAI-SearchBot to surface websites in search results in ChatGPT's search features.

    developers.openai.com · retrieved

  2. OpenAI uses GPTBot to crawl content that may be used in training its generative AI foundation models.

    developers.openai.com · retrieved

  3. OpenAI uses ChatGPT-User for certain user actions in ChatGPT and Custom GPTs, and it may visit a web page when a user asks a question.

    developers.openai.com · retrieved

  4. Attensira reports crawlers and models separately, because a crawler fetches pages while a model answers prompts, and a high crawl rate does not produce mentions.

    docs.attensira.com · retrieved

Want software that does your AI-search visibility work, with you in control of every change?

Start freeBook a demo

7 days free · card required