For the complete documentation index, see llms.txt. Every page on this site is also served as Markdown: append `.md` to any URL, or send `Accept: text/markdown`.
Skip to content
Attensira LogoAttensira
Book a demo

Bots

AI product crawlers

AI product crawlers feed an AI product other than a search index or a training corpus. Google-CloudVertexBot crawls a site for Vertex AI Agents when the site's owner asks for it, and ImagesiftBot collects public images for Hive's ImageSift products. Both say they respect robots.txt, and neither is how an assistant finds pages to cite.

Start freeBook a demo

7 days free · card required

AI product crawlers2 agents
agents documented
2
publish IP ranges or reverse DNS
1 of 2
requests to attensira.com, 30 days
0
of n=143,601 AI crawler requests

The rule each entry recommends, from the Google-CloudVertexBot and ImagesiftBot entries, per the operator's documentation.

User-agent: Google-CloudVertexBotDisallow: / User-Agent: ImagesiftBotDisallow: /
Researcher agentResearcherOperator documentation and our AI traffic board (Attensira MCP) ·

Which AI product crawlers are documented?

Feeds an AI product that is neither a search index nor a training corpus, by the operator's own description. Not how an assistant finds pages to cite.

Which AI product crawlers fetched attensira.com in the last 30 days?

None of these agents had a request filed on attensira.com in the window. User agents as sent, not checked against address lists.

See which crawlers visit your site.

AI traffic · attensira.com0 of 2 seen
Requests to attensira.com by agent, last 30 days to 4 Oct 2026, and attensira.com's robots.txt rule for each
AgentRequestsOur robots.txt
Google-CloudVertexBot0Follows Googlebot
ImagesiftBot0No named group
AI traffic board, read over the Attensira MCP · · n=143,601 requests

How do the AI product crawlers compare?

Each row is transcribed from the operator's own documentation. The attensira.com column is ours: what our live robots.txt says to the agent.

  • Google-CloudVertexBot

    Operator
    Google
    robots.txt
    Respected
    Verify by
    IP ranges + reverse DNS
    attensira.com
    Follows Googlebot

  • ImagesiftBot

    Operator
    Hive
    robots.txt
    Respected
    Verify by
    Nothing published
    attensira.com
    No named group

Each row read from the operator's documentation, 3 Sep 2026 to 4 Sep 2026.

Why would Google-CloudVertexBot crawl my site?

Google documents its crawls as requested by the site owner, for a Vertex AI agent built over that site. If nobody on your side set one up, Google's documented reason for the visit does not apply, and the request is worth verifying against Google's common-crawlers.json before you trust it.

Google states that a rule for Google-CloudVertexBot has no effect on Google Search. Name the token in its own group: a Googlebot group can also address it, so a rule written for search can reach it by accident.

VerificationGoogle-CloudVertexBot
Reverse DNS ends in
*.googlebot.com*.geo.googlebot.com
IP range file
common-crawlers.json

One of Google's common crawlers, so it shares Googlebot's verification path. Run it before you treat an unexpected visit as unauthorised, and before you go looking for the team that asked for the crawl: Google's crawler respects robots.txt, but an obscure token nobody has a rule for is a convenient disguise. The Google-CloudVertexBot entry, and the method step by step: how to verify a crawler.

Researcher agentResearcherTranscribed from Google documentation ·

Can I verify ImagesiftBot?

No. ImageSift publishes no IP ranges and no reverse DNS suffix for the crawler; its page offers an email address for questions and opt-outs. A request claiming to be ImagesiftBot cannot be confirmed from anything the operator has published.

ImageSift documents crawl-delay support, so you can slow it down instead of shutting it out. Blocking it keeps your images out of ImageSift's index and changes nothing about how search engines or assistants see your pages.

VerificationImagesiftBot

Hive publishes no reverse-DNS suffix and no IP range file for ImagesiftBot, so a visit can be matched only by its user agent.

The ImagesiftBot entry, and the method step by step: how to verify a crawler.

Researcher agentResearcherTranscribed from Hive documentation ·

What else do people ask?

What counts as an AI crawler in this directory?
A crawler that feeds an AI product but is neither a search index nor a training crawl, by its operator's own description. GPTBot and ClaudeBot are filed under AI training, and OAI-SearchBot and PerplexityBot under AI search, because their operators say that is what the fetch is for.
Does blocking Google-CloudVertexBot affect Google Search?
No. Google states that preferences addressed to Google-CloudVertexBot do not affect Google Search. Write it its own group so that a Googlebot rule does not reach it by accident.
Can I slow ImagesiftBot down instead of blocking it?
Yes. ImageSift documents support for the crawl-delay directive. Because it publishes no addresses, robots.txt is the only lever its operator offers.
Do AI product crawlers decide whether AI answers cite me?
No. Neither operator describes the crawl as feeding an assistant's search index or a training corpus. The agents that put a page in front of an assistant are the AI search crawlers and the user-initiated fetchers, which have their own categories here.

Where do these facts come from?

5 claims on this page, each with the page it was read from; last checked .

  1. Google documents that only one robots.txt group is valid for a particular crawler. Its crawlers follow the group with the most specific user agent that matches and ignore the others, and user-agent-specific groups are not combined with the * group.

    developers.google.com · retrieved

    “Only one group is valid for a particular crawler.”

  2. Google documents that crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners for building Vertex AI Agents, and that it has no effect on Google Search or other products.

    developers.google.com · retrieved

    “Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners' for building Vertex AI Agents.”

  3. Google states that its common crawlers always respect robots.txt rules for automatic crawls.

    developers.google.com · retrieved

    “always respect robots.txt rules for automatic crawls”

  4. Google documents verifying its common crawlers by reverse DNS lookup to a hostname ending in googlebot.com or geo.googlebot.com followed by a forward lookup back to the same address, or by matching the address against its published common-crawlers.json range file.

    developers.google.com · retrieved

  5. ImageSift documents ImageSiftBot as a web crawler that scrapes the internet for publicly available images to support its suite of web intelligence products.

    imagesift.com · retrieved

    “web crawler that scrapes the internet for publicly available images to support our suite of web intelligence products”

Want whatever it takes to win the answer?

Start freeBook a demo

7 days free · card required