Bots
YouBot: You.com's search crawler
YouBot is the crawler behind You.com's search engine. It indexes pages for real-time search results, respects robots.txt and crawl-delay, and caches robots.txt for 30 minutes. Blocking it removes you from You.com results. You.com publishes three ways to verify a visit: reverse DNS to search.you.com, an address range, and signed requests under Web Bot Auth.
7 days free · card required
0requests filed as YouBot · 30 days
of 143,601 AI crawler requests from 7 operators
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| YouBot (this page) | 0 | Allowed |
| OAI-SearchBot | ≈6,480 | Allowed |
| PerplexityBot | 621 | Allowed |
| Amzn-SearchBot | 0 | Parent's rule |
| Claude-SearchBot | 0 | Allowed |
| meta-webindexer | 0 | Wildcard |
| MistralAI-Index | 0 | Wildcard |
2 of 7 had requests filed in the window. User agents as sent, not checked against address lists.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP, and our live robots.txt · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block YouBot in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves YouBot today.
# Keep YouBot, and slow it down if its load is a problem.User-agent: YouBotAllow: /# Crawl-delay: 10 # You.com's own documented block, if you decide to leave# its results:# User-agent: YouBot# Disallow: /User-agent: YoubotAllow: /Disallow: /api/We name YouBot and allow it everywhere except /api/, the same as every AI and search crawler we allow by name.
Read the whole file, or write yours with the robots.txt generator.
Who runs YouBot, and what is it for?
You.com runs it, and we file it under search indexing. Discovers and indexes pages so You.com can return them as real-time search results to its users.
From You.com’s documentation, read . Why a training fetch and a search fetch are different decisions: the AI crawler guide.
How do I slow YouBot down without blocking it?
Crawl-delay is honoured,3 but your server can also throttle without a file edit or the 30-minute robots.txt cache: a 429 reduces the crawl rate and a 503 pauses crawling temporarily. For excessive crawling there is a published contact, spider@you.com.5
Does You.com say what YouBot's content is used for?
Search, as far as its documentation goes. You.com documents YouBot as indexing pages to return real-time search results to its users, and its page makes no statement about training on the content.1 This page makes none on its behalf, so treat any training claim you read elsewhere as unsourced.
How do I verify that a visit really came from YouBot?
You.com concedes that user agent strings can be spoofed and publishes three checks.4 Prefer the signature: requests carry HTTP Message Signatures under Cloudflare's Web Bot Auth, validated against Ed25519 keys at You.com's well-known directory. A valid signature proves possession of the key and survives infrastructure moves. Next is reverse DNS to youbot-{octets-with-hyphens}.search.you.com, so 68.67.112.106 resolves to youbot-68-67-112-106; resolve forward and confirm the same address. The range 68.67.112.0/24 is the quickest filter, but proves only control of that block. The general method is in verifying a crawler.
- Reverse DNS ends in
- *.search.you.com
You.com also documents the address range 68.67.112.0/24 and Web Bot Auth signed requests for YouBot, with no range file to download.
ResearcherTranscribed from You.com documentation · What user-agent string does YouBot send?
Match the token, never the version. A rule pinned to a version stops matching silently the next time the operator ships.
The version in this string changes; check You.com’s page before relying on the exact text.
What does allowing YouBot cost?
Allowing it costs crawl load, which you can slow with crawl-delay, and keeps you in You.com's search results. Blocking it removes you from those results. You.com caches robots.txt for 30 minutes, so a robots.txt change can take up to 30 minutes to reach it.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https:/Should I allow YouBot in my situation?
Concrete cases, and what each one implies for this agent.
YouBot is crawling faster than your origin likes.
Set Crawl-delay for YouBot rather than disallowing it, and allow 30 minutes.
You.com documents honouring crawl-delay directives and adaptive rate limiting, and caches robots.txt for 30 minutes. That makes throttling a real option here, unlike with operators that state they do not support the directive — where a crawl-delay line is a no-op that looks like a fix.
You need to prove, not assume, that a request came from a search crawler.
Validate the HTTP Message Signature against You.com's published key directory.
Reverse DNS and address ranges are inferences from infrastructure control; a signature is proof of key possession. You.com is the only operator in this cluster documenting signed requests, and it is the strongest verification any crawler here offers.
A reverse-DNS check returns a hostname you do not recognise.
Check it against the documented pattern before deciding it is fake.
You.com documents hostnames of the form youbot-{octets-with-hyphens}.search.you.com, where the octets match the connecting address — so 68.67.112.106 appears as youbot-68-67-112-106. A hostname that looks unfamiliar is usually just this pattern; one that does not match it, or does not resolve forward to the same address, is the actual red flag.
What else do people ask?
How long does a robots.txt change take to reach YouBot?
Why is a signed request stronger proof than reverse DNS?
Where does You.com publish YouBot's signing keys?
Where do these facts come from?
5 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
You.com states that YouBot is the web crawler that powers the You.com search engine, automatically discovering and indexing web pages to provide real-time, accurate search results for You.com users.
you.com · retrieved
“YouBot is the web crawler that powers the You.com search engine. It automatically discovers and indexes web pages to provide real-time, accurate search results for You.com users.”
You.com documents YouBot's user-agent string as "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36", noting that X.X.X.X represents the Chrome version number.
you.com · retrieved · changes often, check the source
You.com states that YouBot respects robots.txt directives including user-agent specific rules and crawl-delay settings, implements adaptive crawl rate limiting, and caches robots.txt for 30 minutes so that changes can take up to 30 minutes to take effect.
you.com · retrieved
“YouBot fully respects robots.txt directives, including user-agent specific rules and crawl-delay settings.”
You.com documents three ways to verify YouBot — HTTP Message Signatures under Cloudflare's Web Bot Auth validated against Ed25519 public keys published at its well-known directory, reverse DNS to hostnames of the form youbot-{octets-with-hyphens}.search.you.com with a matching forward lookup, and the originating address range 68.67.112.0/24 — and states that user agent strings can be spoofed.
you.com · retrieved
“Since user agent strings can be spoofed, You.com supports three ways to verify that requests are genuinely from YouBot”
You.com states that YouBot respects standard HTTP status codes, reducing crawl rate on 429 Too Many Requests and temporarily pausing crawling on 503 Service Unavailable, and publishes spider@you.com as the contact for excessive crawl activity.
you.com · retrieved
Want whatever it takes to win the answer?
7 days free · card required