AI crawler access checker

The checker fetches /robots.txt from the origin you give it, parses it into user-agent groups, and evaluates each crawler in the table below against the path you supplied. Every verdict quotes the exact line that produced it, with its line number, so you can go and change that line.

Matching follows the Robots Exclusion Protocol as published in RFC 9309, which is what the major crawlers implement:

• A crawler obeys the group whose User-agent token matches its own name, case-insensitively. If no group names it, it falls back to the User-agent: * group. If there is no wildcard group either, nothing restricts it.
• Within the group that applies, the longest matching path rule wins — not the first one. Disallow: / followed by Allow: /blog/ leaves /blog/ reachable, because the allow rule is longer and more specific.
• On an exact length tie, Allow beats Disallow.
• * matches any run of characters and a trailing $ anchors the end of the path. An empty Disallow: value is an explicit permit-everything, not a block.

This reports what robots.txt says. It is not proof a crawler can reach you. A firewall rule, a bot-management product, a geographic block, a login wall or an aggressive rate limiter can all refuse a request that robots.txt permits — and none of that is visible from the outside. Equally, robots.txt is a request, not an enforcement mechanism: a crawler that ignores it is not stopped by it. The only way to know which crawlers actually fetched your pages is to read your own server logs.

There is no score here, deliberately. Turning seventeen independent allow/block facts into one number out of a hundred would need a weighting we would have to invent, and the invented part would be the part you acted on. Each row states the crawler, the verdict and the line responsible.

Each of these is a product token its operator documents publicly. Tokens nobody documents are not included, because a verdict about a user-agent string no crawler actually sends is worthless. The right column separates the two things people conflate: whether blocking a crawler keeps your pages out of model training, and whether it keeps them out of the answers users see today.

• Everything is blocked by a single Disallow: /. Usually a staging robots.txt that shipped to production. It is the most common and most expensive finding here, and it takes one line to undo.
• The search crawler is blocked but the trainer is not. This is backwards for most sites. GPTBot, ClaudeBot and Google-Extended govern training. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot govern whether you appear in answers now. If you want citations without contributing to training, permit the search crawlers and disallow the training ones — they are separate tokens for exactly this reason.
• Rules written above the first User-agent line. They belong to no group and are ignored by every conforming crawler. The report lists them so you can move them under a group.
• A blanket block with an Allow that is shorter than the block. Disallow: /docs/ with Allow: /d does not open anything: the disallow is longer, so it wins. Make the allow rule more specific than the rule it is meant to override.
• Blocking by user-agent to stop scraping. It does not work — a scraper that ignores robots.txt is unaffected, and the only parties who honour it are the ones you might have wanted. Rate limiting and bot management are the tools for abuse; robots.txt is for stating intent to crawlers that comply.
• No sitemap declared. Not an access failure, but a missed one. Add a Sitemap: line and check the file it points at with the sitemap validator.

Every other thing you do for AI search assumes the page can be fetched. Structured data, heading hierarchy, an llms.txt, a rewritten answer-first opening paragraph — none of it is read by a model that got a 403 or was told not to come. Access is not one factor among many; it is the gate in front of the rest.

It is also the failure that hides best. A blocked crawler does not file a complaint or show up as an error in your analytics. The page simply never appears in an answer, and the absence is indistinguishable from not being good enough to cite. That is why this check is worth running against production on a schedule rather than once, and why the file is worth diffing after any deploy that touches infrastructure.