AI crawler URL inspector

This matters enough to state twice. Every request this tool makes carries our user-agent, with the crawler we are asking about named inside it, something of the shape Attensira (asking-as GPTBot; not GPTBot). We do not send GPTBot as our user-agent, because sending somebody's crawler token to a stranger's server is a claim to be that company.

The honest consequence: a site that varies its response by user-agent (a bot-management product, a paywall with a crawler allowlist, an edge rule keyed on the token) may answer the real crawler differently than it answered us. What you get here is a real response from your server to a real request, which is far more than robots.txt gives you, and it is not a recording of what OpenAI or Anthropic saw. We cannot see that, we do not intercept anyone's traffic, and nothing on this page should be read as if we did.

The only record of what a crawler genuinely received is the one your own server wrote. That is a question your access logs answer, not this tool.

robots.txt is a statement of permission. It says what a crawler may fetch, and nothing about what it gets when it does. This inspector reads the permission too, at the site root, because that is the one path every site has. Then it fetches the page and adds what the file cannot show: the status code, the redirect chain and the markup the request received.

A page can be allowed by robots.txt and return a 404. It can be allowed and 302 to a login screen. It can return a clean 200 whose body is an empty container waiting for a script. Each of those is invisible to a robots.txt check and each of them ends the same way: nothing to cite. To write the permission side, use the robots.txt generator.

The two percentages are shares of the same denominator, the bytes the server returned, and they do not add up to 100. The remainder is tags, attributes and CSS: neither readable prose nor executable script.

There is no pass mark here, and we are not going to invent one. No operator publishes a threshold for how much server-rendered text a page needs, so any line we drew would be a number we made up, and it would be the number you acted on. What the split tells you is factual and narrow: an agent that fetches your HTML without running a browser has exactly the words in the text column available to it. If that column is nearly empty on a page you want quoted, the words you want quoted are not in the response.

Whether a given crawler executes JavaScript is a separate question, and not one this tool can answer. Operators change that behavior without announcing it, and several documented crawlers fetch HTML only. We measure what was served; what each crawler does with it is the operator's to say, which is why every token in the table links to its own operator documentation.