Bots
Screaming Frog SEO Spider in your logs
Screaming Frog SEO Spider is a desktop SEO crawler a person runs from their own machine. Seeing its token means someone is auditing your site, not Screaming Frog the company. It obeys robots.txt by default, but its user can switch that off or change the user agent. It feeds no AI engine, so blocking is rarely worth it.
7 days free · card required
0AI-view requests from Screaming Frog SEO Spider · 30 days
filed as conventional traffic (SEO tools, uptime checks, link unfurlers)
| Agent | AI requests, 30 days | attensira.com robots.txt |
|---|---|---|
| Screaming Frog SEO Spider (this page) | not counted | Wildcard |
| AhrefsBot | not counted | Blocked |
| AhrefsSiteAudit | 0 | Wildcard |
| DataForSeoBot | 0 | Wildcard |
| dotbot | not counted | Wildcard |
| Google-InspectionTool | 0 | Parent's rule |
| MJ12bot | not counted | Wildcard |
| rogerbot | 0 | Wildcard |
| SemrushBot | not counted | Blocked |
None of the 9 had a request filed in the window. "Not counted" is off the board's AI view, not a zero.
Analyst · reads the AI traffic boardAI traffic board, read over the Attensira MCP; it files Screaming Frog SEO Spider as conventional traffic, off its AI view · · n=143,601 AI requests
Karl-Gustav Kallasmaa, Founder & CEOLast updated How do I allow or block Screaming Frog SEO Spider in robots.txt?
Paste this into the robots.txt at the root of your domain. Beside it, the group attensira.com serves Screaming Frog SEO Spider today.
# The token is a phrase with spaces. Copy it exactly.# Without a group of its own, the Spider follows your# Googlebot group.User-agent: Screaming Frog SEO SpiderAllow: / # To refuse audits of your site instead:# User-agent: Screaming Frog SEO Spider# Disallow: /# No group names Screaming Frog SEO Spider, so it reads# the wildcard group.User-agent: *Allow: /Disallow: /api/We do not name Screaming Frog SEO Spider. It falls through to the wildcard group, which allows everything except /api/.
Read the whole file, or write yours with the robots.txt generator.
Who runs Screaming Frog SEO Spider, and what is it for?
Screaming Frog runs it, and we file it under product operations. Runs because a person started an audit of your site from their own machine. The crawl goes into their audit report and feeds no search or AI engine.
From Screaming Frog’s documentation, read .
Which robots.txt group does the Screaming Frog SEO Spider obey?
The token is the phrase Screaming Frog SEO Spider, spaces included. Most tokens are one word, and a group written as User-agent: ScreamingFrog matches nothing.
Screaming Frog gives a group under that phrase as the way to stop the Spider crawling your site.2Without one, its FAQ says the Spider checks robots.txt the way Google does and follows the directives for Googlebot, or for all user agents.4That differs from most crawlers: if your file has a Googlebot group and no Screaming Frog group, an audit sees what you allow or disallow for Googlebot. And the user-agent setting decides which directives the tool follows,1 so if the person running it switches to a Googlebot preset, your Screaming Frog group is no longer the one being read.
Our robots.txt generator writes the group with the spaces in the right places.
How do I verify that a visit really came from the Screaming Frog SEO Spider?
You cannot. The software runs on its user's own machine, so there is no crawling infrastructure to publish a range file or a reverse DNS suffix for.5 The token is also configurable, with presets that impersonate Googlebot and Bingbot, so its absence proves as little as its presence: a crawl under a Chrome user agent from a residential address may be this tool. Rogerbot and SemrushBot are unverifiable because their operators publish no ranges; here no range could exist. "Is this really Screaming Frog?" has an organisational answer, not a DNS one: ask who is running an audit. The general methods are in verifying a crawler.
What can my infrastructure actually control?
Rate. The tool crawls as fast as its user's connection and settings allow, and the user can turn off robots.txt compliance or change the user agent from a dropdown.3 Behaviour is the one thing they cannot reconfigure away, so a rate limit at the edge is the control that holds when a Disallow does not.
Screaming Frog publishes no reverse-DNS suffix and no IP range file for Screaming Frog SEO Spider, so a visit can be matched only by its user agent.
ResearcherTranscribed from Screaming Frog documentation · What user-agent string does Screaming Frog SEO Spider send?
Match the token, never the version. A rule pinned to a version stops matching silently the next time the operator ships.
Should I allowlist it when my server returns 403?
Screaming Frog's fix for 403, 429 and 503 responses is to allowlist its user agent at the server or CDN.6 That creates a bypass keyed on a string anyone can send, for a client you cannot verify. If a consultant asks for it, pair the token with their source address and give the entry an owner and an end date. An allowlist entry nobody owns tends to outlive the engagement it was made for.
What does allowing Screaming Frog SEO Spider cost?
Allowing it costs the load of one person's audit crawl. Blocking it costs no search or AI visibility, but its user can switch off robots.txt or change the user agent, so a block mostly inconveniences honest auditors.
Screaming Frog SEO Spider (the default user agent; configurable by the person running the software)Agents filed beside Screaming Frog SEO Spider
Should I allow Screaming Frog SEO Spider in my situation?
Concrete cases, and what each one implies for this agent.
A crawl claiming to be the SEO Spider is coming from an address you do not recognise and you want to know whether to trust it.
Find out internally who is running it. There is no technical check available.
The requests originate from whatever connection the person running the software is on, so no published range or reverse DNS suffix can exist. Unlike an operator-run crawler, the answer to "is this real" is an organisational question rather than a DNS lookup.
A consultant reports that your site cannot be crawled and asks you to allowlist the tool at the CDN.
Grant it narrowly and temporarily, ideally alongside an IP restriction you agree with them.
Screaming Frog's own troubleshooting advice for 403, 429 and 503 responses is to add the "Screaming Frog SEO Spider" user-agent to an allowlist in the server or CDN security settings. That works, and it also creates a permanent bypass keyed on a string anyone can send. Scope it to the engagement.
You disallowed the SEO Spider and somebody crawled the site anyway.
Accept that robots.txt was never going to settle this, and rate-limit instead.
Screaming Frog documents an ignore robots.txt setting and states that using it, and changing the user agent, is the responsibility of the user. Against a tool with those switches, robots.txt expresses a preference to a cooperative operator; it is not an access control.
What else do people ask?
What is the Screaming Frog SEO Spider?
Does it obey robots.txt?
Who is crawling me when I see this token?
Where do these facts come from?
6 claims on this page, each with the page it was read from; last checked . The numbers in the text point here.
Screaming Frog documents the SEO Spider as a crawler whose requests are made by the person running the application, with configurable user agent and robots.txt behaviour, used to crawl a site for SEO analysis.
screamingfrog.co.uk · retrieved
“The user-agent configuration allows you to switch the user-agent of the HTTP requests made by the SEO Spider and which robots.txt directives are followed.”
Screaming Frog states that by default the SEO Spider makes requests using its own Screaming Frog SEO Spider user-agent string, and that its user agent is "Screaming Frog SEO Spider" for the purposes of a robots.txt rule.
screamingfrog.co.uk · retrieved
“Its user agent is 'Screaming Frog SEO Spider' so you can include the following in your robots.txt if you wish the Spider not to crawl your site”
Screaming Frog states that by default the SEO Spider will obey robots.txt protocol and is set to "Respect robots.txt", and that there is an option to ignore robots.txt and change user-agent which is down to the responsibility of the user entirely.
screamingfrog.co.uk · retrieved
“By default the SEO Spider will obey robots.txt protocol and is set to 'Respect robots.txt'.”
Screaming Frog's FAQ states that the SEO Spider checks robots.txt in the same way as Google, checking the robots.txt of the (sub) domain and following directives specifically for Googlebot, or for all user-agents.
screamingfrog.co.uk · retrieved
“follow directives specifically any for Googlebot, or for all user-agents”
Screaming Frog documents a custom user-agent setting that lets the person running the SEO Spider specify any user agent, including presets that impersonate Googlebot and Bingbot.
screamingfrog.co.uk · retrieved
“it has inbuilt preset user-agents for Googlebot, Bingbot, various browsers and more”
Screaming Frog's troubleshooting guidance for 403, 429 and 503 responses is to add the "Screaming Frog SEO Spider" user-agent to an allowlist in the server, hosting platform or CDN security settings.
screamingfrog.co.uk · retrieved
Want whatever it takes to win the answer?
7 days free · card required