Let our crawler in
How seoscorebot, our crawler, identifies itself and follows robots.txt, and how to let it past your firewall or bot protection so your audits work.
- Updated
On this page
Every audit, page audit, League scan and checker loads your pages through our crawler, seoscorebot. If your site keeps it out, the audit fails with the reason and your site shows Blocks checks. This page shows how to let it in. Our crawler's page has the technical detail for whoever looks after your firewall.
How it identifies itself
Every request carries the same user agent:
seoscorebot/1.0 (+https://seoscore.app/bot)
Its requests are also signed with Web Bot Auth, so bot protection that supports it can check a request really comes from us. We never pretend to be a browser or another crawler, and we never try to get past a bot check.
robots.txt
We read your robots.txt before anything else and follow it. If it keeps seoscorebot, or every bot (User-agent: *), out of the page, we don't load the page: the audit fails, gives its credits back and says robots.txt asked us not to read it.
To let us in while other bots stay out, add a group for us by name:
User-agent: seoscorebot
Allow: /
If your robots.txt answers with a server error, or doesn't answer, we treat the whole site as closed until it does. We read it again within an hour.
How fast it goes
Requests to one site are at least 1 second apart. A Crawl-delay in your robots.txt makes us wait longer, up to 60 seconds, so an audit of a site with a long delay takes a little longer. If your site answers 429 or 503, we wait for its Retry-After before asking again.
Let it past your firewall
A firewall, CDN or bot protection that turns away automated visitors stops audits too. The audit names the answer we got, such as HTTP 403, or says we were shown a bot check ("Just a moment...", a captcha). To let us in:
- If your provider lists
seoscorebotamong its verified bots, allow verified bots, or its SEO category. - Otherwise add a rule that allows requests whose user agent contains
seoscorebot, or skips bot checks for them. - Press Run new audit on the site's page to try again.
A site that asks automated visitors to pay for access (HTTP 402) can't be audited until it lets us in.
The League Table
The League scans the home pages on the table about once a week, with the same crawler and the same rules. A site that keeps us out has no score or position, and we ask again the next week. See Sites we can't check.
Try it on your own site. The Free plan needs no card.