Skip to content

Our crawler

seoscorebot

seoscorebot is the crawler behind seoscore, an SEO audit service. It loads a site's public pages to work out its SEO score: when the site's owner or one of our customers audits it, and about once a week for the home pages on the public SEO League Table. It doesn't index pages for a search engine, and it doesn't collect content to train AI models.

How it identifies itself

Every request carries this user agent, and only seoscorebot sends it:

seoscorebot/1.0 (+https://seoscore.app/bot)

We never send a browser's user agent or another crawler's name, and we don't try to get past a firewall or a bot check that turns us away. A site that does gets no score, and we say so.

How to check it's really us

Every request seoscorebot sends is signed with Web Bot Auth, the HTTP Message Signatures standard (RFC 9421) for bots. Bot managers that support it check the signature for you. Each request carries three headers:

  • Signature-Agent: "https://seoscore.app"
  • Signature-Input, with the tag web-bot-auth, the covered components "@authority" and "signature-agent", and a one-minute expiry;
  • Signature, made with an Ed25519 key.

Our public keys are in our key directory, https://seoscore.app/.well-known/http-message-signatures-directory. A request that calls itself seoscorebot without a valid signature from one of those keys isn't from us.

What it fetches, and how often

  • For an audit: the page being audited, plus /robots.txt and /llms.txt from the same site, and the site's icon. It reads at most 5 MB of a page, doesn't run JavaScript, doesn't follow links, and never logs in or fills in a form.
  • For the League Table: the home page of each site on the table, one site at a time, about every 7 days. A visit that times out or hits a server error is tried once more about an hour later. A site that turns us away is asked again the next week, not sooner.
  • For the League Table's icons: the site's favicon, at most once every 30 days.
  • When a customer asks: their own site, as often as they run audits or watch it, the outbound links on its home page when they check for broken links, and pages they check with tools such as the AEO and GEO checker or our browser extension.

Page speed is measured by Google PageSpeed Insights, which loads the page from Google's own servers under Google's name, not ours.

How fast it goes

  • Requests to one site are at least 1 second apart, across all our servers.
  • If your robots.txt sets a Crawl-delay for seoscorebot (or for every bot), we wait that long between requests instead, up to 60 seconds.
  • If your site answers 429 Too Many Requests or 503 Service Unavailable, we send nothing more until its Retry-After has passed (a minute when there isn't one), up to an hour.

robots.txt

seoscorebot reads your robots.txt before anything else and follows it (RFC 9309) for every page it fetches: audits our customers run, League Table scans, the checkers and the browser extension. Its rules for User-agent: seoscorebot come first; without them, the rules for User-agent: * apply. If robots.txt answers with a server error, or doesn't answer, we fetch nothing until it does. We read it again at most an hour later.

To keep seoscorebot away from your site, add this to your robots.txt:

User-agent: seoscorebot
Disallow: /

If your robots.txt keeps every bot out but you want your own audits to work, let seoscorebot in by name:

User-agent: seoscorebot
Allow: /

One file is read without asking robots.txt: when the owner of a site connects our WordPress plugin, we fetch the one address the plugin gives us to prove the site is theirs. To take your site off the League Table altogether, ask us to remove it.

Let it past your firewall

If your firewall, CDN or bot protection turns away automated visitors, your audits fail and your site has no score on the League Table. To let seoscorebot in:

  1. If your provider lists seoscorebot among its verified bots, allowing verified bots (or its SEO category) is enough.
  2. Otherwise, find your firewall's custom or allow rules (most CDNs and hosting control panels have them).
  3. Add a rule for requests whose user agent contains seoscorebot and whose Web Bot Auth signature checks out.
  4. Set it to allow the request, or to skip bot checks and challenges for it.

Then run an audit, or wait for the next weekly League scan. Your audit says what went wrong if we still can't get in, such as the HTTP status your site answered with.

Questions

seoscorebot is run by 2MRWLABS LTD, the company behind seoscore. If it is misbehaving on your site, email [email protected] with your domain and what you saw. You can also read how the score works.