Skip to content

What is technical SEO? Elements, examples and a checklist

Technical SEO is the work that lets search engines and AI crawlers find, render, understand and index your pages. Here is what it covers, how to spot the common problems and what to fix first.

The seoscore team 12 min read

On this page
  1. Technical SEO vs on-page SEO vs off-page SEO
  2. How search engines crawl, render and index a page
  3. Crawlability: robots.txt, status codes and crawl paths
  4. Indexability: noindex, canonicals and duplicate content
  5. Site architecture, internal links and XML sitemaps
  6. Redirects and broken links
  7. JavaScript rendering
  8. Speed, Core Web Vitals and mobile-first indexing
  9. HTTPS and security basics
  10. Structured data
  11. Technical SEO for AI search
  12. Examples of technical SEO problems and their fixes
  13. How to run a technical SEO check on your site
  14. A technical SEO checklist, in priority order
  15. Keep it fixed with seoscore
  16. Next steps

Technical SEO is the work that lets search engines and AI crawlers find, render, understand and index your pages. It deals with how a site is built and served rather than what it says: robots.txt, status codes, canonicals, redirects, sitemaps, JavaScript, speed, mobile, HTTPS and structured data. It comes first because a page that can't be crawled or indexed can't rank, however good its content or links.

People shorten it to tech SEO, and most guides turn it into a list of 50 items with no order. This one follows the path a page takes into a search engine, shows what breaks at each step with a real example, and ends with a checklist ordered by impact.

Technical SEO vs on-page SEO vs off-page SEO

Technical SEO On-page SEO Off-page SEO
What it covers How pages are crawled, rendered, indexed and served What is on the page: content, titles, headings Signals from other sites: links, mentions, reviews
Who usually does it Developer or technical SEO Writer or marketer Marketer or PR
Example tasks Fix robots.txt, redirects, canonicals, speed Rewrite a title, answer the query better Earn links, get listed in directories
Sign it's the problem The page isn't indexed, or the wrong URL is Indexed but ranks poorly for its query A good page ranks below weaker pages with more links

The lines blur. Internal links are technical (they decide what gets crawled) and on-page (the anchor text describes the target). Page speed depends on code and hosting, but also on the images an editor uploads. Structured data is code that describes content. A quick test: if the page isn't in Google at all, or the wrong URL is, start with technical SEO. If it's indexed but ranks poorly, look at content and links first.

How search engines crawl, render and index a page

Google's How Search works guide describes three stages: crawling, indexing and serving. In practice there are five steps, and every element of technical SEO serves one of them.

  1. Discovery. The engine finds a URL through links, sitemaps or redirects.
  2. Crawling. A crawler such as Googlebot downloads the page, if robots.txt allows it and the server responds.
  3. Rendering. Google runs the page's JavaScript to see the final content. According to Google's JavaScript SEO basics, a page "may stay on this queue for a few seconds, but it can take longer than that."
  4. Indexing. Google analyses the rendered page, decides whether it duplicates another URL, picks a canonical and stores it. In Google's words, "Indexing isn't guaranteed; not every page that Google processes will be indexed."
  5. Serving. When someone searches, Google chooses pages from the index to show.

In November 2025 Google moved its crawler documentation to a separate crawling site, because the same crawlers feed Shopping, News, Gemini and other products. Older guides still link to the old addresses.

Crawlability: robots.txt, status codes and crawl paths

robots.txt is a text file at the root of your domain that tells crawlers which paths they may fetch. It controls crawling, not indexing: a blocked URL can still be indexed from links pointing to it, just without its content.

Rules match by prefix, which is how sections get blocked by accident. This was meant to hide the basket:

User-agent: *
Disallow: /shop

It also blocks /shop/, /shop-guides/ and /shopping-tips/. Be specific instead:

User-agent: *
Disallow: /shop/basket/

Status codes tell crawlers what happened:

  • 200: the page loaded.
  • 301 or 308: moved permanently.
  • 404 or 410: gone. Fine for pages that really are gone, a problem when your own links point to them.
  • 5xx: server error. Persistent errors make Google slow its crawling.
  • Soft 404: a page that says "not found" but returns 200. Google treats it as an error and keeps recrawling it.

Orphan pages, which no internal link points to, can only be found through a sitemap or other sites, and crawlers treat them as unimportant.

Crawl budget, the number of URLs Google will crawl on your site in a given time, only matters for big sites. Google's crawl budget guide is aimed at sites with over a million unique pages that change weekly, sites with over 10,000 pages that change daily, and sites with many URLs stuck in "Discovered – currently not indexed". If your new pages are crawled the day you publish them, Google says you don't need it.

Indexability: noindex, canonicals and duplicate content

A crawled page still needs permission and a clear identity to be indexed.

noindex tells search engines not to index a page. It can sit in the HTML, or in an HTTP header for files such as PDFs:

<meta name="robots" content="noindex">
X-Robots-Tag: noindex

Google has to crawl a page to see its noindex, so don't block the same page in robots.txt.

Canonical tags name the preferred version when several URLs show the same content:

<link rel="canonical" href="https://example.com/shoes/trail-runner/">

Google treats a canonical as a strong hint, not an order. Its guide to duplicate URLs explains that it also weighs redirects, sitemaps, internal links and HTTPS, and may pick another URL when those signals conflict, for example when the canonical names one URL and every internal link points to another.

Common sources of duplicates:

  • tracking and sorting parameters such as ?utm_source= and ?sort=price
  • http and https, or www and non-www, versions that both load
  • URLs with and without a trailing slash
  • paginated category pages: give each page its own canonical rather than pointing them all at page one

Reading the Page indexing report

In Google Search Console, Google's free tool for site owners, Indexing > Pages lists why URLs aren't indexed. Google's help page explains every status; these are the ones to read first:

Status What it usually means
Excluded by 'noindex' tag Intended, or left over from staging
Blocked by robots.txt Check the rule is deliberate
Not found (404) Fix the links to it, or redirect it
Soft 404 Empty or thin pages returning 200
Duplicate, Google chose different canonical than user Your signals conflict
Crawled – currently not indexed Google saw it and chose not to index it
Discovered – currently not indexed Known but not yet crawled

Not every excluded URL is a problem. Redirected and canonicalised URLs are meant to stay out.

Architecture is how your pages link together. In a flat structure, important pages sit a few clicks from the home page, so they are found and recrawled sooner than a product buried six clicks deep. A common rule of thumb is three clicks for anything that matters.

Internal links should use descriptive anchor text ("trail running shoes", not "click here") and be ordinary <a href> links, not JavaScript click handlers a crawler can't follow.

An XML sitemap lists the URLs you want indexed. From Google's sitemap guide:

  • each sitemap holds up to 50,000 URLs or 50MB uncompressed; larger sites split it and submit a sitemap index file
  • Google ignores <priority> and <changefreq>
  • it uses <lastmod> only when it is consistently accurate, so change it for significant updates only

Include only canonical, indexable URLs that return 200. Submit the sitemap in Search Console and add a Sitemap: https://example.com/sitemap.xml line to robots.txt. Google no longer accepts sitemap pings.

Use a 301 (or 308) when a page has moved for good and a 302 (or 307) when the move is temporary. A 302 on a permanent move can keep the old URL in the index for longer.

A redirect chain is several hops in a row, such as /old to /new to /newer. Each hop slows the page and crawlers only follow a limited number. A loop (A to B and back to A) never loads at all. Point every redirect straight at the final URL, and update internal links so they skip the redirect entirely.

Migrations are where these problems multiply. Map every old URL to its closest new one, redirect each directly, update canonicals, sitemaps and internal links to the new URLs, and keep the redirects in place for as long as possible.

JavaScript rendering

Content that only appears after JavaScript runs reaches Google only after rendering, which can be delayed. Two details from Google's JavaScript documentation matter: if the original HTML carries a noindex, Google may skip rendering, so JavaScript can't remove it later; and since late 2025 Google says pages returning anything other than 200 might not be rendered.

Server-side rendering (the server sends finished HTML) is safer than client-side rendering (the browser builds the page). To see what a crawler gets, compare View Source, the raw HTML, with the rendered HTML in Search Console's URL Inspection tool, or check from the command line:

curl -s https://example.com/product/ | grep "Waterproof upper"

If the text isn't in the raw HTML, many AI crawlers won't see it. A Vercel and MERJ analysis from December 2024 found that OpenAI's, Anthropic's and Perplexity's crawlers download JavaScript files but don't run them.

Speed, Core Web Vitals and mobile-first indexing

Core Web Vitals are Google's three measures of loading, responsiveness and visual stability, taken at the 75th percentile of real visits (web.dev):

Metric Measures Good Poor
LCP (Largest Contentful Paint) Loading 2.5s or less over 4.0s
INP (Interaction to Next Paint) Responsiveness 200ms or less over 500ms
CLS (Cumulative Layout Shift) Visual stability 0.1 or less over 0.25

INP replaced First Input Delay in March 2024. Google's page experience documentation says its ranking systems use Core Web Vitals. Be wary of articles claiming the thresholds tightened in 2026: web.dev still lists the figures above.

Google now crawls and indexes the mobile version of pages (mobile-first indexing). The usual technical culprits are oversized hero images, render-blocking scripts, images without set dimensions, heavy third-party tags, and mobile templates that drop content, links or structured data the desktop version has.

HTTPS and security basics

Google announced HTTPS as a lightweight ranking signal in 2014. Today it is a baseline: browsers label plain HTTP pages as not secure. Redirect every http URL to https with a single 301, use https in canonicals and sitemaps, and fix mixed content, where an https page loads an image or script over http.

Audits also flag missing security headers such as Strict-Transport-Security, Content-Security-Policy, X-Content-Type-Options and Referrer-Policy. These protect visitors more than rankings, but they are cheap to add at the server.

Structured data

Structured data is code, usually JSON-LD using the Schema.org vocabulary, that states facts about a page: this is a product, this is its price, this organisation publishes the site.

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Ltd",
  "url": "https://example.com",
  "logo": "https://example.com/logo.png",
  "sameAs": ["https://www.linkedin.com/company/example"]
}

Rich results keep shrinking. FAQ and HowTo rich results were cut back sharply in 2023, and in June 2025 Google announced it would phase out seven more: Book Actions, Course Info, ClaimReview, Estimated Salary, Learning Video, Special Announcement and Vehicle Listing. Check the Search Gallery before adding markup to win a rich result. Organization, Product, Article and Breadcrumb markup is still worth having because it tells engines which entities a page is about. Validate it with the Rich Results Test and the Schema Markup Validator.

Answer engines such as ChatGPT, Perplexity and Google's AI Overviews add a step to technical SEO: making sure AI crawlers can read your pages and quote them. This is the technical side of AEO (answer engine optimisation).

Decide which AI crawlers to allow. The main robots.txt user agents:

User agent Company Used for
GPTBot OpenAI Model training
OAI-SearchBot OpenAI ChatGPT search
ClaudeBot Anthropic Model training
PerplexityBot Perplexity Perplexity search
Google-Extended Google Gemini training and grounding (a token, not a crawler)

Google-Extended doesn't affect Google Search, AI Overviews included; those follow Googlebot and snippet controls such as nosnippet and max-snippet. To stay out of training but remain citable in ChatGPT search:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Check each vendor's own page, such as OpenAI's crawler documentation, because names change.

Serve content without JavaScript. As above, many AI crawlers don't render.

Consider llms.txt. llms.txt is a proposed Markdown file that points AI tools to your key pages. Google staff have said Google doesn't use it, so treat it as low-cost and optional. The free llms.txt generator creates and checks one.

Mark up your entities with Organization and sameAs, as shown above.

The free AEO and GEO checker runs these checks on any page: AI crawler access in robots.txt, content without JavaScript, snippet rules, llms.txt, entity markup and more. Whether assistants then name your brand is GEO, covered in the GEO guide.

Examples of technical SEO problems and their fixes

  • Staging noindex left on. After a relaunch, "Excluded by 'noindex' tag" climbs in Search Console. Fix: remove the tag or untick WordPress's "Discourage search engines" setting, then request indexing for key pages.
  • Canonical pointing to the wrong URL. A template bug sets every product's canonical to the home page, and products drop out. Fix: make each canonical point to the page itself.
  • Redirect chain. http://example.com/blog goes to https, then to www, then adds a trailing slash: three hops. Fix: one 301 straight to https://www.example.com/blog/.
  • JavaScript-only product description. View Source shows an empty <div id="description">. Fix: render the description on the server.
  • Blocked CSS folder. Disallow: /assets/ stops Google loading stylesheets, so it renders a broken page. Fix: allow CSS and JavaScript files.
  • Missing sitemap. New products on a large shop are found slowly through deep category links. Fix: generate a sitemap of canonical URLs and submit it.

How to run a technical SEO check on your site

1. Start with Google Search Console

Read the Page indexing report, the Core Web Vitals report and the Sitemaps report, and run URL Inspection on your five most important pages.

2. Run an automated audit

Search Console reports what Google saw; an audit tests the site directly. A free seoscore audit checks crawlability, indexability, redirects, speed, mobile, security and structured data, and scores the site out of 100. The Technical area is worth 17 points, scored beside Performance (16), Mobile (12), Security (9) and AEO (15), which covers AI crawler access. Every check and weight is published on the methodology page.

3. Prioritise by impact

Don't fix 200 warnings in the order a tool lists them. Fix whatever removes pages from the index first, then what confuses indexing, then speed and polish. seoscore ranks every fix by the points it adds, which gives you that order.

A technical SEO checklist, in priority order

Blocks indexing entirely

  • No stray noindex on pages you want indexed
  • robots.txt doesn't block important sections, CSS or JavaScript
  • Key pages return 200; no recurring 5xx errors
  • HTTPS everywhere, with http redirecting in one 301

Confuses or delays indexing

  • Canonicals point to the page itself or a live 200 URL
  • Main content is in the server-sent HTML
  • Sitemap submitted, with only canonical 200 URLs
  • No redirect chains or loops; internal links point to final URLs
  • No broken internal links or orphan pages

Affects rankings and experience

  • Core Web Vitals pass at the 75th percentile
  • Mobile pages carry the same content, links and structured data
  • No mixed content; security headers set

Helps engines and AI understand you

  • Organization, Product or Article, and Breadcrumb markup validated
  • AI crawler rules in robots.txt are deliberate
  • llms.txt, if you want one

Recheck after any deploy that touches templates, robots.txt or redirects, run a full audit monthly, and crawl the whole site before and after a migration.

Keep it fixed with seoscore

Technical problems usually come back with a deploy, not a decision. Automatic audits re-audit a site weekly or daily and email the team when the score drops or the site can't be reached; they are on Starter and up, with a free trial. The SEO agent works through your audits each week, finds internal and broken links, and proposes fixes for titles, headings, structured data, canonicals and internal links that change nothing until you approve them. If you deploy through GitHub, the GitHub Actions guide shows how to audit after each deploy and fail the build on a drop.

Next steps

  1. Open Search Console's Page indexing report and read the top three reasons pages aren't indexed.
  2. Check robots.txt for prefix rules that block more than you meant.
  3. Run an audit and fix the top three items by impact.
  4. View Source on your main templates and confirm the key content is there.
  5. Decide which AI crawlers you allow, and write it down in robots.txt.

Questions people ask

Is technical SEO hard to learn?

The basics are learnable by any site owner: reading status codes, robots.txt and the Search Console reports. JavaScript rendering, crawl management on very large sites and site migrations take more experience, and that is where specialists earn their keep.

Do I need a developer for technical SEO?

Not always. Many fixes live in your CMS or a plugin, such as noindex settings, sitemaps and redirects. Problems in templates, server rendering, HTTP headers or hosting usually need someone with access to the code or server.

Is technical SEO more important than content?

It works more like a threshold. Until pages can be crawled and indexed, content can't rank at all. Once that threshold is met, extra technical polish usually brings less than better content and links.

Does technical SEO matter for small sites?

Yes, but the list is shorter. Crawl budget is irrelevant for a small site, yet a single noindex tag or robots.txt mistake can remove the whole site from search. Check indexing, HTTPS, mobile and speed, and you have covered most of the risk.

How long does Google take to notice a technical fix?

It depends on how often Google recrawls the page. Google says busy news homepages may be recrawled every few minutes, while pages that rarely change might be revisited about monthly. For important pages, request indexing with the URL Inspection tool in Search Console.

Sources

  1. In-Depth Guide to How Google Search Works developers.google.com
  2. developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics developers.google.com
  3. Things to Know about Google's Web Crawling developers.google.com
  4. Crawl Budget Management developers.google.com
  5. developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls developers.google.com
  6. support.google.com/webmasters/answer/7440203 support.google.com
  7. Build and Submit a Sitemap developers.google.com
  8. How the Core Web Vitals metrics thresholds were defined  |  Articles  |  web.dev web.dev
  9. developers.google.com/search/docs/appearance/page-experience developers.google.com
  10. developers.google.com/search/blog/2023/10/mobile-first-is-here developers.google.com
  11. developers.google.com/search/blog/2014/08/https-as-ranking-signal developers.google.com
  12. Simplifying the search results page  |  Google Search Central Blog  |  Google for Developers developers.google.com
  13. developers.google.com/search/docs/appearance/structured-data/search-gallery developers.google.com
  14. search.google.com/test/rich-results search.google.com
  15. platform.openai.com/docs/bots platform.openai.com
  16. llmstxt.org llmstxt.org

seoscore

What does your site score?

Run a free audit: a score out of 100 and every fix ranked by the points it adds.

Get my free score
More on Technical SEO

Technical SEO

Technical SEO agencies: what they do and how to choose one

A client-side guide to hiring a technical SEO agency: what the work covers, when you need it, how to tell real analysis from a rebadged crawler report, and how to hold the agency to measurable results.

9 min read