What is technical SEO? Elements, examples and a checklist
Technical SEO is the work that lets search engines and AI crawlers find, render, understand and index your pages. Here is what it covers, how to spot the common problems and what to fix first.
The seoscore team 12 min read
On this page
- Technical SEO vs on-page SEO vs off-page SEO
- How search engines crawl, render and index a page
- Crawlability: robots.txt, status codes and crawl paths
- Indexability: noindex, canonicals and duplicate content
- Site architecture, internal links and XML sitemaps
- Redirects and broken links
- JavaScript rendering
- Speed, Core Web Vitals and mobile-first indexing
- HTTPS and security basics
- Structured data
- Technical SEO for AI search
- Examples of technical SEO problems and their fixes
- How to run a technical SEO check on your site
- A technical SEO checklist, in priority order
- Keep it fixed with seoscore
- Next steps
Technical SEO is the work that lets search engines and AI crawlers find, render, understand and index your pages. It deals with how a site is built and served rather than what it says: robots.txt, status codes, canonicals, redirects, sitemaps, JavaScript, speed, mobile, HTTPS and structured data. It comes first because a page that can't be crawled or indexed can't rank, however good its content or links.
People shorten it to tech SEO, and most guides turn it into a list of 50 items with no order. This one follows the path a page takes into a search engine, shows what breaks at each step with a real example, and ends with a checklist ordered by impact.
Technical SEO vs on-page SEO vs off-page SEO
| Technical SEO | On-page SEO | Off-page SEO | |
|---|---|---|---|
| What it covers | How pages are crawled, rendered, indexed and served | What is on the page: content, titles, headings | Signals from other sites: links, mentions, reviews |
| Who usually does it | Developer or technical SEO | Writer or marketer | Marketer or PR |
| Example tasks | Fix robots.txt, redirects, canonicals, speed | Rewrite a title, answer the query better | Earn links, get listed in directories |
| Sign it's the problem | The page isn't indexed, or the wrong URL is | Indexed but ranks poorly for its query | A good page ranks below weaker pages with more links |
The lines blur. Internal links are technical (they decide what gets crawled) and on-page (the anchor text describes the target). Page speed depends on code and hosting, but also on the images an editor uploads. Structured data is code that describes content. A quick test: if the page isn't in Google at all, or the wrong URL is, start with technical SEO. If it's indexed but ranks poorly, look at content and links first.
How search engines crawl, render and index a page
Google's How Search works guide describes three stages: crawling, indexing and serving. In practice there are five steps, and every element of technical SEO serves one of them.
- Discovery. The engine finds a URL through links, sitemaps or redirects.
- Crawling. A crawler such as Googlebot downloads the page, if robots.txt allows it and the server responds.
- Rendering. Google runs the page's JavaScript to see the final content. According to Google's JavaScript SEO basics, a page "may stay on this queue for a few seconds, but it can take longer than that."
- Indexing. Google analyses the rendered page, decides whether it duplicates another URL, picks a canonical and stores it. In Google's words, "Indexing isn't guaranteed; not every page that Google processes will be indexed."
- Serving. When someone searches, Google chooses pages from the index to show.
In November 2025 Google moved its crawler documentation to a separate crawling site, because the same crawlers feed Shopping, News, Gemini and other products. Older guides still link to the old addresses.
Crawlability: robots.txt, status codes and crawl paths
robots.txt is a text file at the root of your domain that tells crawlers which paths they may fetch. It controls crawling, not indexing: a blocked URL can still be indexed from links pointing to it, just without its content.
Rules match by prefix, which is how sections get blocked by accident. This was meant to hide the basket:
User-agent: *
Disallow: /shop
It also blocks /shop/, /shop-guides/ and /shopping-tips/. Be specific instead:
User-agent: *
Disallow: /shop/basket/
Status codes tell crawlers what happened:
- 200: the page loaded.
- 301 or 308: moved permanently.
- 404 or 410: gone. Fine for pages that really are gone, a problem when your own links point to them.
- 5xx: server error. Persistent errors make Google slow its crawling.
- Soft 404: a page that says "not found" but returns 200. Google treats it as an error and keeps recrawling it.
Orphan pages, which no internal link points to, can only be found through a sitemap or other sites, and crawlers treat them as unimportant.
Crawl budget, the number of URLs Google will crawl on your site in a given time, only matters for big sites. Google's crawl budget guide is aimed at sites with over a million unique pages that change weekly, sites with over 10,000 pages that change daily, and sites with many URLs stuck in "Discovered – currently not indexed". If your new pages are crawled the day you publish them, Google says you don't need it.
Indexability: noindex, canonicals and duplicate content
A crawled page still needs permission and a clear identity to be indexed.
noindex tells search engines not to index a page. It can sit in the HTML, or in an HTTP header for files such as PDFs:
<meta name="robots" content="noindex">
X-Robots-Tag: noindex
Google has to crawl a page to see its noindex, so don't block the same page in robots.txt.
Canonical tags name the preferred version when several URLs show the same content:
<link rel="canonical" href="https://example.com/shoes/trail-runner/">
Google treats a canonical as a strong hint, not an order. Its guide to duplicate URLs explains that it also weighs redirects, sitemaps, internal links and HTTPS, and may pick another URL when those signals conflict, for example when the canonical names one URL and every internal link points to another.
Common sources of duplicates:
- tracking and sorting parameters such as
?utm_source=and?sort=price - http and https, or www and non-www, versions that both load
- URLs with and without a trailing slash
- paginated category pages: give each page its own canonical rather than pointing them all at page one
Reading the Page indexing report
In Google Search Console, Google's free tool for site owners, Indexing > Pages lists why URLs aren't indexed. Google's help page explains every status; these are the ones to read first:
| Status | What it usually means |
|---|---|
| Excluded by 'noindex' tag | Intended, or left over from staging |
| Blocked by robots.txt | Check the rule is deliberate |
| Not found (404) | Fix the links to it, or redirect it |
| Soft 404 | Empty or thin pages returning 200 |
| Duplicate, Google chose different canonical than user | Your signals conflict |
| Crawled – currently not indexed | Google saw it and chose not to index it |
| Discovered – currently not indexed | Known but not yet crawled |
Not every excluded URL is a problem. Redirected and canonicalised URLs are meant to stay out.
Site architecture, internal links and XML sitemaps
Architecture is how your pages link together. In a flat structure, important pages sit a few clicks from the home page, so they are found and recrawled sooner than a product buried six clicks deep. A common rule of thumb is three clicks for anything that matters.
Internal links should use descriptive anchor text ("trail running shoes", not "click here") and be ordinary <a href> links, not JavaScript click handlers a crawler can't follow.
An XML sitemap lists the URLs you want indexed. From Google's sitemap guide:
- each sitemap holds up to 50,000 URLs or 50MB uncompressed; larger sites split it and submit a sitemap index file
- Google ignores
<priority>and<changefreq> - it uses
<lastmod>only when it is consistently accurate, so change it for significant updates only
Include only canonical, indexable URLs that return 200. Submit the sitemap in Search Console and add a Sitemap: https://example.com/sitemap.xml line to robots.txt. Google no longer accepts sitemap pings.
Redirects and broken links
Use a 301 (or 308) when a page has moved for good and a 302 (or 307) when the move is temporary. A 302 on a permanent move can keep the old URL in the index for longer.
A redirect chain is several hops in a row, such as /old to /new to /newer. Each hop slows the page and crawlers only follow a limited number. A loop (A to B and back to A) never loads at all. Point every redirect straight at the final URL, and update internal links so they skip the redirect entirely.
Migrations are where these problems multiply. Map every old URL to its closest new one, redirect each directly, update canonicals, sitemaps and internal links to the new URLs, and keep the redirects in place for as long as possible.
JavaScript rendering
Content that only appears after JavaScript runs reaches Google only after rendering, which can be delayed. Two details from Google's JavaScript documentation matter: if the original HTML carries a noindex, Google may skip rendering, so JavaScript can't remove it later; and since late 2025 Google says pages returning anything other than 200 might not be rendered.
Server-side rendering (the server sends finished HTML) is safer than client-side rendering (the browser builds the page). To see what a crawler gets, compare View Source, the raw HTML, with the rendered HTML in Search Console's URL Inspection tool, or check from the command line:
curl -s https://example.com/product/ | grep "Waterproof upper"
If the text isn't in the raw HTML, many AI crawlers won't see it. A Vercel and MERJ analysis from December 2024 found that OpenAI's, Anthropic's and Perplexity's crawlers download JavaScript files but don't run them.
Speed, Core Web Vitals and mobile-first indexing
Core Web Vitals are Google's three measures of loading, responsiveness and visual stability, taken at the 75th percentile of real visits (web.dev):
| Metric | Measures | Good | Poor |
|---|---|---|---|
| LCP (Largest Contentful Paint) | Loading | 2.5s or less | over 4.0s |
| INP (Interaction to Next Paint) | Responsiveness | 200ms or less | over 500ms |
| CLS (Cumulative Layout Shift) | Visual stability | 0.1 or less | over 0.25 |
INP replaced First Input Delay in March 2024. Google's page experience documentation says its ranking systems use Core Web Vitals. Be wary of articles claiming the thresholds tightened in 2026: web.dev still lists the figures above.
Google now crawls and indexes the mobile version of pages (mobile-first indexing). The usual technical culprits are oversized hero images, render-blocking scripts, images without set dimensions, heavy third-party tags, and mobile templates that drop content, links or structured data the desktop version has.
HTTPS and security basics
Google announced HTTPS as a lightweight ranking signal in 2014. Today it is a baseline: browsers label plain HTTP pages as not secure. Redirect every http URL to https with a single 301, use https in canonicals and sitemaps, and fix mixed content, where an https page loads an image or script over http.
Audits also flag missing security headers such as Strict-Transport-Security, Content-Security-Policy, X-Content-Type-Options and Referrer-Policy. These protect visitors more than rankings, but they are cheap to add at the server.
Structured data
Structured data is code, usually JSON-LD using the Schema.org vocabulary, that states facts about a page: this is a product, this is its price, this organisation publishes the site.
{
"@context": "https://schema.org",
"@type": "Organization",
"name": "Example Ltd",
"url": "https://example.com",
"logo": "https://example.com/logo.png",
"sameAs": ["https://www.linkedin.com/company/example"]
}
Rich results keep shrinking. FAQ and HowTo rich results were cut back sharply in 2023, and in June 2025 Google announced it would phase out seven more: Book Actions, Course Info, ClaimReview, Estimated Salary, Learning Video, Special Announcement and Vehicle Listing. Check the Search Gallery before adding markup to win a rich result. Organization, Product, Article and Breadcrumb markup is still worth having because it tells engines which entities a page is about. Validate it with the Rich Results Test and the Schema Markup Validator.
Technical SEO for AI search
Answer engines such as ChatGPT, Perplexity and Google's AI Overviews add a step to technical SEO: making sure AI crawlers can read your pages and quote them. This is the technical side of AEO (answer engine optimisation).
Decide which AI crawlers to allow. The main robots.txt user agents:
| User agent | Company | Used for |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT search |
| ClaudeBot | Anthropic | Model training |
| PerplexityBot | Perplexity | Perplexity search |
| Google-Extended | Gemini training and grounding (a token, not a crawler) |
Google-Extended doesn't affect Google Search, AI Overviews included; those follow Googlebot and snippet controls such as nosnippet and max-snippet. To stay out of training but remain citable in ChatGPT search:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Check each vendor's own page, such as OpenAI's crawler documentation, because names change.
Serve content without JavaScript. As above, many AI crawlers don't render.
Consider llms.txt. llms.txt is a proposed Markdown file that points AI tools to your key pages. Google staff have said Google doesn't use it, so treat it as low-cost and optional. The free llms.txt generator creates and checks one.
Mark up your entities with Organization and sameAs, as shown above.
The free AEO and GEO checker runs these checks on any page: AI crawler access in robots.txt, content without JavaScript, snippet rules, llms.txt, entity markup and more. Whether assistants then name your brand is GEO, covered in the GEO guide.
Examples of technical SEO problems and their fixes
- Staging noindex left on. After a relaunch, "Excluded by 'noindex' tag" climbs in Search Console. Fix: remove the tag or untick WordPress's "Discourage search engines" setting, then request indexing for key pages.
- Canonical pointing to the wrong URL. A template bug sets every product's canonical to the home page, and products drop out. Fix: make each canonical point to the page itself.
- Redirect chain.
http://example.com/bloggoes to https, then to www, then adds a trailing slash: three hops. Fix: one 301 straight tohttps://www.example.com/blog/. - JavaScript-only product description. View Source shows an empty
<div id="description">. Fix: render the description on the server. - Blocked CSS folder.
Disallow: /assets/stops Google loading stylesheets, so it renders a broken page. Fix: allow CSS and JavaScript files. - Missing sitemap. New products on a large shop are found slowly through deep category links. Fix: generate a sitemap of canonical URLs and submit it.
How to run a technical SEO check on your site
1. Start with Google Search Console
Read the Page indexing report, the Core Web Vitals report and the Sitemaps report, and run URL Inspection on your five most important pages.
2. Run an automated audit
Search Console reports what Google saw; an audit tests the site directly. A free seoscore audit checks crawlability, indexability, redirects, speed, mobile, security and structured data, and scores the site out of 100. The Technical area is worth 17 points, scored beside Performance (16), Mobile (12), Security (9) and AEO (15), which covers AI crawler access. Every check and weight is published on the methodology page.
3. Prioritise by impact
Don't fix 200 warnings in the order a tool lists them. Fix whatever removes pages from the index first, then what confuses indexing, then speed and polish. seoscore ranks every fix by the points it adds, which gives you that order.
A technical SEO checklist, in priority order
Blocks indexing entirely
- No stray noindex on pages you want indexed
- robots.txt doesn't block important sections, CSS or JavaScript
- Key pages return 200; no recurring 5xx errors
- HTTPS everywhere, with http redirecting in one 301
Confuses or delays indexing
- Canonicals point to the page itself or a live 200 URL
- Main content is in the server-sent HTML
- Sitemap submitted, with only canonical 200 URLs
- No redirect chains or loops; internal links point to final URLs
- No broken internal links or orphan pages
Affects rankings and experience
- Core Web Vitals pass at the 75th percentile
- Mobile pages carry the same content, links and structured data
- No mixed content; security headers set
Helps engines and AI understand you
- Organization, Product or Article, and Breadcrumb markup validated
- AI crawler rules in robots.txt are deliberate
- llms.txt, if you want one
Recheck after any deploy that touches templates, robots.txt or redirects, run a full audit monthly, and crawl the whole site before and after a migration.
Keep it fixed with seoscore
Technical problems usually come back with a deploy, not a decision. Automatic audits re-audit a site weekly or daily and email the team when the score drops or the site can't be reached; they are on Starter and up, with a free trial. The SEO agent works through your audits each week, finds internal and broken links, and proposes fixes for titles, headings, structured data, canonicals and internal links that change nothing until you approve them. If you deploy through GitHub, the GitHub Actions guide shows how to audit after each deploy and fail the build on a drop.
Next steps
- Open Search Console's Page indexing report and read the top three reasons pages aren't indexed.
- Check robots.txt for prefix rules that block more than you meant.
- Run an audit and fix the top three items by impact.
- View Source on your main templates and confirm the key content is there.
- Decide which AI crawlers you allow, and write it down in robots.txt.
Questions people ask
Is technical SEO hard to learn?
The basics are learnable by any site owner: reading status codes, robots.txt and the Search Console reports. JavaScript rendering, crawl management on very large sites and site migrations take more experience, and that is where specialists earn their keep.
Do I need a developer for technical SEO?
Not always. Many fixes live in your CMS or a plugin, such as noindex settings, sitemaps and redirects. Problems in templates, server rendering, HTTP headers or hosting usually need someone with access to the code or server.
Is technical SEO more important than content?
It works more like a threshold. Until pages can be crawled and indexed, content can't rank at all. Once that threshold is met, extra technical polish usually brings less than better content and links.
Does technical SEO matter for small sites?
Yes, but the list is shorter. Crawl budget is irrelevant for a small site, yet a single noindex tag or robots.txt mistake can remove the whole site from search. Check indexing, HTTPS, mobile and speed, and you have covered most of the risk.
How long does Google take to notice a technical fix?
It depends on how often Google recrawls the page. Google says busy news homepages may be recrawled every few minutes, while pages that rarely change might be revisited about monthly. For important pages, request indexing with the URL Inspection tool in Search Console.
Sources
- In-Depth Guide to How Google Search Works developers.google.com
- developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics developers.google.com
- Things to Know about Google's Web Crawling developers.google.com
- Crawl Budget Management developers.google.com
- developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls developers.google.com
- support.google.com/webmasters/answer/7440203 support.google.com
- Build and Submit a Sitemap developers.google.com
- How the Core Web Vitals metrics thresholds were defined | Articles | web.dev web.dev
- developers.google.com/search/docs/appearance/page-experience developers.google.com
- developers.google.com/search/blog/2023/10/mobile-first-is-here developers.google.com
- developers.google.com/search/blog/2014/08/https-as-ranking-signal developers.google.com
- Simplifying the search results page | Google Search Central Blog | Google for Developers developers.google.com
- developers.google.com/search/docs/appearance/structured-data/search-gallery developers.google.com
- search.google.com/test/rich-results search.google.com
- platform.openai.com/docs/bots platform.openai.com
- llmstxt.org llmstxt.org
seoscore
What does your site score?
Run a free audit: a score out of 100 and every fix ranked by the points it adds.
Get my free score