Technical SEO best practices are the rules for building a site that search engines can crawl, render, index and serve without guesswork. In 2026 the essentials are: one canonical URL per page, crawlable <a href> links, content present in the HTML, honest status codes, robots.txt for crawling and noindex for indexing, a clean sitemap, lean and fast templates, mobile parity, HTTPS, structured data that matches the page, hreflang for international sites, and a deliberate policy for AI crawlers.
Each rule below comes with the reason and a link to the Google documentation behind it. The last section lists six popular "best practices" that Google says do nothing.
Key takeaways
- Most technical SEO comes down to removing ambiguity: one URL per page, one clear signal for indexing, one version of the content for every device.
- robots.txt controls crawling, not indexing. A URL blocked in robots.txt can still be indexed, and Google can't see a noindex tag on a page it isn't allowed to fetch.
- Googlebot reads only the first 2 MB of an HTML file (Google documentation, updated February 2026). Put the main content early and keep inline scripts out of the way.
- Google's July 2026 guide to generative AI features says llms.txt files, special AI markup and content 'chunking' are not needed for Google Search.
- Blocking OAI-SearchBot or PerplexityBot removes you from those AI search answers; blocking GPTBot or ClaudeBot only opts you out of model training.
Best practices vs a checklist
A checklist tells you what to verify on an existing site; our technical SEO audit checklist has 40 checks with pass thresholds. Best practices are the decisions you make when building or rebuilding, so the checklist comes back clean. If you work on a React, Vue or Next.js product site, the technical SEO for SaaS guide applies these rules to app subdomains, docs and pricing pages.
The 12 technical SEO best practices
1. Give every page exactly one canonical URL
The same content reachable at several URLs (http and https, www and non-www, with and without a trailing slash, with tracking parameters) splits signals and wastes crawling. Pick one version, 301-redirect every other version to it in a single hop, add a self-referencing rel="canonical", and make sure internal links, the sitemap and hreflang tags all use that exact URL.
Canonical tags are a strong hint, not a command. Google picks the canonical from several signals, and its canonicalization troubleshooting guide (updated July 2026) warns that changes take time to be re-evaluated. Check the "Google-selected canonical" in URL Inspection rather than assuming your tag won.
2. Link every important page with crawlable links, close to the homepage
Google follows <a> elements with an href attribute. It does not reliably follow links built from buttons, onclick handlers or JavaScript routes that never render an href, per its link best practices. Keep money pages within a few clicks of the homepage, use breadcrumbs on deep sections, and link related pages to each other with descriptive anchors. Our guide on why internal links matter covers the anchor-text side.
3. Put the content in the HTML, not only in JavaScript
Google renders JavaScript, but rendering is a second step that can be delayed, and other crawlers (including many AI crawlers) fetch HTML without running scripts at all. Server-side rendering or static generation is the safe default for any page that needs to rank. Google's documentation describes dynamic rendering (serving bots a different pre-rendered version) as a workaround, not a recommended long-term solution.
Verify with URL Inspection → Test live URL → View tested page → HTML: if the main text and links aren't in there, Google doesn't have them.
4. Use status codes that tell the truth
- 200 for real pages only. A "not found" page that returns 200 becomes a soft 404 in Search Console.
- 301 or 308 for permanent moves, 302 or 307 for temporary ones. Google follows up to 10 hops, but keep it to one.
- 404 or 410 for removed pages. Neither hurts the rest of your site.
- 503 (with a Retry-After header) for planned maintenance, so Google retries instead of dropping pages.
- 429, 500 or 503 to slow Googlebot down. Google has asked site owners not to use 403 or 404 for rate limiting, because those codes mean "gone" or "forbidden", not "slow down".
Login-protected areas should return 401 or 403; our 401 vs 403 guide explains which.
5. Use robots.txt for crawling and noindex for indexing
robots.txt tells crawlers which URLs not to fetch. It does not remove pages from the index: a blocked URL can still be indexed (without its content) if other pages link to it. To keep a page out of Google, let Google crawl it and add <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header. Use robots.txt for crawl waste: internal search results, infinite filter combinations, cart and session URLs.
6. Treat the sitemap as the list of URLs you want indexed
An XML sitemap should contain only canonical, indexable URLs that return 200, with a lastmod that changes only when the content meaningfully changes. Google's sitemap documentation says it ignores priority and changefreq and uses lastmod only when it is consistently accurate. Limits are 50,000 URLs or 50 MB uncompressed per file; split larger sites with a sitemap index. Step-by-step instructions are in how to make a sitemap.
7. Keep HTML lean and templates fast
Two different limits apply. The hard one: Google's Googlebot documentation says Googlebot fetches only the first 2 MB of an HTML file (uncompressed) and indexes what it got. The soft one: Core Web Vitals, measured on real Chrome users at the 75th percentile.
| Metric | Good | What usually fixes it |
|---|---|---|
| Largest Contentful Paint | ≤ 2.5 s | Faster server response, a preloaded and properly sized hero image that is never lazy-loaded |
| Interaction to Next Paint | ≤ 200 ms | Less JavaScript on the main thread, fewer third-party scripts |
| Cumulative Layout Shift | ≤ 0.1 | Width and height on images and embeds, reserved space for banners |
Fix performance per template, not per URL: one change to the product template fixes every product page. Google's page experience documentation is explicit that great content can still rank with mediocre scores, so speed work should follow indexing work, not replace it.
8. Serve the same content on mobile and desktop
Google crawls and indexes with its smartphone crawler. Whatever is missing from your mobile version (text in collapsed tabs that never loads, links dropped from the mobile menu, structured data only in the desktop template) is missing from Google. Responsive design makes parity automatic; separate mobile templates are where it breaks.
9. Serve everything over HTTPS
HTTPS is a confirmed but lightweight ranking signal, and browsers label HTTP pages as "Not secure". Redirect all HTTP URLs to HTTPS with 301s, fix mixed content, and consider HSTS once you are sure every subdomain supports HTTPS. Our SSL certificate explainer covers exactly what HTTPS does and doesn't fix.
10. Add structured data that matches what users see
Use JSON-LD for the types that describe your pages (Organization, Article, Product, BreadcrumbList, LocalBusiness) and only mark up content that is visible on the page. Expect rich results to change: Google stopped showing FAQ rich results on May 7, 2026 and removed several other rich result types in 2025 and 2026 (see its Search Central changelog). Markup still helps Google understand entities; just don't build a strategy on one SERP feature.
11. Use hreflang if you serve several languages or regions
Each language or regional version should list every alternate, including itself, with valid language-region codes, plus an x-default for the fallback page. The annotations must be reciprocal: if the French page points to the German one, the German page must point back. Our SEO localization guide covers URL structure and translation choices.
12. Decide which AI crawlers you allow, on purpose
AI companies now separate crawlers that collect training data from crawlers that fetch pages for live answers. Blocking the wrong one by accident is common, because many sites copied a "block all AI bots" robots.txt snippet in 2023. Here is what each one does, according to each company's documentation:
AI crawlers and what blocking them does (October 2026)
| OAI-SearchBot | OpenAI | Indexes sites for ChatGPT search | You drop out of ChatGPT search answers | ✓ |
| GPTBot | OpenAI | Collects data that may train OpenAI models | Your content is excluded from training only | ✓ |
| ChatGPT-User | OpenAI | Opens a page when a user asks ChatGPT about it | Rules may not apply, since a user asked | ✕ |
| PerplexityBot | Perplexity | Indexes sites for Perplexity search results | You drop out of Perplexity results | ✓ |
| Perplexity-User | Perplexity | Fetches a page to answer a user's question | Generally ignored by this fetcher | ✕ |
| Claude-SearchBot | Anthropic | Indexes sites to improve Claude's search results | Less visibility in Claude's search answers | ✓ |
| ClaudeBot | Anthropic | Collects training data for Claude models | Your content is excluded from future training | ✓ |
| Claude-User | Anthropic | Fetches pages when a user asks Claude | Less visibility in user-directed search | ✓ |
| Google-Extended | A robots.txt token for Gemini training and grounding in Gemini apps | No effect on Google Search inclusion or ranking | ✓ | |
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode | You leave Google Search entirely | ✓ |
Click a column header to sort.
Sources: OpenAI, Perplexity, Anthropic and Google. Google states that Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are part of Search and are fed by Googlebot, so blocking Google-Extended won't take you out of them. More on that trade-off in what is Google-Extended.
A sensible default for most businesses: allow the search and user-triggered crawlers, then decide on training crawlers as a business question. Opening the door doesn't guarantee citations; Outrank's AI visibility tracking shows whether ChatGPT, Perplexity and Google's AI features actually mention you afterwards.
Six technical SEO myths to drop
| Myth | What Google says | Source |
|---|---|---|
| You need an llms.txt file to show up in AI search | Google Search ignores llms.txt; it neither helps nor hurts | AI features guide, July 2026 |
| AI Overviews need special schema or "chunked" content | Structured data isn't required for generative AI features, and there's no special markup; no need to break content into small pieces | Same guide |
| Every site must manage crawl budget | The crawl budget guide is written for sites with about 1 million+ pages, or 10,000+ pages that change daily | Crawl budget guide |
| Sitemap priority tells Google which pages matter | priority and changefreq are ignored | Sitemap docs |
| Blocking a URL in robots.txt removes it from Google | A blocked URL can still be indexed; use noindex instead | robots.txt intro |
| Return 403 or 404 to slow Googlebot down | Use 429, 500 or 503; 403 and 404 can get URLs dropped | Search Central blog |
The pattern: Google's systems reward a clean, crawlable, fast site with useful content, and ignore most of the special files and tricks that circulate on social media. If you want to see how AI systems read a page in practice, our explainer on how AI reads your website shows the fetch-and-parse step.
How to keep it that way
Best practices decay with every release. Three habits keep them in place:
- Add SEO checks to your release process. A pre-deploy test that fails when robots.txt contains
Disallow: /, when a template outputsnoindex, or when the canonical points to a staging host catches the most expensive mistakes. - Watch Search Console weekly. Indexing → Pages and Settings → Crawl stats show regressions within days.
- Run the full technical SEO audit every quarter and around every migration.
FAQ
What are technical SEO best practices?
Technical SEO best practices are the rules that let search engines crawl, render and index a site reliably: one canonical URL per page, crawlable links, content in the HTML, accurate status codes, robots.txt for crawling and noindex for indexing, a clean XML sitemap, fast templates, mobile parity, HTTPS, accurate structured data, hreflang for international sites and deliberate AI crawler rules.
What is the most important part of technical SEO?
Indexing. If Google can't crawl a page, can't see its content after rendering, or chooses a different canonical, nothing else matters. Fix crawling and indexing first, then speed and structured data. Search Console's Pages report is the best single place to see whether you have an indexing problem.
Does technical SEO matter for AI search?
Yes. Google's AI Overviews and AI Mode draw on pages that are indexed and eligible for a snippet, and ChatGPT, Perplexity and Claude use their own crawlers that must be allowed in robots.txt. Most AI crawlers read raw HTML without running JavaScript, so server-rendered content matters even more for them than for Google.
Do I need an llms.txt file?
Not for Google. Google's 2026 guide to generative AI features says Google Search ignores llms.txt files and that they won't help or hurt rankings. Some other tools may read them, so keeping one is harmless, but it isn't a substitute for crawlable, server-rendered pages.
How often should I review technical SEO?
Check Search Console's Pages and Crawl stats reports weekly, run a full technical audit every quarter, and audit before and after every migration, redesign or CMS change. Adding automated checks for noindex, robots.txt and canonicals to your deployment pipeline prevents most regressions.
