Technical SEO Audit: What to Check in 2026, Including AI Search
By Irfan Ghaffar, SEO specialist. 4 years in SEO, technical issues solved on 34+ websites, 18+ technical audit clients. Published 8 November 2026, updated 8 November 2026.
A technical SEO audit checks whether Google and AI assistants can reach, read and index the pages that earn you money, then lists what blocks them in order of impact. In 2026 that means two layers: the classic checks (indexing, crawling, redirects, speed, structured data) and the newer ones (AI crawler access, CDN bot rules and the Search generative AI setting in Search Console). Below is the process I use, with real numbers from my own audit work.
Most guides ranking for this topic were written between 2022 and 2024. Several still tell you to use tools Google has retired, or to score a metric Google replaced. This guide is for owners and marketers running clinic, trades and Shopify sites in the UK, USA, Australia and New Zealand who want an audit that ends in fixes, not a long PDF.
What is a technical SEO audit?
It is a structured check of how search engines crawl, render, index and serve your site. It looks at the system underneath your content: status codes, robots rules, canonical tags, internal links, page speed, structured data and how your server answers bots. Content quality and backlinks belong to separate audits, although the findings often overlap.
The goal is simple. Every page you want found should be crawlable, indexable and fast. Every page you don’t want found should stay out of the way.
Google is direct about why this now matters beyond blue links. To be eligible for AI Overviews or AI Mode, a page must be indexed and eligible to show with a snippet (Google Search Central, updated 10 July 2026). If Google can’t index a page, no AI feature built on its index can cite it.
Why does technical SEO matter more in 2026?
Clicks are harder to win, so losing the ones you could win hurts more. Ahrefs found that an AI Overview correlates with a 58% lower click rate for the top ranking page, using 300,000 keywords and Search Console data up to December 2025 (Ahrefs, February 2026).
Page one is also no longer the only way in. In March 2026, Ahrefs found that 37.9% of URLs cited in AI Overviews also appeared in the top 10 for the same query, across 863,000 keyword results and 4 million cited URLs (Ahrefs, March 2026). Pages ranking lower can still be cited, but only if they are indexed and readable.
Two new settings can also switch a site off quietly. Search Console added a Search generative AI control, and Cloudflare changed how its bot rules treat Googlebot. Neither shows up in a standard crawler report, so both now belong in every audit.
What does a technical SEO audit include in 2026?
A full audit covers ten areas: indexing, crawling, status codes, duplicates, speed, structured data, bot access at the edge, AI crawler access, Search Console AI settings and log files. The first six are classic technical SEO. The last four are where most older checklists stop short.
| Area | What I check | Main tools |
|---|---|---|
| Indexing | Page indexing report, noindex tags, the canonical Google chose | Search Console, URL Inspection |
| Crawling | robots.txt, XML sitemaps, crawl depth, orphan pages | Screaming Frog or Sitebulb, Search Console |
| Status codes | 404s, 410s, 5xx errors, redirect chains and loops | Crawler, server logs |
| Duplicates | Parameter URLs, http and https or www versions, Shopify collection paths | Crawler, URL Inspection |
| Speed and experience | LCP, INP and CLS from field data | PageSpeed Insights, Core Web Vitals report |
| Structured data | Errors, and markup that doesn’t match the visible page | Rich Results Test |
| Bot access at the edge | CDN, firewall and security plugin rules | Cloudflare dashboard, server logs |
| AI crawler access | OAI-SearchBot and other assistant crawlers in robots.txt | robots.txt, server logs |
| Search Console AI settings | Search generative AI setting, Generative AI performance report | Search Console |
| Log files | What Googlebot actually requests, and crawl waste | Log file analyser |
How to run a technical SEO audit, step by step
Start by confirming the site can be indexed at all, then work through indexing reports, a full crawl and bot access, then status codes, logs, speed, markup and links. Finish by verifying every fix on the live site. Keep this order, because the early steps catch the problems that cost the most.
1. Confirm the site is allowed into the index at all
Check the homepage and your main service or collection pages for a noindex tag, an X-Robots-Tag header, a password wall or a robots.txt rule that blocks the whole site. On WordPress, also check the “Discourage search engines” box under Settings, then Reading.
These checks sound basic. They are also the fixes with the biggest payoff when something is wrong, because one setting can hide an entire site.
2. Read the Page indexing report in Search Console
Open Indexing, then Pages. Google lists every URL it knows about and the reason any of them are not indexed. Older guides still call this the Coverage report.
Focus on three statuses: crawled but not indexed, discovered but not indexed, and duplicates where Google chose a different canonical than yours. Then run URL Inspection on your top money pages to see which canonical Google selected and when it last crawled them.
3. Crawl the site the way a search engine would
Run a crawler such as Screaming Frog or Sitebulb with JavaScript rendering switched on. Export pages that can and can’t be indexed, status codes, canonical tags, crawl depth and pages with no internal links.
Compare the crawl with your XML sitemap. Every URL in the sitemap should return a 200 status, be indexable and point to itself as canonical. Anything else is noise Google has to sort out for you.
4. Check that bots can get past your CDN and firewall
This is the step older checklists miss. Your robots.txt file can say “allow” while a CDN or security plugin blocks the request before it ever reaches your server.
Cloudflare now sorts AI bots into Search, Agent and Training groups. From 15 September 2026, blocking Training also blocks crawlers that do both search and training, which includes Googlebot, Bingbot and Applebot (Help Net Security, 2 July 2026). New domains also get Training and Agent bots blocked by default on pages that show ads.
If your site runs on Cloudflare, open the AI bot policies and confirm the choice for each group. Then check your server logs to confirm Googlebot gets a 200 response and not a 403.
5. Check AI search crawlers in robots.txt
ChatGPT search uses a crawler called OAI-SearchBot. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, apart from navigational links (OpenAI crawler documentation). Blocking GPTBot is a separate decision, because GPTBot covers model training, not search.
Look for a wildcard Disallow rule, or a rule added by a security plugin, that catches these bots by accident. Other assistants publish their own search crawlers, such as PerplexityBot, and the same check applies to them.
6. Check the Search generative AI setting in Search Console
Search Console has a property setting called Search generative AI. It decides whether your links and content can appear in AI Overviews, AI Mode and the generative features in Discover. Include is the default, and the control took effect on 17 June 2026 (Search Engine Roundtable, June 2026).
Google says a site must be included in this setting to be eligible for its generative AI features (Google Search Central). Google also says excluding a site is not used as a ranking signal in other parts of Search. It is a quick check, and someone with access to your property may have changed it.
7. Fix status codes, redirects and duplicates
Work through 404 pages that still get internal links, redirect chains, redirect loops and 5xx server errors. Point every internal link at the final URL, not at a redirect.
For duplicates, choose one version of each page and make every signal agree: canonical tag, internal links, sitemap and redirects. On Shopify, theme links to the longer collection path for a product, instead of the plain product URL, send Google mixed signals (Amsive).
When a URL is gone for good and has no replacement, a 410 status tells crawlers it is not coming back. That is how one rule removed 111 spam URLs in the log file project in the next step.
8. Read your server logs
A crawler shows what could be crawled. Server logs show what Googlebot actually requested. This is the core of log file analysis, and it is the clearest way to find crawl budget spent on pages that should not exist.
Here is what one of my log file projects found:
| Finding | Result |
|---|---|
| Bot hits analysed | 1,000 |
| URLs | 375 |
| Crawl waste | 38% |
| Spam URLs removed by one 410 rule | 111 |
| Redirects cleaned | 25 |
| Pages set to noindex | 5 |
| Fixes verified live | 47 of 47 |
The 30 day recheck on this project is still pending, so I am not claiming any traffic or ranking gains from it yet. What it does show is how much of a crawler’s attention can go to URLs that have no reason to be crawled.
Google points owners of very large and frequently updated sites to its crawl budget guide (Google Search Central). On smaller sites, logs still earn their place, because they reveal spam or hacked URLs you didn’t know existed, as they did here.
9. Measure Core Web Vitals from field data
Core Web Vitals are LCP for loading, INP for responsiveness and CLS for visual stability. INP replaced First Input Delay on 12 March 2024 (Google Search Central), so any guide still scoring FID is out of date.
Use the Core Web Vitals report in Search Console and the field data at the top of PageSpeed Insights. Lab scores help you debug, but field data is what real visitors experience. Google also notes that good Core Web Vitals scores don’t guarantee good rankings, so fix pages that are clearly slow and then move on.
For mobile, the Mobile-Friendly Test and the Mobile Usability report in Search Console were retired on 1 December 2023 (Search Engine Land, December 2023). Test with Lighthouse and on a real phone instead.
10. Check that structured data matches the page
Run your key templates through the Rich Results Test. Fix errors, and make sure the markup describes what a visitor can actually see on the page.
Two points are worth knowing before you pay for schema work. Since August 2023, FAQ rich results only show for well known, authoritative government and health sites, and HowTo rich results were later dropped altogether (Google Search Central, August 2023). Google also says structured data is not required for its AI features, although it still helps with rich result eligibility.
11. Check HTTPS, international versions and images
Every page should load over HTTPS, and every http or www variation should redirect to one version in a single step. Google confirmed HTTPS as a ranking signal back in 2014 (Google Search Central). Look for mixed content too, where a secure page loads an image or script over http.
If you sell into more than one of the UK, USA, Australia and New Zealand with separate versions, check the hreflang tags. Each version should return a 200 status, list the others, and point back to itself. Then check which version Google actually shows in each country.
For images, look for oversized files that slow your main pages, broken images and missing alt text. Ahrefs’ audit guide recommends serving images at the smallest workable size and in a format such as WebP (Ahrefs).
12. Fix internal links and orphan pages
Every page you want to rank needs links from related pages, and your money pages need the most. A page with no internal links depends on the sitemap or outside links to be found at all.
Check crawl depth for your service, location and collection pages. If a key page sits several clicks from the homepage, link to it from the pages that already get traffic.
13. Prioritise, fix and verify
Sort every finding by two questions. How many money pages does it touch, and how hard is it to fix?
Sitewide indexing blocks come first. Cosmetic warnings come last, or not at all.
Then verify each fix on the live site. In the log file project above, all 47 fixes were verified live. A fix that sits in a ticket queue has not fixed anything yet.
How does the audit differ for clinics, trades and Shopify stores?
The checks stay the same, but the order changes. Clinics need every location page indexed, trades need fast service pages on mobile, and stores need crawl waste under control. Where the money comes from decides where I look first.
| Site type | What I check first | Why |
|---|---|---|
| Clinics and dental practices | Indexing and canonicals for every location and treatment page | Each location needs its own indexed page for searches near it |
| Trades and home services | Indexing of service pages, mobile speed, duplicate suburb pages | Most enquiries come from phones, and thin copies of one page compete with each other |
| Shopify and ecommerce stores | Filter and sort URLs, collection path links, crawl waste in logs | Large catalogs can spend crawl activity on near duplicate URLs |
For a clinic or trades business, the audit and the Google Business Profile belong together. Your profile sends people to your website, so the page it links to has to be indexed, fast on a phone and clear about the service.
For one Sydney clinic with three locations, local SEO and profile work produced 454 tracked enquiries: 278 calls from Google Maps and 176 website form enquiries (Sydney clinic case study). See how the two fit together in my local SEO services.
For stores, technical problems scale with the catalog. Filters, variants and leftover app code can create far more URLs than you have products. My ecommerce SEO services page covers the Shopify side in detail.
Which outdated checks can you drop?
Drop the Mobile-Friendly Test, FID scores, chasing FAQ and HowTo rich results, and adding llms.txt for Google. Google has retired or ignores each of these, yet popular guides still recommend them. Dropping them frees time for the checks that do matter.
| Old advice | What is true now | Source |
|---|---|---|
| Run the Mobile-Friendly Test | Retired on 1 December 2023; use Lighthouse | Search Engine Land, 2023 |
| Score First Input Delay | INP replaced FID on 12 March 2024 | Google, 2024 |
| Add FAQ and HowTo markup for rich results | FAQ rich results are limited to government and health sites; HowTo rich results are gone | Google, 2023 |
| Add an llms.txt file to show in Google’s AI features | Google Search ignores llms.txt files | Google, 2026 |
| Break content into tiny chunks for AI | Google says there is no need to chunk content | Google, 2026 |
| You must rank in the top 10 to be cited | 37.9% of AI Overview citations were also top 10 results | Ahrefs, 2026 |
How do you measure whether the audit worked?
Track the pages you fixed, not the whole site. In Search Console, watch the Page indexing counts for the templates you changed, the Crawl stats report, and impressions and clicks for the fixed URLs.
For AI features, use the Generative AI performance report. It shows impressions in AI Overviews, AI Mode and generative features in Discover, broken down by page, country and date (Google Search Central, June 2026). Google finished rolling it out to all sites worldwide by the end of August 2026 (Search Engine Journal, 2026).
Set a recheck date before you start. Google needs to recrawl the affected pages, and a result measured too early can mislead you. That is why the log file project above still has its 30 day recheck pending.
When the work does land, the numbers can move a long way. One site I worked on had been written off as dead. After recovery work it reached 224,000 clicks in 28 days, with 3.6 million impressions, a 6.2% click through rate and an average position of 5.4, measured in Google Search Console (case studies).
Which tools do you need?
Search Console, PageSpeed Insights and the Rich Results Test cover most checks for free. Add one crawler such as Screaming Frog or Sitebulb, plus access to your server logs and CDN dashboard. Paid tools mainly save time on larger sites.
| Tool | Cost | What it is for |
|---|---|---|
| Google Search Console | Free | Indexing, Core Web Vitals, crawl stats, Generative AI report, AI setting |
| PageSpeed Insights and Lighthouse | Free | Field and lab speed data, mobile checks |
| Rich Results Test | Free | Structured data errors |
| Screaming Frog or Sitebulb | Free version or trial, then paid | Full site crawl with JavaScript rendering |
| Server logs and a log analyser | Depends on your host and tool | What bots really request |
| Cloudflare dashboard, if you use it | Included in your plan | AI bot policies for Search, Agent and Training |
How much does a technical SEO audit cost, and how long does it take?
Semrush’s guide, last updated in September 2024, puts a one time professional audit at roughly $300 to $2,000, and puts audits of large or complex sites at $2,000 to over $10,000 (Semrush, 2024). Doing it yourself mainly costs your time plus tool fees.
The same guide gives these timeframes:
| Site size | Audit | Fixes |
|---|---|---|
| Under 500 pages | About one day | Three to seven days |
| 500 to 2,000 pages | One to three days | One to three weeks |
| Over 2,000 pages | About one week | One to two months |
My own pricing works differently. The free audit ends with a fixed quote, so you know the cost before you commit to anything.
Frequently asked questions
Get a second pair of eyes on your site
If you’d like me to look at your site, I run free audits for clinics, trades and Shopify stores in the UK, USA, Australia and New Zealand. You get a list of fixes sorted by impact and a fixed quote, with no obligation. For local businesses, I review your Google Business Profile alongside the website, because both decide whether a search turns into a call.