Crawl errors are one of those technical issues that look minor in a report but compound over time. A 404 that nobody redirected. A redirect chain that was never cleaned up after a migration. A page that got a noindex tag applied by mistake and has been invisible to Google for six months. These are the things that suppress rankings quietly, there is no sudden drop, just a persistent ceiling on how well the site performs.
When I do a technical audit, crawl issues are usually in the first section I report because they are foundational. You can have excellent content and strong backlinks, but if Google cannot reliably crawl and index your site, none of that other work reaches its full potential.
How to find crawl errors
Google Search Console is your first port of call. The Coverage report (now called Indexing in newer versions of GSC) shows you exactly what Google has found and what problems it has encountered. The categories to focus on are “Excluded” and “Error”, not just the error count but the specific reasons. “Crawled, currently not indexed”, “Discovered, currently not indexed”, and “Page with redirect” tell different stories and need different fixes.
Like this stuff? Get it weekly.
The Wednesday Roundup. SEO and AI search news, tactics and what is changing this week. No fluff.
Subscribe →Screaming Frog gives you the crawler’s perspective from outside GSC. Run it against your domain with JavaScript rendering enabled and you will see what a bot encounters when it visits your site sequentially, status codes, redirect chains, meta robots tags, canonical errors, and broken internal links. The combination of GSC data and a Screaming Frog crawl gives you a complete picture: what Google has already found and what it encounters when it tries to find more.
WORK WITH JAMES
Get a straight answer on your SEO.
No account managers. No generic reports. Direct access to someone who has been doing this for 25 years.
404 errors: when to fix and when to leave
Not all 404 errors need to be fixed. A 404 that has never had any inbound links and receives no traffic is not a problem, it is just a missing page. Trying to fix every 404 on a large site is a distraction from the issues that actually affect rankings.
The 404s that matter are: pages that have external backlinks pointing to them (you are losing link equity on every request), pages that previously ranked and drove organic traffic, and pages that are linked internally from other live pages (which creates a poor crawl signal and wastes crawl budget).
The fix is a 301 redirect to the most relevant live equivalent. If the content has moved, redirect to the new URL. If the content no longer exists, redirect to the nearest relevant category or section page rather than the homepage, a homepage redirect for a deleted product page tells Google almost nothing useful. If there is genuinely no relevant destination, a 410 (Gone) is more correct than a 301 to an irrelevant page, and Google handles it appropriately.
Redirect chains: the silent link equity drain
A redirect chain occurs when URL A redirects to URL B which redirects to URL C. This is extremely common on sites that have been through migrations, rebrands, or URL restructures, each migration adds a new layer to existing chains. Google follows redirect chains, but each hop reduces the link equity passed to the final destination. A chain of three redirects can reduce the value reaching the end URL by 15-20%.
The fix is to update every redirect in the chain to point directly at the final destination. This sounds simple but requires mapping every URL that points to the intermediate redirect rather than the end destination. Screaming Frog’s “Redirect Chains” report does this automatically, it shows you every chain, every URL in it, and the final destination.
Internal links that point to redirected URLs are a related issue. Every time a page on your own site links to a URL that redirects, it is sending Google through an unnecessary hop. Internal links should always point to canonical, live URLs, not to URLs that will redirect. Fixing these is quick and eliminates friction in Google’s crawl of your site.
Soft 404s: the error that does not look like an error
A soft 404 is a page that returns a 200 OK status code but contains no meaningful content, effectively an empty or near-empty page that Google treats as a 404 equivalent. These appear in GSC as “Soft 404” errors and they waste crawl budget on pages that provide no value while potentially diluting the overall quality signals of your site.
Common sources: filtered search pages on ecommerce sites that return no results, tag and category pages with only one or two posts, location pages generated programmatically with identical content, and user account or basket pages that require login but are somehow accessible to crawlers. The fix depends on the cause, noindex the page, add a canonical to the primary version, or add enough meaningful content to justify indexation.
Crawl budget: when it matters and when it does not
Crawl budget, the number of pages Googlebot will crawl on your site in a given period, is not a concern for most small and medium-sized sites. If your site has a few hundred to a few thousand pages and none of them have the issues described above, Google will crawl them all. Crawl budget becomes a real consideration on larger sites, particularly ecommerce sites with faceted navigation, pagination, and large catalogues that generate thousands of URL variations.
If you have a large site and GSC shows that a significant proportion of your pages are in “Discovered, currently not indexed”, crawl budget may be the issue. The solution is to eliminate low-value URLs from crawlable space, using robots.txt to block parameter-driven duplicate URLs, implementing rel=”nofollow” on internal links to faceted filter pages, and consolidating thin content pages into fewer, more substantial ones.
Frequently asked questions
The most common causes are pages that have been deleted without a redirect, URLs that have changed without updating internal links, server errors returning 5xx status codes, URLs blocked by robots.txt, and soft 404 pages that return a 200 status but display no real content. Coverage errors in Search Console tell you which category each issue falls into.
A hard 404 is a page that does not exist and correctly returns a 404 HTTP status code. A soft 404 is a page that returns a 200 OK status but contains no real content, typically an empty product page, a broken dynamic URL, or a page that says “no results found.” Soft 404s are a crawl budget problem because Google wastes time crawling pages that have no value.
If the page no longer exists and has no equivalent, let the 404 stand, it is the correct response. If the content has moved, set up a 301 redirect from the old URL to the new one. If the page should exist but is returning a 404 due to a broken link or slug error, restore the page or correct the URL. Never serve a 200 status on a page that does not exist.
Crawl budget is the number of pages Googlebot will crawl on your site within a given time period. For small sites under a few hundred pages it is rarely a concern. For large sites with thousands of pages, thin content, or significant duplicate content, Googlebot may not crawl all important pages regularly. Fixing crawl errors and removing low-value pages improves how Googlebot allocates its crawl on larger sites.
Googlebot needs to recrawl the affected pages before any ranking improvement is possible. For pages already in the index, recrawling typically takes days to weeks. Newly fixed redirects can be picked up within days on well-crawled sites. Meaningful ranking improvement after fixing crawl issues depends on whether those issues were genuinely suppressing rankings, which varies by site.
TESTIMONIALS
