SEO Term Definition
Crawl Errors
Crawl errors occur when search engine bots attempt to access a URL on your site but encounter a problem — such as a 404 Not Found, 5xx server error, or redirect loop — that prevents them from successfully fetching the page.
Reviewed by Alexander Yarovenko · Updated: 2026-09-22
What are Crawl Errors?
Crawl errors are problems encountered by search engine crawlers (Googlebot) when attempting to access pages on your website. They prevent the crawler from successfully fetching and indexing the affected pages. Unresolved crawl errors waste crawl budget, can leave important pages unindexed, and signal technical health problems to Google.
Common Crawl Error Types
| Error type | HTTP status | Meaning | Priority |
|---|---|---|---|
| Not Found | 404 | Page doesn't exist | High — if pages have external links or traffic |
| Server Error | 500–599 | Server failed to respond | Critical — immediate attention needed |
| Redirect Error | — | Redirect loop or chain too long | High |
| Soft 404 | 200 (but empty/irrelevant content) | Page returns 200 but has no useful content | Medium |
| Blocked by robots.txt | — | Crawler blocked from accessing URL | Medium — if blocking important pages |
| Unauthorised | 401/403 | Crawler denied access | High — if important pages unintentionally blocked |
How to Find and Fix Crawl Errors
- Find errors: GSC → Pages → "Not indexed" section; also run a Screaming Frog crawl to identify all HTTP error responses
- Fix 404 errors:
- If the page moved → 301 redirect to the new URL
- If the page no longer exists and has no equivalent → return a proper 410 Gone response
- If still relevant → restore the page
- Fix 5xx errors: investigate server logs; check hosting resources; fix application errors
- Fix redirect loops: map all redirects; ensure none create circular paths
- Fix soft 404s: add meaningful content, or 301 redirect to a more relevant page, or 404/410 if the page should not exist
Technical decision
| Correct use | Classify errors by cause and value: a deliberate 404 differs from a broken internal link, a redirect loop or a temporary 5xx. |
|---|---|
| Validation | Reproduce the response, locate every internal source, fix the underlying template or server issue, and verify the final URL after deployment. |