๐
FREE
+50 XP
Crawl & Indexation Audit
๐ท๏ธ
Screaming Frog finishes the crawl
"14 pages returning 404. 3 important pages blocked in robots.txt. 7 pages have redirect chains with 3+ hops. A 30-minute crawl audit โ and there's a month of work already."
๐ Crawl audit โ checking how search robots crawl and index the site. Goal: ensure important pages are accessible and unnecessary ones are blocked.
Key Crawl Audit Checks
- Pages with 4xx/5xx errors โ broken or inaccessible pages
- Redirect chains and loops โ chains slow down crawling
- Noindex on important pages โ a critical error that's easy to make accidentally
- Pages in robots.txt Disallow โ check that CSS/JS Google needs isn't blocked
- Orphan pages โ pages with no internal links, invisible to Googlebot
- Duplicates (canonical) โ URLs with parameters, www/non-www, http/https
HTTP Status Codes in Crawling
| Code | Meaning | Action |
|---|---|---|
| 200 | All OK | Normal |
| 301 | Permanent redirect | Update internal links |
| 404 | Page not found | Restore or redirect |
| 503 | Server unavailable | Urgent โ Google may deindex |
๐ฏ Tip: set up 404 monitoring in GSC โ Indexing โ Pages. New 404s often appear after CMS updates or redesigns.
๐ฎ Test yourself: which HTTP code means a page was permanently moved?
Lesson Task
Test your knowledge and earn +20 XP