Crawling and Indexing Audit
A crawling and indexing audit is a check of whether search crawlers can reach the important pages and get from them what they need. It is the first block of any audit because it answers the cheapest and most important question: does the page meet the requirements by which Google considers it at all. Below: what the response codes mean in Google's terms, what to check in a crawler and in the indexing report, and what conclusions to draw. The structure of the checks is practice, and the answers to "what happens with code N" come from the documentation.
The basis is Google's help on HTTP status codes and network errors, redirects and the page indexing report. The general order of an audit is in the lesson on the audit system.
What response codes mean to Google
| Code | How Google acts | What to do |
|---|---|---|
| 200 | passes what it received to indexing; may index, not guaranteed | normal; but if the page shows an error message it will be a soft 404 |
| 301, 308 | follows the redirect; the target is a strong signal to process | replace internal links with the final address |
| 302, 307 | follows by default, but the target is a weak signal | make sure the redirect really is temporary |
| 4xx (except 429) | doesn't index such URLs; removes already indexed ones; crawl frequency gradually decreases | for removed content 404 or 410 are fine; for important pages restore or redirect |
| 429 | treated as a signal that the server is overloaded and counted as a server error | check the load |
| 5xx | Google temporarily slows crawling; indexed URLs are preserved for now but eventually dropped; persistent errors lead to removal from the index | deal with the server urgently; after recovery the crawl rate rises gradually |
Three corrections to familiar beliefs follow from the table:
- "301 passes weight, 302 does not" is not how it is described. 301 is a strong signal that the target should be processed, 302 a weak one; and permanent redirects don't cause a loss in PageRank. More in the lesson on redirects;
- "503: Google may deindex" is true only of a persistent error: with a single glitch indexed addresses are preserved;
- 401 and 403 are not used to limit the crawl rate: only 429 and 5xx affect it; 4xx do not affect the rate.
What to check in a crawler
- Addresses with 4xx and 5xx. Which of them matter, where links to them come from. Not every 404 is a problem: a removed page should return 404. The problem is a 404 on a page that should exist, and internal links to something nonexistent;
- Soft 404. This is when the server answers 2xx but the page content is an empty page or an error message; Search Console will show a soft 404. Do not redirect many old addresses to one irrelevant page, such as the home page: that can also be treated as a soft 404;
- Redirect chains and loops. Googlebot follows by default up to 10 redirect hops, but it is better to lead straight to the final address: chains add latency for users. Check with the redirect checker;
- noindex on important pages. A critical and easy mistake. For noindex to work the page must not be blocked by robots.txt;
- robots.txt blocks. Blocked CSS and JavaScript get in Google's way: it won't do a good job analyzing pages that depend on those resources. See the lesson on robots.txt and meta robots;
- Orphan pages. Google finds pages by links and from sitemaps. A page with no internal links is not necessarily "invisible": if it is in the sitemap Google learns of it. But it is more dependable to give it ordinary links in a elements with an href. See the lesson on sitemaps;
- Duplicates and canonical addresses. Variants with parameters, www and non-www, HTTP and HTTPS: the lesson on canonical.
What to look at in Search Console
In the page indexing report you see the state of addresses and the reasons for exclusion. Not everything in it is a problem: the status "Duplicate without user-selected canonical" is not an error but working as intended. New 404s after a CMS update or a redesign are visible in the same report; for bulk work on addresses the URL audit helps, and for 404s with organic traffic the organic 404 report. The internal links worth adding first are shown by the internal link audit. A crawl of the whole site is the crawler.
On "crawl budget": per Google's documentation it matters first of all for very large sites; for small ones crawl limits are usually not the cause of problems. More in the lesson on crawlability.
What is not in the documentation
- A claim that a 301 "passes link weight" and a 302 does not;
- "Orphan pages are invisible to Googlebot": Google also finds pages through sitemaps;
- "Chains slow crawling" as a formula: it names latency for users and a limit of 10 hops;
- A mandatory number of problems or a norm like "a good audit in 30 minutes".
Checklist
- Important pages return 200 and contain no error messages.
- Removed content returns 404 or 410, not 200 and not a redirect to the home page.
- There are no redirect chains or loops; internal links lead to final addresses.
- Important pages have no noindex and no robots.txt blocks.
- CSS and JavaScript needed for rendering are open to Google.
- Important pages have ordinary internal links and are in the sitemap.
- 5xx and 429 errors are dealt with: they slow crawling.
Mistakes
- Treating any 404 as a problem and curing it with a redirect to the home page.
- Using 401 and 403 to limit the crawl rate.
- Blocking in robots.txt pages that carry noindex.
- Blocking CSS and JavaScript.
- Leaving internal links on addresses that redirect.
What comes next
Next is the content audit: once pages are reachable and indexed, we check whether they give the reader what is needed.