How Search Engines Find Pages
How search engines find pages is a chain of three different steps: discovery, crawling and indexing. They get mixed up often, and their failures are fixed differently. This lesson covers how Google finds addresses, why a crawl does not guarantee indexing, where robots.txt and noindex collide, and who actually needs to think about crawl budget.
It rests on the crawl budget guide and the introduction to robots.txt.
Step one: how Google finds addresses
There are two main routes: links and the sitemap. Googlebot follows links literally: it parses the response for URLs in the href attribute of HTML links, and a link only counts when it is an a element with an href attribute. The sitemap is the second channel, but not a mandatory one: the overview states that if you don't need results for video, images or news, you might not need a sitemap. More in the lesson on XML sitemaps.
The consequence for structure: a page reached by at least one ordinary link from a section Google already knows is found more reliably than a page that exists only in the sitemap. What a link crawl actually reached is what the site crawler reports, and which pages lead nowhere is internal link analysis. The structural principles are in the lesson on information architecture.
Step two: crawling — and what limits it
Google does not crawl everything: a site's crawl budget is determined by two main elements — the crawl capacity limit and crawl demand. The limit exists so that servers are not overwhelmed: it factors in the number of parallel connections and their duration. For Googlebot, demand varies with a site's size, update frequency, page quality and relevance compared with other sites.
The budget is counted per host: subdomains are treated as separate sites with separate crawl budgets.
Who actually needs this
The most common exaggeration in the topic. Google's guide is intended for a narrow group: sites with 1 million+ unique pages whose content changes weekly, sites with 10,000+ unique pages whose content changes daily, and sites with a large portion of URLs classified in Search Console as "Discovered - currently not indexed". It adds that these figures are a rough estimate rather than thresholds. And further: if your pages are crawled the same day they are published, you do not need to read the guide — keeping the sitemap current and checking the indexing report regularly is enough.
For a site of a few hundred pages "crawl budget" is almost never the cause of problems — the cause is quality, duplicates and structure.
Step three: crawled does not mean indexed
The same guide states that not every page that is crawled will necessarily be indexed — after crawling, each page must be evaluated, consolidated and assessed for its suitability for the index. In the indexing report these are two different statuses, and they have to be told apart:
| Status in the report | What it means | What to do |
|---|---|---|
| Discovered - currently not indexed | The page was found but not crawled: Google wanted to crawl it but expected to overload the site, so it rescheduled | Check response speed and the volume of surplus URLs |
| Crawled - currently not indexed | The page was crawled but not indexed; it may be indexed later, and there is no need to resubmit it | Look at quality and uniqueness, not at crawling |
Address-by-address analysis is what URL auditing does.
Where robots.txt and noindex collide
The most expensive confusion in the technical part. robots.txt is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a page out, block indexing with noindex or password-protect the page.
And now the pairing people get wrong most often. The noindex rule only works if the page is not blocked by robots.txt and is otherwise accessible to the crawler; if it is blocked, the crawler will never see the rule, and the page can still appear in results, for example if other pages link to it.
The state of the rules is read in the robots.txt report. And if the crawler needs scripts and stylesheets to understand a page, do not block them, or Google won't do a good job of analyzing the page.
What wastes crawling
| Problem | Why it hurts | What to do |
|---|---|---|
| Filters and sorting as separate addresses | A finite catalogue becomes an unbounded set of URLs; there is a dedicated guide | Decide which combinations search needs; do not crawl the rest |
| Links to 404s | Crawling is spent on what does not exist | Fix the links; find them in organic 404s |
| Redirect chains | Several requests instead of one | Link straight to the final address: redirect check |
| A slow server | The crawl capacity limit falls | Read the Crawl Stats report |
| A stale sitemap | Addresses that no longer exist, new ones missing | Track it with sitemap monitoring |
Norms such as "server response under 200 ms" or "an important page no deeper than four clicks" do not appear in Google's documentation. Speed is checked against actual crawl statistics, and reachability by whether a meaningful path exists, not by a click count.
JavaScript and the mobile crawler
Google processes JavaScript web apps in three main phases: crawling, rendering, indexing, and if a URL is disallowed in robots.txt, Googlebot skips the request and Google Search won't render JavaScript on blocked pages. Links that are not in the HTML and appear only after a user action stay a blind spot.
There are two crawlers: Googlebot Smartphone and Googlebot Desktop, which obey the same robots.txt token, so you cannot selectively target just one. For most sites Google Search primarily indexes the mobile version of the content, and the majority of crawl requests are made with the mobile crawler. So links and content that are missing from the mobile version are effectively missing for Google.
Diagnosis step by step
- Was the page discovered? Is there an ordinary link and a sitemap entry.
- Was it crawled? Is it blocked in robots.txt, does the server respond, does it show in the crawl stats.
- Was it indexed? Is there a noindex, is it a duplicate, how does it differ in substance.
- Do the signals agree? Disallow and noindex should not sit on the same page.
Mistakes
- Using Disallow to remove a page from search.
- Putting Disallow and noindex on the same page.
- Chasing "crawl budget" on a site whose pages are crawled the day they are published.
- Assuming a sitemap entry guarantees indexing.
- Resubmitting the same page rather than working out why it is not indexed.
- Checking only the desktop version.
What comes next
The next lesson is about the sitemap: what it actually does, and what should and should not be in it.