robots.txt: Crawling, Not Indexing

robots.txt controls crawling, not indexing. The difference is not academic: a single line in this file can zero out a site's visibility, and using it to remove a page from search usually fails — and the documentation explains why.

The lesson follows Google's specification: the introduction to robots.txt and how Google interprets the file. The second is rarely read, and it describes the behaviour everyone eventually runs into: caching, server responses and limits.

What robots.txt does

robots.txt is a text file at the root of a host: https://example.com/robots.txt. It tells well-behaved crawlers which URLs they may and may not crawl. Its main purpose, as the documentation puts it, is to avoid overloading your site with requests.

The difference between crawling and indexing is what people trip over most. Disallow stops a crawler from visiting a page. But if other sites link to it, Google may index the URL without visiting it: the result appears without a description. That is documented plainly, and it means something simple — blocking a page from crawling did not remove it from search, it only removed Google's ability to know what is on it.

The same fact creates the main trap: if a page is blocked in robots.txt, the crawler never sees the noindex on it. Two prohibitions together cancel each other out. The glossary covers the terms: robots.txt and noindex.

Basic Syntax

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

Scope and limits

Three things from the specification that save hours of debugging.

The full syntax, with directive examples and the order rules are applied in, is in the guide to creating the file.

Dangerous Lines

User-agent: *
Disallow: /

This combination blocks crawling for the whole site. It is sometimes used on development environments and accidentally shipped to production.

User-agent: *
Disallow:

An empty Disallow means nothing is blocked. It is not an error, but it can look ambiguous. On a public site, a sitemap-only file or a clear comment is often easier to read.

User-agent: *
Disallow: /*?sort=
Disallow: /*?filter=

Patterns can limit noisy parameters, but test them against real URLs. One broad mask can accidentally block useful filters, pagination, or listings that should receive search traffic.

What happens when the file is unavailable

Nobody thinks about this until it happens. And it happens regularly: the server went down, a deploy broke the route, the CDN returned an error. Google's behaviour is described in the specification and differs by response.

And one more thing that makes a fix look like it failed: Google generally caches the file's contents for up to 24 hours. Removing a stray Disallow: / will not take effect instantly — that is normal, not a reason to edit the file again.

The noindex rule

If the goal is to remove a page from search, do not start with Disallow. Leave the page crawlable and add:

<meta name="robots" content="noindex">

For files with no HTML — PDFs, images — the X-Robots-Tag header does the same job. Both methods and their syntax are in the documentation on the robots meta tag, and the overall removal procedure in the guide to blocking indexing.

The order matters: first let Google see the noindex and drop the URL from the index, then restrict crawling if you still need to. Done the other way round, the page stays in the results — without a description and with no way to fix it.

How to check the file

The check takes five minutes and comes in three forms, from fast to precise.

1. Open /robots.txt in a browser. Does it return 200? Is there a stray Disallow: / on the production site?
2. Look at the robots.txt report in Search Console: which version Google currently sees and when it fetched it. Parsing errors show up here too.
3. Check a specific URL with URL Inspection. If the reason is "Blocked by robots.txt", you have found the problem. The URL auditor does this in bulk.

How many pages dropped out of the index and why is in the page indexing report; the gap between what exists on the site and what Search sees is the coverage gap. The glossary covers the terms: crawl budget and index coverage.

A Normal robots.txt for a Small Site

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search

Sitemap: https://example.com/sitemap.xml

Practice: check your own robots.txt

Five steps, ten minutes:

1. Open https://your-domain/robots.txt. Read the whole file — it is usually twenty lines, half of which stopped being needed long ago.
2. Write out every Disallow line and say what exactly it blocks. If you cannot, the line is a candidate for deletion.
3. Check that CSS, JS and images are not blocked. Without them Google understands the page less well — look at it as a crawler does with the page analyzer.
4. Make sure no page carrying noindex is also blocked in robots.txt. That is the most common pair of mistakes, and it leaves the page in the results permanently.
5. Check the subdomains: each has its own file. A crawl covers the whole site, and sitemap monitoring the state of the sitemap.

How to tell the file is in order

The sign is not that the file is "correct" but that you can explain every line in it. robots.txt is a rare case where a short file beats a thorough one: the fewer rules, the fewer ways to block something you needed.

Three questions to check yourself. Do you know what each line in your file blocks? Do you understand the difference between "do not crawl" and "do not index"? Would you notice a Disallow: / reaching production the same day?

What not to expect: that robots.txt removes a page from search or protects a private area. The file is public — anyone can open it — and every path listed in it is visible to everyone. Secret directories are not hidden by it, they are advertised.

What comes next

This is the last lesson of the SEO Foundations course. If you did not start at the beginning, Google Search Console is worth going back to: nearly every check in this lesson happens there. The previous lesson, on platforms and tools, is here.