robots.txt and meta robots in Detail

robots.txt and meta robots are the two tools people confuse most often. The first controls crawling, the second indexing and display. The rule the whole lesson rests on: robots.txt is used mainly to avoid overloading your site with requests, not to keep a page out of Google. For that there is noindex or a password. Next: how Google parses the file, which directives it supports, how rules conflict and what happens on server errors.

The basis is Google's specification on how it interprets robots.txt and the description of the robots meta tag and X-Robots-Tag. The basic introduction is in the lesson on robots.txt; here are the details.

Where the file lives and what it covers

• The file must be UTF-8 plain text at the root of the host;
• it applies only to its own host, protocol and port: a file at example.com does not cover a subdomain, http or another port, and a file in a subfolder is not checked at all;
• the size limit is 500 KiB; whatever comes after is ignored;
• field names are case-insensitive, but the paths in rules are not;
• Google generally caches the file for up to 24 hours.

Which directives are supported

That is the whole list. A Crawl-delay directive is often named too, but Google says "other fields such as crawl-delay aren't supported". Invalid lines are ignored. More on sitemaps is in the lesson on XML sitemaps.

How a group and a rule are chosen

• The group. Google picks the group with the most specific matching user agent; the order of groups does not matter. If several specific groups are declared for a crawler, their rules are combined, and the asterisk group is not combined with them;
• The rule. The most specific rule, by the length of the path, applies; for conflicts, including those with wildcards, the least restrictive rule is used. So "Allow always beats Disallow" is imprecise: the path length matters;
• Wildcards. Two are supported: the asterisk (any number of any characters) and the dollar sign (end of the URL). A trailing asterisk is ignored.

User-agent: *
Disallow: /cart/
Disallow: /*?sessionid=
Allow: /cart/help

User-agent: Googlebot-Image
Disallow: /drafts/

Sitemap: https://example.com/sitemap.xml

What happens on errors

• 4xx responses (except 429) Google treats as if the file did not exist: no restrictions;
• 5xx responses. For the first 12 hours Google stops crawling the site and keeps trying to fetch the file; then for up to 30 days it uses the last good version; if there is none, it assumes there are no restrictions. So a robots.txt that is down is no small thing;
• do not use 401 and 403 to limit the crawl rate: they have no effect on it.

The state of the file is visible in the robots.txt report in Search Console: which files Google found for the top hosts, when it crawled them and what warnings occurred. The old "robots.txt tester" in the tools menu is out of date.

What to block and what not to

It makes sense to block what creates load without value for search: a cart, internal search, addresses with session identifiers. Do not block what Google needs to understand a page: otherwise it won't do a good job analyzing pages that depend on those resources. For JavaScript there is an extra rule: Google doesn't run scripts from blocked files; details are in the lesson on JavaScript and SEO.

The robots meta tag

A meta tag in the head or an X-Robots-Tag header controls indexing and display. Rules can be combined with commas and are case-insensitive. The ones that matter here:

Two clarifications. First, Google no longer uses noarchive, because the cached link no longer exists in results. Second, with conflicting rules the more restrictive one applies: if a page has both max-snippet:50 and nosnippet, nosnippet applies.

X-Robots-Tag: for PDFs and more

The HTTP header lets you set the same rules for files that have no head: PDFs, images, documents. A crawler can be named before the rules: X-Robots-Tag: googlebot: nofollow. Rules with no crawler named apply to all.

HTTP/1.1 200 OK
X-Robots-Tag: noindex

The main trap: Disallow plus noindex

This is the best-known trap. The rule from the guide to blocking indexing: for noindex to be effective, the page must not be blocked by robots.txt and must be otherwise accessible to the crawler. If it is blocked, the crawler never sees the rule, and the page can still appear in results, for example if other pages link to it. For a page you want removed the order is: first allow crawling, add noindex, wait for it to drop out, and only then, if needed, block.

robots.txt, noindex and canonical

These are three different tools for three different jobs: load on the site, visibility in results and the choice of main URL. For duplicates the documentation warns directly: do not use robots.txt for canonicalization, and noindex is not recommended for preventing selection of a canonical page within a site. More in the lessons on duplicate content and the canonical tag.

How to check

1. Open the robots.txt report in Search Console and make sure the file is found without errors for every host.
2. Check there is no accidental Disallow: /.
3. Use the page analyzer to check that pages meant to be in search are not marked noindex.
4. Crawl the site with the crawler and find pages with noindex that are also blocked in robots.txt.
5. For a bulk check of indexing statuses, the URL audit. General crawlability questions are in the lesson on crawlability.

Checklist

1. robots.txt is at the root of each host, UTF-8, no larger than 500 KiB.
2. Only supported fields are used: user-agent, allow, disallow, sitemap.
3. There is no accidental Disallow: /.
4. What creates load but is not needed for search is blocked; resources needed for rendering are open.
5. Pages with noindex are not blocked in robots.txt.
6. PDFs and other files use X-Robots-Tag.

Mistakes

• Blocking in robots.txt what should be removed from search.
• Disallow and noindex on the same page at once.
• Using Crawl-delay and expecting an effect in Google.
• Blocking CSS and JavaScript needed for rendering.
• Using robots.txt for canonicalization.

What comes next

Next is the canonical tag: how to state a preferred URL without blocking duplicates from crawling.