Duplicate Content

Duplicate content is identical or very similar content at more than one URL. Contrary to a common belief, it is not punished in itself: Google clusters such pages and picks one as the main one. The problems are different: an unpredictable choice, blurred statistics and wasted crawling. This lesson covers where duplicates come from, how Google chooses the main URL, how you can influence that choice, and why blocking duplicates in robots.txt is directly discouraged in the documentation.

It rests on Google's pages on canonicalization and on consolidating duplicate URLs.

Where duplicates come from

Google names five causes, and each has its own remedy:

Some of these cases are questions of URL structure (parameters, case, trailing slash), others of crawlability. For HTTP versus HTTPS, redirection works; it is covered below.

How Google chooses the main URL

The principle is described on the canonicalization page: Google clusters similar pages, chooses the one that, based on the signals collected, is "objectively the most complete and useful for search users", and marks it as canonical. The canonical page is crawled most regularly; duplicates are crawled less frequently. Factors in the choice include the protocol (HTTP versus HTTPS), redirects, presence in a sitemap and rel="canonical". Google's choice can differ from your preference.

For HTTPS there is a caveat: Google prefers it over HTTP as the canonical version, except when there are issues or conflicting signals, such as an invalid certificate or insecure dependencies.

What you can do: signal strength

The consolidation page ranks the methods by how strongly they can influence canonicalization:

• redirects — a strong signal that the redirect target should become canonical;
• rel="canonical" — a strong signal that the specified URL should become canonical;
• sitemap inclusion — a weak signal.

The methods can stack, and none of them is required: the site will likely do fine without explicit signals. Why specify them, then? The documentation gives four reasons: to choose the URL shown in results, to consolidate signals, to simplify tracking metrics and to avoid spending crawling time on duplicates.

For redirects the documentation separately distinguishes permanent and temporary ones: with a permanent redirect the target is used as a signal of canonical, with a temporary one it is not, and the source page stays in results. Use permanent redirects when you are sure they will not be reverted. See where a chain leads with the redirect checker, and for details read the lesson on redirects.

Recommendations from the documentation

• Do not use robots.txt for canonicalization: Google may still index URLs disallowed in robots.txt, without their content. This directly contradicts the common advice "block the duplicate in robots.txt";
• do not use the URL removal tool for canonicalization: it hides all versions of a URL from Search;
• do not specify different canonical URLs for the same page using different methods, for example one in a sitemap and another in rel="canonical";
• do not specify a URL fragment as canonical;
• include rel="canonical" on the canonical page itself (a self-referential canonical);
• use absolute rather than relative paths: relative ones are supported but can cause problems;
• do not use noindex to prevent selection of a canonical page within a site: it completely blocks the page from Search, and rel="canonical" is the preferred solution;
• link internally to the canonical URL, not to a duplicate.

If pages are rendered with JavaScript, it is best to specify the canonical URL in the HTML source and make sure JavaScript does not change it; if you cannot set it in the HTML, leave it out there and set it only with JavaScript. rel="canonical" annotations with hreflang, lang, media and type attributes are not used for canonicalization: alternate versions are specified with other annotations. More on the tag is in the lesson on canonical, and on the limits of robots.txt in the lesson on robots.txt and canonical.

What you see in Search Console

In the page indexing report two statuses are about duplicates. "Duplicate without user-selected canonical" means a page duplicates another without indicating a preferred one; Google chose the other as canonical and does not serve this one. The report says "This is not an error, but is working as intended, because Google does not serve duplicate pages". "Duplicate, Google chose different canonical than user" means your choice and Google's differ; compare them with URL inspection. To find such pages in bulk, use the URL audit based on Search Console data, and to walk the site itself, the crawler.

This is not cannibalization

Query cannibalization is a different phenomenon: different pages with different content compete for one query. It is not a Google term, and it is treated not with canonical tags but by separating intents. Our cannibalization tool checks such overlaps against Search Console data. Pages that merely differ little in their text are a question of content uniqueness.

How to check a site

1. Check the protocol and www variants: they should redirect to one version. Chains through the redirect checker.
2. Find URLs with parameters, sorting and filters in a site crawl.
3. For each group choose a method: a redirect if the duplicate is not needed, rel="canonical" if it is.
4. Make sure the canonical page has a self-referential canonical and that links lead to it.
5. Review duplicates in the indexing report and compare Google's choice with yours.
6. Check the HTML through page analysis: the canonical must not be changed by a script.

Checklist

1. A canonical page is chosen for every group of duplicates.
2. Duplicates that are not needed are redirected permanently.
3. The others point to the canonical with rel="canonical" and an absolute URL.
4. Canonical pages reference themselves.
5. The sitemap lists only canonical URLs, with no conflict with rel="canonical".
6. robots.txt and noindex are not used for canonicalization.

Mistakes

• Treating duplicates as a "penalty" and spending effort on what Google decides itself.
• Blocking duplicates in robots.txt.
• Putting noindex on a duplicate instead of canonical.
• Specifying different canonical URLs by different methods.
• Confusing duplicate pages with query cannibalization.

What comes next

Next in the course is the group of lessons on crawl control: redirects, robots.txt and canonical and canonical tags.