SEO Experiments: Testing Changes on Data

An SEO experiment is a comparison of the results of a group of pages where you changed one thing with a group where you changed nothing. Such a comparison reduces the influence of seasonality, Google updates and competitors’ actions, but does not prove causation conclusively: search is not deterministic, and Google publishes no test methodology or durations. So below: what is documented (how not to harm a site while testing, how Google advises checking the effect of markup) and what is practice to apply carefully: group size, control, duration, effect calculation.

What Google’s documentation says

Google has no direct guide to SEO experiments. There are two sources worth knowing.

• A before-and-after check. The guide to structured data suggests running a before-and-after test on a few pages: choose pages with several months of Search Console data that are not seasonal and are stable, add the markup, confirm it is valid and found (URL inspection), and record results over a few months in the Performance report, filtering by URL.
• How not to harm a site when A/B testing. The help page “Minimize A/B testing impact in Google Search” speaks not of methodology but of caution. It concerns tests in which you show users different variants of a page; an SEO split test, where changes are made to a group of pages, is built differently, but the same cautions apply.

Cautions from the help page

1. Don’t cloak. Don’t show one set of URLs to Googlebot and a different set to humans: that violates the spam policies whether done by server logic, robots.txt or another method (spam policies). Googlebot generally doesn’t support cookies, so if the variant is chosen by a cookie it will see the version for browsers without cookies.
2. Use rel="canonical" on alternate URLs pointing to the original: the help page recommends it rather than noindex because it more closely matches your intent (the canonical tag).
3. Use 302, not 301, for redirecting to variations: a temporary redirect keeps the original URL in the index (redirects).
4. Small changes (the size, color or placement of a button or image, the text of a call to action) often have little effect on the snippet and ranking. If Google crawls your site often enough to detect your experiment, it will probably also index the eventual updates quickly.

How to build an experiment (practice)

How to calculate the effect

The simplest calculation compares changes, not levels: effect = (test after ÷ test before) ÷ (control after ÷ control before) − 1. An example with invented numbers: in the test group clicks grew from 10,000 to 11,800 (+18%), in the control from 10,000 to 10,200 (+2%); the relative effect is 1.18 ÷ 1.02 − 1 ≈ +15.7%. This is a teaching example, not the result of a real test. Then look at how stable the effect is: does it grow on most pages of the group or is it explained by a few, were there shifts in the control, and was the growth caused by the change itself rather than a new query or a link to one of the pages.

What is documented about the data: position in the report is a complex metric and is best tracked over time, and small fluctuations in position happen at any time and are not in themselves a reason to change anything (Search Console for an analyst). Causes of drops worth excluding: an algorithm update, a site-wide security or spam issue, seasonality and changing interests, a technical issue, a reporting glitch (debugging traffic drops; the lesson on traffic drops).

Platform tools

• Wins & Losses compares two Search Console periods and helps find pages that grew and fell; it has Google update date presets.
• Google updates monitoring overlays the core update history on traffic to separate the algorithm’s influence from your own edits (core updates).
• Other sources and services are in the lesson on advanced tools.

What remains practice, not fact

• “At least 4 weeks”. Google names no such duration; in practice it is a lower bound because pages need time to be recrawled and to accumulate data. For rare queries more is needed.
• “The only way to prove causation”. A control group reduces the influence of external factors but does not exclude them: for example, an update can affect different types of pages differently.
• Significance thresholds and group sizes. They are not set in the documentation; choose them from the spread of your pages’ data.

Checklist

1. The hypothesis and the metric are written down before the test starts.
2. The test and control groups are similar; there are enough pages.
3. One thing changes; the control is untouched; no parallel edits are made to these pages.
4. No cloaking; for variants with separate URLs, canonical to the original and 302.
5. The “before” and “after” periods are the same length, with the crawl delay taken into account.
6. Seasonality and Google updates are checked; the result is compared with the control.
7. The decision and result are recorded: what was changed, where, when, what was seen.

Mistakes

• Rolling out a change from a blog story without checking it on your own pages.
• Comparing “before” and “after” without a control group and drawing a conclusion about cause.
• Changing several things at once and attributing the effect to one of them.
• Showing the crawler and people different versions.
• Ending a test before recrawling and data accumulation.

What next

Next come advanced tools: which data sources and services help collect and compare experiment data and what to look at when choosing them.