What Are Outliers in SEO A/B Testing?

In this post, we volition sermon what outliers are successful SEO A/B divided testing. We volition shed airy connected immoderate of the issues that origin outliers to hap portion making recommendations connected dealing with them.

Let’s dive close in.

Sometimes we spot postulation spikes wrong an SEO A/B divided test; it is an denotation that determination are outlier URLs wrong the information set. The postulation spike is occurring arsenic determination is simply a misalignment of data.

There are respective types of outliers successful SEO A/B divided testing:

• Traffic Outliers
• Seasonality Outliers
• Page Element Outliers
• URL Exclusions

How Does Organic Traffic Affect SEO A/B Split Testing With Outliers?

SEO A/B tests necessitate a radical of pages to beryllium divided into 2 equitable groups (control and variant) based upon the integrated traffic. 

It besides required 100-days of pre-test integrated postulation earlier splitting URLs into power and variant groups. 

Each trial requires a minimum of 100,000 clicks to the acceptable of URLs being divided tested to person a statistically important trial result.

When postulation anomalies wrong the radical are split, these postulation outliers are excluded from the power and variant groups.

An Example of Organic Traffic Outlier Spikes:

*The 100 time exemplary successful the supra illustration runs betwixt Jan 30th, 2022, and May 9th, 2022.

While the investigating play of 30 days has not yet been completed, we tin spot determination are issues with the anterior 100-day exemplary done postulation spikes.

When we person spikes successful the 100 time exemplary betwixt the power and variant groups, it is an denotation that the exemplary is not suitable for SEO A/B divided investigating owed to the postulation not matching up. 

In addition, the wide airy bluish shading shows america the postulation exemplary is not good.

Read our cognition basal nonfiction connected How to work SplitSignal trial results.

Topical Relevance successful URLs And Traffic Modeling 

Some sites person postulation differences based upon topics, adjacent if the investigating leafage template is the same. The quality successful postulation tin make outliers.

When this happens, we urge utilizing URLs from akin topics. These URLs tin past beryllium uploaded into SplitSignal, wherever you would past re-run your test.

How Can Outliers Occur With Page Elements?

Outliers successful leafage elements tin hap erstwhile investigating a alteration successful leafage elements. You should guarantee that each the URLs successful your power and variant groups person the constituent connected the leafage you privation to change. Pages that are included that bash not person the constituent volition beryllium seen arsenic an outlier.

Page Elements Outlier Example:

If we were looking to alteration an h3 tag successful blog articles to an h2 tag successful our test, we would not presume each pages person the h3 tag.

Instead, your interior squad oregon an approved SplitSignal agency should crawl the tract with a instrumentality specified arsenic Screaming Frog oregon Deep Crawl to drawback the nonstop acceptable of URLs wrong the tract operation you are looking to bash your trial that has the desired on-page element.

In the illustration above, we would lone look to find the blog pages with an h3 heading wrong the tract structure.

What Causes Seasonality Outliers successful SEO A/B Split Testing?

When you tally an SEO divided trial before, during, oregon aft a holiday, you whitethorn find that your trial results person a postulation spike wrong them.

This is owed to URL(s) associated with the holiday. This spike of postulation throws disconnected the trial results.

The cardinal present is to place the URL(s) successful question, reappraisal them to spot if they are seasonal owed to the vacation and region them from the trial group. 

SplitSignal allows you to filter your pages to exclude pages that are outliers. You tin past re-run the test. If you inactive are seeing a spike of traffic, you volition request to reappraisal your URL(s) further arsenic determination are further pages that are outliers. Repeat the supra process.

Seasonality Outlier Example:

In this example, we volition amusement 3 tests of a planetary marketplace for creating unsocial designs and products with customization.

Test 1:

Timeframe: July 29th, 2021 - Nov 2nd, 2021

Hypothesis: By adding personalized to the h1, we volition summation clicks.

Analysis:

The trial had a affirmative effect size of 30.7%. 

Because the effect size was truthful large, we decided to rerun the trial to guarantee accuracy.

The trial showed a affirmative alteration successful existent clicks wrong the variant group.
 

Test 2:

Timeframe: Oct 8th, 2021 - Jan 12th, 2022.

Hypothesis: By removing personalized to the h1, we volition summation clicks.

Analysis:

The trial had a affirmative effect size of 8.6%. 

The trial showed a affirmative alteration successful predicted clicks wrong the power group.

Our investigating lone changed due to the fact that Test # 1 included the text: personalized to the h1 tag. In Test #2, we were removing it from the h1.

Why did we analyse some tests further?

• Both SEO A/B divided tests had affirmative results.
• Test # 1 has affirmative results for the variant group.
• Test # 2 has affirmative results for the power group.
• There were important holidays betwixt some timeframes: Christmas, Hanukkah, and New Year. 

Here is what we found:

Both divided trial groups had a affirmative trial that led america to look for a URL with a larger magnitude of postulation successful the divided investigating period, but not successful the 100-day exemplary anterior to the test.

We were capable to isolate a URL: /ornaments, and this URL received a larger fig of clicks than different URLs wrong some groups, and it was holiday-related. 

In addition, successful Test # 1, it resided successful the variant group, and successful Test # 2, it resided successful the power group.

Since the postulation irregularities happened successful investigating and not wrong the anterior 100-day model, SplitSignal did not exclude the URL.

We manually excluded this URL and re-ran the trial 1 much time.

Test 3:

Timeframe: Nov 10th, 2021 - Feb 16th, 2022

Hypothesis: By removing personalized to the h1, we volition summation clicks.

Additional Clicks

Pages Tested

*We manually removed the outlier URL earlier moving this test.

URLs visited by Googlebot:

Analysis:

This trial had a affirmative effect size of 3.7%.

While this trial is inactive running, we tin bring you these results arsenic Googlebot has visited each of the 128 URLs successful the variant group.

These trial results person normalized present that the seasonal outlier URL has been removed.

We besides cognize that the trial is statistically important due to the fact that the assurance level is astatine 96%, and a assurance level of 95% oregon supra is required for statistical significance.

Tests 1 and 2 were besides statistically significant. However, their results pointed to further factors that required further investigation to corroborate those results. These factors yet allowed america to spot the seasonal outlier URL. Tests # 1 and Test # 2 were skewed owed to a seasonal outlier URL.

URL Exclusions That SplitSignal Automatically Removes

SplitSignal excludes URLs from the anterior 100-day’s integrated traffic:

• URLs With Traffic Inconsistency - URLs with inconsistent click postulation with the different URLs wrong the pre-test 100-day postulation exemplary are excluded.

SplitSignal does a large occupation astatine handling postulation inconsistency exclusions, but occasionally URLs that are included bash not person postulation inconsistencies successful the 100-day pre-traffic model.

• Redirected URLs - URLs with redirects are excluded from inclusion.
• Not Found URLs - URLs that are not recovered are excluded from inclusion. 
• Blocked URLs - URLs that are blocked done leafage meta robots oregon a robots.txt record are excluded from inclusion. 

SplitSignal respects the directives from a robots.txt record and leafage meta robots arsenic this is the aforesaid behaviour that Google bot would have.