Why Your Split Test Is Going To Take Forever & Produce Mediocre Results

Imagine for a moment that you are about to start using split testing on your website. You find a list of things that you want to test, maybe you have consulted a few blog posts like this one about obvious things to test with A/B testing software.

You launch your experiment and you check every few hours to see the progress. Initially your new version shoots into the lead, then it falls down a little and after some time the conversion rates start to level out a bit and look like they are becoming more stable.

Unfortunately, after a week, two weeks, three weeks, the experiment is still not showing statistical significance. One variation will be in the lead for a few days, and then the other will get a sale and take the lead…

This Is A Common Problem!

The reality is that some types of split tests can take a long time. Longer than most people realize, and unless you have a huge volume of traffic, this can mean two things:

  • Your tests will take a long, long time to produce a conclusion
  • You will never get round to testing new ideas because each test takes so long

The real cost is that if one test drags on, you can’t start the next, and if you need several months to run a single test, that might (or might not) give you a 10% boost in conversions, it is going to be a long time before you see anything like an ROI.

There Must Be A Better Way!

There, is. I actually wrote a post over at Moz recently: How to do conversion rate testing with little traffic. In that I wrote that the first rule is to test big changes which have a chance of producing big results. But in this post I am going to go into more detail about how to ensure that you stick by that plan, and avoid wasting time on slow tests.

First of all, let’s have a look at a fictional website that wants to run some split tests.

A Blog About Apples

This blog is about apples, they are fairly popular, and they have just launched an email list to collect email subscribers. At first, they appear to be getting about a 20% sign up rate, which is good, but they think they can do better.

They currently have about 100 visitors per day, which isn’t huge, but they are happy with it for now. They are just going to test one trial variation (Page B) at a time as per Rule 3 in my Moz post.

So how long is it going to take to run a split test and find a winner? For that we can use Wingify’s duration calculator. But first, we need to take a guess at how much of an improvement Page B will show. This is of course impossible to know until we test it, but let’s shoot for a 50% improvement:

As you can see, this test will take 5 days. That’s not bad at all is it?

But a 50% improvement would be pretty good going, so what if we don’t actually hit that target? The truth is that most split tests make improvements in the region of 0-10%.

And don’t forget of course that it’s also possible that Page B could be be worse (ie, the improvement might be less than 0%).

So let’s look at how different levels of improvement will affect the time taken to run the test:

  • Page B converts at 30% -> 50% improvement = 5 days
  • Page B converts at 26% -> 30% improvement = 14 days
  • Page B converts at 24% -> 20% improvement = 32 days
  • Page B converts at 22% -> 10% improvement = 128 days
  • Page B converts at 21% -> 5% improvement = 512 days

Here’s that same data in a nice graph:

This graph shows you how the time to complete the test will vary according to the conversion rate of Page B. As you can see, a small improvement is not just disappointing, but will take a much longer time to complete.

Obviously we are aiming to test pages which will have a big impact on conversions, but until we run the test, we have no way of knowing. So we might think that Page B will convert at 30%, but if it only converts at 25%, not only will it be disappointing, but the test will take 4 times as long to complete.

Taking A Different Strategy

Instead of wasting our time with small improvements, we are going to go for broke. To make big leaps in conversion rates we need to try big ideas which can either work well, or not at all.

This recent post on Wordstream by Larry Kim: Everything You Know About Conversion Rate Optimization Is Wrong, includes this image, based on their extensive analysis:

Obviously a good conversion rate for you is different than a good one for someone else. But what this shows is that the top 10% of conversion rates convert 5 times better than the average. That’s the equivalent of our Apples blog discovering that Page B is a 500% improvement over the original.

We’re going to go a little more conservative and shoot for a 50% improvement, and to do that we are going to need to run lots of tests and try lots of ideas:

The Methodology

First of all, you need to think big. Testing changes in the layout won’t get the results we want (probably), so we need to think about changes in the messaging, in what is being offered and the overall value proposition. In plain English this means things like:

  • Try changing what you offer (free ebook, ecourse, etc…)
  • Add, change or remove social proof signals
  • Changing the headline (and accompanying copy)

Imagine that you are starting your page / website from scratch, create user personas note all of the reasons why someone might want to sign up for your mailing list (or otherwise convert) and all of the reasons they might not.

This is also a good time to survey your existing visitors and subscribers for more ideas (I recommend testing Five Second Test as a quick way to get ideas for split test variations), as well as using Analytics and heatmapping tools.

Create 5 to 10 landing page ideas that are as different from one another as possible in terms of what they are offering, and then start testing them against the original.

Finding Unicorns

Remember that we want a 50% improvement as a minimum. So instead of letting the test run until statistical significance is reached, all we care about is being (95%) certain that our new version IS NOT at least 50% better than the original.

For our example website, a 50% improvement will only take 5 days of testing to prove. This means that no test need to run for more than 5 days – that is as long as it will take to discover whether or not our “Page B” is a star.

The Pro’s & Con’s

The downside to this strategy is that you may end up rejecting page versions which were actually better than the original (but better by a margin of less than 50%). But the upside is that you can test more variations much faster, which maximizes your chances of creating a substantial improvement.

In our example, you would be able to test 10 different variations in less than 2 months

Essentially you are trading – you risk missing out on small wins for a chance at creating a much bigger win.

How To Do This Yourself

The exact numbers depend on your own situation and how your site currently works. In general tests run faster if your conversion rate is higher. So if you want to improve a conversion rate of 1% you will need to improve it by a much bigger margin.

  • Brainstorm page variation ideas
  • Collect the best / most dramatic ideas
  • Run the Wingify test duration tool
  • Use your current traffic and conversion figures
  • Play with the improvement %

The key is finding a compromise between a realistic improvement margin and a tolerable “time to completion”. Remember that you will need to multiply the time to completion by the number of tests you want to run (the number of variations to test).

Start running your tests, one by one until one of your tests shows the target improvement. When this happens, significance should be reached within the original “to to completion” window, assuming your traffic and other variables have remained constant.

Leave a Comment

Your email address will not be published. Required fields are marked *