Most Popular Multivariate Testing Tools in 2022

So you’re currently running A/B tests and looking to scale out your testing program.

You have good traffic, you’re seeing great results and now you want to run far more tests and get that data asap.

But how can you start testing faster? Or better still, how can you run more tests and get more information during the same time period?

That’s where multivariate testing comes in. It allows you to test multiple variations all at once, to see which particular one works best.

In this guide, we’re going to break down what MVT is, how it works, what you need in place to use it and break down a selection of top multivariate testing tools available.

Everything from their price and features, to how they do business so you can make the right choice when deciding which multivariate testing tool to get.

So let’s dive into it…

What Is Multivariate Testing?

The best way to understand multivariate testing is to compare it to other testing methods and see how it differs.

So let’s look at an A/B test and a multivariate test and how they compare.

With an A/B test, we test a single variation against an original control version. Maybe we have an idea of how to improve the headline, so we create that variation and then see which performs best.

Your A/B test will look like this:

Headline 1 (control) vs Headline 2 (the variation idea)

Another test we could run is an A/B/n test.

This is where we can test multiple variations against the control, to find the overall winner. A/B/n tests are univariable.

For example, we could have a heap of headline ideas and are not sure which will work. But because we have the traffic to run a few test variations and still get accurate results, we decide to test multiple headline ideas to see which performs best.

Headline 1 (control) vs Headline 2 vs Headline 3

Once we find the winning headline variation, we might then test a bunch of hero shot images, and then maybe a bunch of CTA angles.

As you can guess, this all takes time, right?

Another option is to run a test where we change a bunch of things all in one go and see if that improves the metrics.

We tend not to do this with most A/B testing unless we’re looking at making a radical change i.e throwing a hail mary to try and find a new winner because all other test ideas are stagnating or are not making any impact.

So why don’t we recommend doing radical changes like this for normal campaigns and what does it have to do with multivariate testing?

We test one element at a time because if we make huge changes and the new page wins or fails, we’re not sure which of those changes caused that lift.

Multivariate testing or MVT allows us to test multiple variants, while also tracking each of those changes and how they affect the result.

Not only can we test multiple variations of elements, but we can even run different combinations of them until we find the ultimate winner.

Let’s say that we wanted to improve our home page and we have a few ideas of what to test to see the most lift:

• The headline
• The hero shot image
• The CTA.

With a multivariate test, we could not only test every headline idea, hero shot image option, and CTA angle, but we can also test each combination of those variations to find the winning combination:

Headline 1 + Hero image 1 + CTA 1 vs

Headline 1 + Hero image 2 + CTA 1 vs

Headline 1 + Hero image 1 + CTA 2 vs 

Headline 2 + Hero image 1 + CTA 1 vs

Headline 2 + Hero image 2 + CTA 1 vs

Headline 2 + Hero image 2 + CTA 2

Multivariate testing can help you find out the best combination of elements and understand what caused those changes.

Still, doing proper multivariate testing involves spending lots of time and resources. Are you willing to invest that many resources, not just for testing but also for QA and analysis? In some scenarios though, it might be the better option. Let’s see what those are.

When to Use Multivariate Testing?

What is multivariate testing used for? Here are a few of the times when an MVT test might be more useful than an A/B test.

1. When You Have High Traffic and Want to Ramp Up Your Testing Program

Traffic is the major crux of multivariate testing. You need a high volume of traffic to be able to test each change and trust your test results.

But if you have the traffic and you want to scale up your testing to get more data and make more decisions, then MVT will allow you to run multiple ideas all at once.

2. When You Want to Increase Lift and Have a Few Ideas but Don’t Want to Wait to Test Them One by One

Some people think that you should only use MVT to refine current web pages with minor changes, due to the time investment, but it’s not true. Sure, you can use it for that, but you can also use it for broader ‘what if’ tests of larger elements.

(Maybe you’re not sure which element will provide the best lift and so you want to try throwing some smartly hypothesized ideas at the wall to see which works.)

If you have the volume of traffic to run an MVT, then it’s a no-brainer to not try and find broad page combinations far faster. Just be sure to measure the effects of that traffic AFTER the test from each variation also, as this will give you further insight into your audience and why it worked.

For example, on Facebook, you can set up basic multivariate test ads, by adding in variations of a thumbnail image, headline and body copy, etc.

The test will run and tell you which variation of that ad gets the highest CTR.

If your goal was to just get clicks, then this ad would win, right?

But if all that traffic then bounces off the page you send them to, then all you’ve found is a variation that causes that audience to click, not one that gets them to stay and convert.

It’s the same when running an MVT test on a page, especially if you’re using it as a broader idea gathering method. Check what happens on the page but also what happens after they click.

3. When Refining a Previous Test to Get Incremental Lift

You can also use MVT to measure and improve small refinements. In fact, a lot of agencies like to use them to eke out even more lift on current winning campaigns and test small changes.

Let’s say that you’ve already A/B tested a page multiple times to find a winning variant, but you want to get as much lift as possible, even if it’s just 0.5%. You can run MVT tests on that page to test further (because at 100,000 visitors a month, every 0.5% helps is noticeable!)

4. When Searching for the Maximum Possible Lift From a Combination of Ideas (Interaction Effects)

Sometimes, a consecutive A/B test can give us winners on paper, but when we combine them, they are not always the highest performing option for our page.

Here’s an example:

• You run a headline test and pick a winner and implement.
• Then you run a hero shot image test on that same page with the winning headline variation and implement.
• Then you run a CTA test on the same page with the previous winning elements and implement.

In theory, this should be the highest performing version of your page but weird things can happen because of interaction or lack thereof. Simply put, it’s possible for a losing element in one test to sometimes outperform your sequential A/B test winner when combined with another element variation.

How?

Well, maybe a losing headline from the first test provides a higher backend conversion rate when combined with the winners from the image and CTA test.

This can happen but you would never know this if you just ran A/B tests. Instead, you would have removed it from the options before you ever tested the other elements. This is why multivariate testing is so powerful. It allows you to find the highest overall page performance, even if it means that it’s taking a less-performing element from one part of the test.

Better still, it can even tell you which element helped the most…

5. When Searching for the Most Important Element on the Page for Your Audience, for Future Tests and Pages

MVT will not only show the winning combination of your different variations and elements, but it will also show you the element that provides the most lift.

Maybe your hero shot gets more lift, but you also see that out of all your elements on your variation page, it contributes most to your end result?

This will then give you some insight into where to test first for even further improvements, so that you can focus on hero shot improvements first, etc.

6. When Applying Machine Learning to Your Testing Process

Due to the nature of MVT requiring and tracking large amounts of data and interactions, it can actually be used as a great source to feed machine learning testing programs.

ML requires large data to work best, but once it gets it, it can actually start to see patterns and predict results and behaviors with less traffic over time. In fact, sites like Amazon are doing exactly this…

Multivariate Testing Examples

Multivariate Testing Example #1

Amazon used MVT and machine learning to adjust the layout of their landing pages, resulting in a 21% increase in purchase rate in just 7 days!

Multivariate Testing Example #2

Booking.com uses multivariate testing to test out new unique features to specific audiences and then measures their performance before rolling them out sitewide.

Multivariate Testing Example #3

Microsoft saw a 40% increase in conversion rate for their SMB website, after running an MVT test on multiple page elements (8×4!)

Multivariate Testing Example #4

It’s not just the juggernauts who can use MVT either though. HawkHost saw a 204% increase in sales when they ran an MVT test on their homepage (testing their headline, subhead, and hero shot).

The Pros & Cons of Multivariate Testing

If multivariate testing is so powerful, why aren’t we all using it?

You see, the beauty of an A/B test is that you can test one idea fast, with a relatively small audience size. This means that most businesses can run a test even if they have a fairly low traffic, then see a result within 14-30 days.

But with a multivariate test, you need a much larger audience to run with, because you’re testing multiple things at once. (Especially so if you’re doing multipage testing or testing multiple combinations of variations.)

Let’s say that we wanted to test a headline on our home page:

• Headline Control
• Headline Variation

We’re just testing the single variation, so that’s 2 options total.

Now as a simple example, let’s say that we want to get 1,000 visitors to each version of the page (the variant and the control) so that we can be sure that the test results are trustworthy.

(Traffic needed can vary based on the page and its conversion rate, but let’s simplify here just so you get the idea.)

That means that we would need 2,000 visitors to run the test and trust the results, with 1000 visitors going to each page.

What about if we wanted to test:

• 3 headlines
• 3 hero images, and
• 3 CTAs?

That’s not 9 variants total, but actually 27 combinations of 9 variants (3x3x3).

This means that if we wanted 1000 visitors per variation, then our traffic requirement would be 27,000 visitors instead to run the test and trust the results (based on our oversimplification of 1,000 visitors being enough per page.)

In reality, you would probably need around 10,000+ visitors per variation to get accurate data, as traffic alone is not enough, you also need to see conversions.

This is why a lot of businesses don’t run MVT tests, simply because they don’t have the traffic to run them effectively.  In fact, most agencies recommend having around 100,000 monthly visitors before you start running MVT campaigns unless the page has a very high conversion rate already. And even then, they still recommend only running maybe 6 variations max, so as to be able to run them faster.

But if you do have that kind of traffic?

Well then MVT can allow you to perform the equivalent of far more tests in almost the exact same time frame…

What if we take that example from above and we allow each test to run for a full 30-day sales cycle to get an accurate measuring window?

Assuming we have the traffic, we could run 27 test combinations all at once on that one page over maybe 30-60 days as a single multivariate test. Not bad when you compare that to running 27 consecutive A/B tests on that page, over a 27-month time frame!

(Again, I’m simplifying here and assuming you run for a full sales cycle on each test.)

But as you can see, multivariate testing allows you to get insights and ideas, implement them faster and lower uptime on consecutive losing tests.

Pretty good, right!?

Obviously, there are some other drawbacks to MVT testing other than traffic requirements.

Because of the large number of variations and combinations you can run, it does greatly increase your QA time pre- and post-test.

You need to make sure that every variation works and also makes sense. (Not showing a headline and image that don’t match etc), and so it does increase your setup time and cost dramatically.

Also, be sure to check on effects.

Just because you see a lift in conversion on that particular page, track it forward to your ultimate goals. The last thing you want is more clicks but poor user experience or a high bounce rate (or more clicks but no sales).

The Pros of Multivariate Testing

• Assuming you have the traffic, multivariate testing allows you to test a lot of variations, fast.
• MVT can help you to scale out your testing program to find more data on what works and what doesn’t.
• It may help you to find winning combinations quicker than consecutive A/B tests.
• It can help you to find winning combinations that you might have missed when running consecutive tests. (Imagine if one headline was 2nd performing and you cut it out, but paired with the 3rd image variation, it somehow doubled lift? With a consecutive test, you would have missed this. A multivariate test takes a higher degree of interaction effects).
• Some A/B testing software will allow you to see which particular element helped provide the most lift (giving you a deeper understanding of your audience).
• It can also be used to help find minor lifts on high-performance pages. (Imagine you have a high-performing page already and you’re shotgunning new ideas to see minor lifts, such as color changes.)

The Cons of Multivariate Testing

• Need a high volume of traffic to run, especially if you want stat sig so that you can trust your results and not see false positives or negatives (type 1 and type 2 errors), and stat power to measure smaller % lift.
• Traffic requirements grow quickly the more variations and combinations that you test, so keep that in mind. Consider removing additional elements if they may not provide much lift. (i.e. test the headline before worrying about testing footer colors, etc. You can always test those after if you really want to.)
• It’s not just traffic that affects sample size either. The conversion rate of the current page and variables, all affect the volume of the sample size required to hit statistical significance. Better conversions can lower the traffic needed, lower conversions increase that requirement.
• The margin of error when testing more variables also increases the volume of traffic required before you can trust the results.
• Need to run intensive QA tests to make sure they all work in advance.
• Need to also run QA to make sure that the variations make sense together and don’t have opposing or conflicting combinations. (CTAs that don’t match offers, headlines for a beach holiday while advertising a ski trip, or even conflicting or blending font and background colors.)
• If you don’t have a high traffic source, you can still run an MVT test, it just might take a LOT longer until you hit stat sig and can end the test. (Sometimes multiple months or longer). In fact, it can sometimes be quicker and more efficient to just run consecutive A/B tests.
• You may lose some audience insights, simply by testing multiple things at once. (Testing a single element or angle can help you understand your audience better sometimes.)

Multivariate Testing Statistics: The Math Behind MVT

So let’s nerd out a little bit…

When it comes to multivariate testing, we use a different statistical analysis method to standard A/B testing or split testing.

I don’t want to get into the math of it all too much, but here’s a simplification:

• With A/B testing, we use a t-test (or z-test) to measure the difference between 2 or more sets of results.
• With multivariate testing, another method is used to find the test results called ANOVA or ‘Analysis of Variance’. You don’t need to know all the details, but here’s the core thing. The main difference is that ANOVA also measures the connection and impact between variables. This is one of the main strengths of MVT because it allows us to see how combinations of elements impact each other. 

Depending on which tool you use, there can be 3 types of multivariate testing methods offered when running an MVT campaign. Each of them affects the math on how your test will work.

We call these:

• Full Factorial,
• Fraction Factorial, and
• Taguchi.

So what do they mean?

Full Factorial

Almost every MVT tool will offer Full Factorial testing as the basis for its main testing model and it’s pretty much what you want to use all the time.

This method refers to the ‘full’ distribution of traffic in your test.

Simply put, it’s where you test every combination at once, dividing the traffic to the test equally between each variant.

With an A/B test, you split the traffic 50:50. But with an MVT, if you had 10 variants, then this would mean that only 10% of the traffic goes to each variant.

Because of the number of variants and the traffic required to reach statistical significance with each variation, full factorial does require a very large traffic source, but it is the most accurate method.

However, some A/B testing platforms will also offer something called ‘Fraction Factorial’ testing as an alternative.

Fraction Factorial

This method attempts to bypass some of the larger traffic requirements by testing just a smaller sample of variations and then estimating end conversion rates based on current test results.

Its goal is not in the accuracy of predicting trustworthy results for scaling out like normal i.e if we scale this out to X audience will it perform the same.

Instead, it’s designed to help you see where the early traction is starting to happen so that you can get insight before you test deeper or with another method.

Let’s say that you have 10 variations you want to test.

Fractional factorial will actually test 5 of them instead and then try to infer the conversion rate on those other 5 tests, based on the results so far.

As you can guess, this is not as accurate and can lead to errors and false positives, and is actually a source of argument among optimizers and marketers. (With almost all recommending full factorial as the main testing method.)

Taguchi

Finally, you have the Taguchi method of multivariate testing. Originally designed as a method of measuring quality control against assumptions of acceptable losses in R+D for product manufacturing.

Most tools won’t offer this because of its complexity to use.

When to Stop a Multivariate Test?

There are 3 main things that we need to consider when running any test before we should stop it:

• For the test to have run long enough to have collected a large enough sample size to each variant, and for the test to have collected enough conversions to be relevant.
• For the test to have run either 1 or even 2 full sales cycles (to balance out traffic and shopping periods over 30-60 days.)
• And finally, for the results to be statistically significant i.e if we take this test out in the wild and scale it up, then we can be 95% sure it will continue to perform the same way.

We recommend that you do not stop any test before this point. Otherwise, the data will be insufficient to work with. We might however stop and then re-start a test if there are issues with performance such as one variant’s payment forms not working etc, but ideally, we try not to even peek when it’s running.

(Some agencies do recommend stopping early if there is a clear winner, or even stopping specific variations if their conversion rate drops below 10%, and then allowing the test to keep running while filtering more traffic to the higher performers. Obviously, this can affect the test results greatly.)

How to Set Up and Perform a Multivariate Test?

The process of setting up a multivariate test is very similar to an A/B test.

Step #1: Decide on Your End Goal

First, you need to form a hypothesis for your page, its goal, its current results, and how you think it can be improved.

You can use our hypothesis generator to help you clarify what you want to achieve, and set some concrete goals.

Step #2: Figure Out What to Test

Then, decide on which elements to test and what variations you think could improve your current results.

For example, you may think that testing the ‘core elements’ above the fold will increase conversions so you want to test the headline, the hero shot, and the CTA button text.

Step #3: Calculate Testing Timeframe

Before you run the test or create any assets, we recommend that you perform a pre-test sample size calculation, so you can estimate the time frame and traffic requirements for this test to run. This will help give you some insight into how many variations you can feasibly run with your recurrent traffic and still get fast results. (Or even let you know if you need to test fewer options).

In the example above, you have 3 elements: the headline, the hero shot, and the CTA button text.

If you want to test a single variation for each that would be 2x2x2=8 combinations. Knowing this, you can start to figure out how much traffic you need.

Input your current weekly page traffic and conversion rate into the calculator (along with any power and confidence settings).

Then, set the number of pages that you want to test. (8 in this example)

This will then give you a time frame for how long you need to run to be able to detect the minimum lift.

Step #4: Build the Test

Next, head across to your tool and set up the different versions and combinations for your multivariate test.

Start by selecting the multivariate test option.

Give it a name and then add the URL of the page you want to test (so the tool can pull it in for the WYSIWYG edits.)

In this example, we’re going to be working on improving our home page conversion rate.

Once the page loads up in the tool, you need to set up your test elements.

Here at Convert, we use 2 naming conversions for the parts to edit in multivariate tests:

• The Section, which refers to the element on the page that you want to test such as a headline or CTA, and
• The Variation, which is your edit of that particular element.

I recommend that you pick one section at a time to edit and add each variation.

Give the section a name such as ‘Headline’, then go ahead and edit and add the variations for that section. (Be sure to save each variation as you work on it.)

Then move on to the next ‘section’ or element that you want to test. In this example, it would be setting up another background hero shot, and then setting up a CTA button text variation.

Step #5: Make Sure It All Works

Once you’ve set up each variation, run a QA test on each new treatment to make sure that they all work and that each element fits together sensibly. (No competing CTAs or mismatched information/images, or double up on font color and buttons.)

Click on ‘save and continue’ and a new page will load. Then, go ahead and click on the “Reports” tab.

From here you will see page variations and information on them. Click on the ‘eye icon’ next to each page to see a preview of it and test the links work.

From here you can run a QA test and use that same URL for mobile tests.

Step #6: Push It Live

As long as each variation is all working, you can now start the test and let it run until you have enough data.

(Notice that there’s an option to push the test live right from the report page where you opened the QA tests.)

Best Multivariate Testing Tools to Mature Your Optimization Program

Now that you understand how multivariate testing works, let’s break down the website optimization platforms on the market that offer multivariate testing. We’ll cover their features along with their pricing if they offer free trials and a few other details.

One quick thing before we do that.

Remember that the tool you use is ONLY as good as the person setting it up. If cash flow is tight, then invest in someone to help internally first (or reach out to an external agency) and just use a cheaper tool for now.

It’s all very well having an amazing MVT testing tool, but it’s pointless if the person can’t set it up correctly, fast enough, or understand the data!

#teamovertools

1. Convert Experiences

2. AB Tasty

3. Google Optimize & Optimize 360

4. Adobe Target

5. Kameleoon

6. Optimizely

7. SiteSpect

8. VWO Testing

9: Webtrends Optimize

Conclusion

So there you have it.

Our list of the best A/B testing tools offering multivariate testing in 2021. We also show which method they use to test, their features, pricing, and reviews.

Now you can compare the best multivariate tools across these different parameters and pick one that would work best for your testing needs.