The Complete Guide to Multivariate Testing

Isn’t multivariate testing (MVT) just several A/B tests running at the same time?

No. That’s one of the most popular myths about MVT. It’s much more nuanced than that.

Multivariate tests are like A/B tests but you’re testing your control against multiple variants, not one. You see, it’s not just A against B, it’s also against C, D, E, and so on. And it’s an efficient way for high-traffic sites to find the combination of changes that performs the best.

So…

• When should you use MVTs instead of A/B tests?
• Is multivariate testing the same as A/B/n testing?
• What are some examples of multivariate tests?
• How can you run one to get reliable results?

We’ll answer your questions about multivariate testing, boosting your experimentation and CRO skills in the process so that at the end of this guide, you can deploy yours in 30 minutes.

If you’re looking to take your experimentation skills to the next level, then this guide is for you.

Sounds good? Let’s begin…

What Is Multivariate Testing?

Imagine you want to test different versions of various elements on a webpage at the same time to learn how they perform together, multivariate test is the test you run.

Multivariate testing (MVT testing) is a technique for testing multiple variables on a web page in different combinations. These possible combinations create more than one variant of the page, hence the “multi”.

So, multivariate testing is similar to A/B/n testing because we are testing two or more variations against the control. The difference is A/B/n tests are univariable, while multivariate tests test a combination of changes in tandem.

For example, if you want to test two different headlines, two images, and two-button colors on the page, your MVT test will look like this:

In the above MVT test, you’re testing different elements (headlines, color, and image) at once in different combinations to find the best one.

But that doesn’t mean it is not possible to do more than two variations of a single element. You can have 3, 4, or more, and test them at the same time. People just prefer one change so they can trace the uplift in conversion rate back to it.

Another important thing to note is that MVT testing is not the same as concurrent testing. Concurrent testing runs multiple experiments on the same sample set at the same time. And this can cause interference.

A concurrent test can look like this: Running an A/B test to find the best headline between two versions and running an MVT test to find the best combo of headline, hero image, and CTA or action button on the same landing page at the same time.

In A/B/n (or multivariate) testing, we are working with multiple variants serving the same user behavior hypothesis. Imagine many A/B tests on that landing page running at once to answer one question. Like, “Which version out of this lot gives us the best lift?”

Multivariate Testing vs. A/B Testing: What’s the Difference?

It may seem that MVT testing is a more advanced form of A/B testing to the untrained eye. But both types of testing are very different.

The difference between A/B testing and multivariate testing lies in the number of variants being tested at once. A/B testing will have a single variant tested against the control, while multivariate testing will have two or more.

A/B testing tests one element. For example, you’re testing adding an image to your web page. Your experiment may look like this:

• Control = No image
• Variation 1 = Image

In this case, you’re only testing the image on your website so it’s an A/B test.

In A/B/n testing, as in multivariate testing, you can have multiple variations. But the difference is, in an A/B/n test, you’re only testing one element (or variable), while in multivariate testing, you’re testing multiple elements in each variant.

Here’s an outstanding example from iProspect, a Convert Partner Agency. The agency ran a test that featured pricing page information. Their experiment featured:

• A control with no pricing information,
• Variation 1 with a low starting price,
• Variation 2 with a high starting price.

Can you guess what type of test this is?

In this case, we’re only testing one element, adding pricing information, so we have an A/B/C (or A/B/n) type of test.

When it comes to which is better between A/B tests and multivariate tests, that depends on what you’re trying to achieve. There are times when MVTs are more efficient than A/B tests.

If you want to test different elements on a page (and their interaction effects), it’s easier to use MVTs instead of A/B tests. With A/B tests, you have to create several consecutive tests for variations of elements. Meanwhile, multivariate handles them at once.

When to Run Multivariate Tests?

When you want to measure how one variant performs compared to your original, use A/B testing. This works if you’re changing one element, for instance, a different headline or a different button color. As long as those changes fall under one variant and you get A vs B.

When you have more than one variant tested against the control, but only change one element at a time, use A/B/n testing.

Let’s illustrate: You want to test if adding a headline to your landing page will improve conversions. You can do an A/B/n test like Split Base, a Convert Partner Agency. They ran a test to determine if adding a benefit-driven headline will increase conversions.

PS: It did! It increased the bottom line by 27%.

You use multivariate testing when you have more than one variant and more than one variable. Say, you have 2 more headline ideas and CTA options and that results in you creating variants C and D, MVT is the most efficient way to see how these changes interact.

“What if I used headline 2 with button color 3? How will that affect the number of sign-ups?” Those are the kind of questions MVTs answer — the effect on performance caused by page element changes when they work together at the same time.

Multivariate testing will go beyond testing one headline and test more variations of that page. It is more complex to design and run, so choosing it over A/B testing has to be worth it.

Oftentimes, A/B testing is enough to provide the answers we’re looking for. MVT is much more suited for instances when you believe one or more variants stand a chance of beating your A and B.

Some of those instances are:

1. You have a high volume of traffic and you want to gather more learning in a shorter period of time. Multivariate testing offers high-traffic sites the opportunity to test a combination of ideas in a shorter time, but these have to be high-quality ideas so as not to waste resources.
2. You need to know which combination of elements resulted in a conversion lift. Because a sequence of various A/B tests will just be redundant.

It is difficult to deploy a variant that has all the desired changes incorporated into it. Maybe your testing platform doesn’t allow that level of freedom. This is one way to work around it. But if it’s impossible to hard-code these changes later, you won’t be able to use your test results.

The Benefits of Multivariate Testing

Multivariate testing has an undeniable impact on conversion rates and revenue. Think about it. If it didn’t, nobody would talk about using it!

One benefit of MVT testing is that it allows you to find out how changes on your website affect conversion rates. And multivariate testing enables you to measure the interaction effect between these changes. You can measure the compounded effect of different small changes and see how that affects your conversation rate.

The Downsides of Multivariate Testing  

There are reasons only websites with a high amount of traffic and big budgets run multivariate tests. Let’s talk about the downsides of running an MVT test.

1. It requires a large sample size

The more element changes you’re testing, the more variations you’ll have. Each of these variants needs to receive sufficient traffic to reach stat sig, so you’ll have to wait for longer to get accurate results. That’s why a lot of websites can’t run MVT tests because they lack the traffic for it.

2. MVT tests take time

As we mentioned above, each variant needs traffic and lots of it. The more combinations you’re testing, the more different variations, and the longer you’ll have to wait.

But this is justified in most CRO cases. Because if you were to split an MVT test into a sequence of A/B tests, you’ll spend more time and traffic.

3. It costs more to run

You’ll need extra time (and money) to develop and QA each variant. That’s why your decision to run these tests over simpler A/B tests needs to be for solid reasons, or else your test ROI takes a hit.

4. You’re more prone to error

When you run A/B/N tests, you don’t apply the same statistics you use for A/B tests. The extra variants call for an adjustment of statistical significance tests. The increased familywise error rate (FWER) means you’re more likely to make a type I error and tag the wrong variant as a winner.

Examples of Multivariate Tests that Have Shown Big Impact

1. Microsoft ran a multivariate test on their SMB site and landed a variant that increased conversion by 40%!

2. In 2009, YouTube ran an MVT on their homepage to get more people to sign up for an account. They call it the “1,024 recipe experiment.” One variant won with a 15.7% lift.

3. Booking.com is known for testing. They use multivariate tests to learn how new features are used with a smaller audience before rolling it out.

4. HawkHost wanted to see lift on their homepage and they got it. These tested their headline, subheading, and hero image in a multivariate test that resulted in a 204% increase in sales. That’s huge.

5. Amazon’s landing page layout is a result of multivariate tests. They used it along with machine learning to find the optimum layout that drove conversion and hit a sweet spot that raised the purchase rate by 21% in 7 days.

Source

Multivariate Testing Statistics: Introduction to Multivariate Analysis 

If you’re curious about how multivariate testing works beneath the fancy UI of testing tools and down to the principles behind it, let’s thrill you:

Multivariate analysis (MVA) is a type of statistical analysis where more than one dependent variable is analyzed along with other variables at the same time. It is based on how the real world works: More than one factor is responsible for a certain outcome.

It is not only used in digital marketing (i.e. conversion rate optimization), as a part of exploratory data analysis, it is applied in various fields like healthcare, manufacturing, transportation, etc. It drives a lot of decision-making in the corporate world as well as in government.

Although it is complex, it is an accurate way to analyze the relationship between dependent and independent variables.

MVT & Traffic: How to Calculate Sample Size for a Multivariate Test? 

The best way is to use our sample size calculator tool. This will provide everything you need to get started with designing a healthy multivariate test.

When to Stop a Multivariate Test? 

Stopping a test too soon can lead to errors, as the data gathered wouldn’t be sufficient to make any accurate analysis.

You may have heard some people say you should stop when there’s a clear winner, or stop a variation if its conversion rate drops below 10% and move its traffic to the top performers — rest assured, this will significantly contaminate the results.

You should never stop a multivariate test before:

1. Collecting a large enough sample size for each variant. So, let it run for longer to collect enough conversions to be relevant.
2. Running 1 or 2 full sale cycles to balance out traffic and shopping periods over 30-60 days.
3. Reaching statistical significance. That means when pushed to live, it should be able to replicate its results with 95% accuracy.

How to Design a Multivariate Test?

A/B testing and multivariate testing are similar in how they’re designed and conducted.

We outline a step-by-step testing methodology to run an MVT below. But here’s a quick take:

1. Identify a problem and formulate a goal: When you go through your data, try to understand what website visitors experience, and find a problem. Formulate a hypothesis (a solution with a goal) and base your test on this.
2. Decide on what to test: Your hypothesis will involve a combination of webpage elements. Figure out what you would do with them to achieve the goal. Create variants of this to go against the original.
3. Estimate your timeframe and sample size: You don’t have to do this yourself. Use our pre-test sample size calculator. It’ll help you figure out the essential numbers you need.
4. Build the test: In your testing platform, set up your variations in their combinations.
5. Ensure it works: Run quality assurance tests on each new treatment to make sure they work as intended.
6. Run your test: Start driving traffic to them. 

Now, let’s take a close look at each step — from data collection to QA:

Data Collection for Multivariate Testing

How is data gathered during multivariate testing? There are two major approaches to pay attention to:

1. Full Factorial

Here’s how it works: Every variant in your MVT gets equal amounts of all the traffic coming to your site. If you have 5 variants, each of them will get 20% of the traffic coming to your site.

Of course, to reach statistical significance, this method will require a lot of traffic and will have to run for a long time. But this trait also makes full factorial is the most accurate option to go with.

2. Fractional Factorial

Unlike its “full” counterpart, fractional factorial only tests a subset of your variants. That’s why there’s “fraction” in its name.

If you have 10 variants, it will test 5 of them, and based on the results, predict the performance of the 5 it didn’t test.

Because of this, it runs faster than full factorial but is significantly less accurate. Its benefit is that you get a glimpse of what the full test will look like before you commit.

What about the Taguchi method? Some say it’s another name for fractional factorial. It is complex to use. So, most tools don’t offer it.

Hypothesis Generation for Multivariate Testing

You must create your multivariate testing hypothesis based on accurate qualitative and quantitative data.

You don’t want to guess or use common testing ideas you found on a blog to run your tests. Choose something relevant, like solving a problem you found while sifting through your data or testing an idea you think can produce lift.

This will serve as the goal of your test. You can use our hypothesis generator to help you come up with a solid one.

Quality Assurance for Multivariate Testing

Without quality assurance, any test can mislead. Advancing without QA is risky and can cause expensive mistakes. Before you trust the insights of your test, pass it through several checkpoints:

• QA set up: Where you decide on the set-up to use for your QA process, identify special use cases to test the user experience on the devices and browsers you’ll be QAing, etc.
• External factors: Recognize factors you can’t control that may affect the test like events or holidays, other tests running at the same time, events, and site traffic.
• Experiment setup: Check the quality of the hypothesis formulation, traffic allocation, if flicker will affect the test, other delays and errors, etc.
• Cross-browser & cross-device: You need to verify typography, fonts, and color scheme across devices and screen resolutions, verify margins and paddings, check usability, how page elements perform on various devices, and more.

And other discrepancies you may have detected: It’s necessary to share the word about this with your team for fixes.

How Do You Run a Multivariate Test? Deploying a Multivariate Test in Convert Experiences.

Creating a multivariate test in Convert Experiences takes only 4 simple steps. Here’s what to do:

1. Name your Experience
   
   In Convert Experiences, select “New Experience”, then “Multivariate”, and name your experience:

   

2. Make the variation
   
   After the URL of the page you use for the test loads, the first variation is ready to be edited. You can edit the copy, add new visuals, you name it. In the example below, we’ve:
   
   – Clicked on an element to change (it’s highlighted in orange)
   – Selected an action in the menu i.e. changing the image source

   

3. Name the variation
   
   This is pretty self-explanatory. You’ll want to pick something that will stand out to you in reporting later on.
4. Define the goals and audiences
   
   Go to the Experiment Summary, and set up your Audience and Experiment Goals.
   
   There you have it! Your experience is up and running.

Now that you understand what multivariate testing is and how to successfully set up a multivariate test, let’s break down a few different platforms you can choose from. Many of the top A/B testing tools on the market offer multivariate testing and A/B testing in one software. So which one is best suited to your needs?

If the multivariate testing method is a deciding factor for you, you’re in the right place. The list below gives you the 9 top players for multivariate testing on the market.

What Are the Best A/B and Multivariate Testing Tools Available?

Here’s a look at the multivariate testing tools preferred by CRO experts, along with the multivariate testing methods they offer:

• Convert Experiences – Full Factorial
• AB Tasty – Full Factorial
• Google Optimize & Optimize 360 – Hybrid approach that’s neither Full Factorial nor Fractional
• Adobe Target – Full Factorial and Taguchi
• Kameleoon – Full Factorial and Fractional Factorial
• Optimizely – Full Factorial, Partial and Taguchi
• Sitespect – Full Factorial and Fraction Factorial
• VWO – Full Factorial
• Webtrends Optimize – Full Factorial and Fractional Factorial

Still not sure what multivariate testing tool to pick? Check this complete breakdown of the best Multivariate Testing tools.

Top Multivariate Testing Mistakes to Avoid

These tests can only help you achieve your optimization goals if you’re using them the right way.

But then we’re human and our tools aren’t perfect, so mistakes do happen. You can reduce the chances of ruining your tests by avoiding these:

1. Not testing your MVT tool for accuracy

Create variations of your page, but don’t change anything. Keep it the same and run the rest. Since people are seeing the same thing, the results should be the same.

2. Not starting with a solid hypothesis

Running a test without a baseline and what to expect is running blind. It doesn’t help you or the site you’re testing on. Generate a hypothesis first.

3. Copying your competitors

Your organization is unique and your goals are different. Copying your competitors is not beneficial. Not only will many of their tests not apply to your case but they might not know what they’re doing.

4. Running tests during seasonal events or major site/platform events

If your test isn’t meant for an event, you shouldn’t be running a multivariate test during one. The results will not reflect the accurate performance of a regular day.

5. Peeking

The temptation is hard to resist. You just want to see how it’s going. You maybe even placed a bet on a variant and want to see which is winning. Don’t.

Not only will you get inaccurate expectations, but you could also make wrong decisions based on them. You can check to find out if they’re running or if there are any errors, and that’s it.

6. Not running your tests long enough

Multivariate tests need to run for a long time. Think of it like separate A/B tests having to get enough traffic to reach statistical significance. Let your multivariate tests run long enough so you can get accurate and significant results.

7. Making changes while the test is running

You can fix errors, but you cannot redesign your website or make copy changes. Anything that changes the look and feel of the variants should remain untouched after the test goes live.

8. Forgetting to stop a test after you get accurate results

It happens more often than you think. What this means is, after you have a clear winner, you’ll still be showing the rest of your traffic a weaker version of your site. You don’t want to do that.

9. Giving up after one test

Maybe all your variants turned out worse than the control, so what? You’ve just learned various ways it doesn’t work. Run more tests and get more insights. All knowledge is valuable.

10. Not checking the validity after the test

When all is said and done and you’ve gotten your results, is that the end? Do you shut down the testing platform and move on? It’s best practice to sit back and check if everything went smoothly before trusting the accuracy of the results.

11. Not running enough tests

When you run a test, whether it was a winner or loser, learn from it. Use your newfound knowledge to create a more informed hypothesis and run again. And again. That’s how you make experimentation work for you.

12. Not documenting tests

Keep a database of your tests. This will improve the performance of your optimization team, even if that team is just you. We all need to learn from previous experiences to make the next one better.

And also, to avoid wasting time on running the same tests twice.

Conclusion

Whether you have an ecommerce store, SaaS, or a simple landing page, there are scenarios where it makes sense to run a multivariate test. You may have a high-traffic website, but you still need to decide if multivariate testing is right for your website. Evaluate the cost, traffic, and need for a multivariate before you run one.