Using Meta-Analysis to A/B Test a New Layout or Theme Across Multiple Products or Websites

If you’re an eCommerce brand, sooner or later, you’re likely going to come across this scenario: Experimenting with a new layout or website theme that applies across different products or websites. To find out if the new element works, you could A/B test each product or website individually…

That should give you some ideas, but what if you wanted to mathematically determine the overall macro-effect of the new theme or layout? It turns out, there is a way, it’s a technique called Meta-Analysis.

What is a Meta-Analysis?

Merriam-Webster defines Meta-Analysis as:

A quantitative statistical analysis of several separate but similar experiments or studies in order to test the pooled data for statistical significance.

So how does this apply to testing a layout or theme change?

In the following paragraphs, I’ll walk you through the best possible way to do it.

The best way to test one theme or layout change across multiple products is to run individual A/B tests for each of your products/sites and then reason in terms of the average effect size.

Let’s see how this works in practice:

1. To make the following calculations valid, one should plan for the same sample size for each A/B test (with 1 variant per test) and ensure that all products tested are independent (one does not influence the other).
   

   

2. For the pre-test analysis, find the product that gets the minimum weekly visitors and then choose a sample size per product that is slightly inferior to that. That way, we can ensure that they all can meet this criterion.
   
3. Then, run a normal pre-test analysis with that amount of weekly visitors and choose the MDE that makes the most sense and that we know has a high likelihood of being achieved:
   

   

4. Run the test and configure the traffic allocation to 100% for the product that gets the least amount of visitors, and a proportional allocation for the other products, so that they all end up testing the same traffic over time.
   

   

5. Wait for each test to accumulate the predetermined number of samples (as calculated in our pre-test analysis) for the chosen MDE.
   
6. When this is done, run the Meta-Analysis. To do so, compute all individual Z values per test to get the overall Z using the formula below (where k is the total number of products/sites):

Using this method, we can get the overall p-value and see if the observed effect was significant or not.

Another way would be to deal with the average conversion rate and consider the total sample sizes across all tests. Then, get the overall Z value from it.

Definitely an advanced use case, but don’t get discouraged! It’s completely doable, following the steps I laid out above.

Have you tried this out or willing to try it? Reach out to me on LinkedIn and let me know what you think of this method and how it worked for you.