Struggling to Prove the ROI of A/B Testing? Let Us Help

What is the value of your A/B testing program? What benefits does it bring to your company?

You already know A/B testing helps to optimize conversion rates and boost revenue. But how exactly can you prove A/B testing impacts your bottom line?

In the words of Matt Gershoff,

A/B testing can mitigate changes in behavior that lead to harm or loss

But all these may not impress the C-suite in your company, who want tangible figures that show the ROI of your A/B testing program. Proving the benefits of your A/B tests can earn you support from executives in times of upheaval, like now.

Here’s how you can prove the value of your experimentation program!

Calculating the ROI of Your A/B Testing Program

Using revenue, you can calculate the ROI of your A/B testing program. The folks at Peak Activity have a great formula for calculating your ROI. Here are a few metrics you should know about:

Revenue Per Session (RPS)

Calculate the RPS for both your control and variant. This is total revenue divided by the total number of sessions for your control and variant.

RPS = (Total Revenue ÷ Total Sessions)

Average Sales Lift

This will involve the RPS of both your control and variant. This metric has to be statistically significant.

Avg Sales Lift = (RPS of your variant – RPS of your control) – 1

Cost of Running Your Actual A/B Test

This is the control revenue multiplied by the average sales lift.

A/B test cost = (Control revenue * Avg sales lift)

Multiplier for Traffic Split

First, calculate the total sessions by adding the total sessions for the control and the variant. Then, use that figure to calculate the traffic distribution for the control and the variant.

Total sessions = ( Control sessions + Variant sessions)

Traffic distribution for Control = (Control traffic ÷ Total sessions) * 100

Traffic distribution for Variant = (Variant traffic ÷ Total sessions) * 100

Value of the Variant Change

This is calculated by multiplying the cost of running A/B tests with 2. This is because standard A/B tests tend to have two treatments. This figure differs for multivariate tests where you have multiple treatments.

Value of change = (Cost of running the test * 2)

Value Gained from Testing Duration

This is basically the cost of running the A/B test removed from the value of the variant change.

Value of testing period = (Cost of running test – Value of change)

Forecast Incremental Revenue If Variant Goes Live

To find the average sales per day, divide the value of the change in the variant by the number of days the tests ran for.

Average sales per day = (Value of variant change ÷ Test duration)

The revenue forecast will be the average sales per day, multiplied by the number of days in your forecast period. Remember to forecast far enough that the effects of seasonality, sales campaigns, customer behavior, etc. are diminished.

Overall A/B Testing Program Value

Now you can calculate the ROI of your A/B testing program:

A/B testing program = (Winning tests + Losing tests + Cost)

Note: losing tests only need the first 3 steps, while winning ones need all the steps.

We’ve asked several companies who have a mature A/B testing program to share how they calculate the ROI of their tests.

Here’s what they said.

Ian Kelly, VP of Operations at NuLeaf Naturals shared:

The first step is to figure out the total cost of the experiment. This includes the cost of the tools used for A/B testing and the time required to conduct the testing. The value of time is equated by multiplying the total time spent by the employees on the project with the hourly salary. Also, add the opportunity cost in case the test results in fewer conversions. Opportunity cost is the extra money you could’ve earned if you hadn’t conducted the experiment.

The second step is to have a clear funnel that ends in the sale of a product. EveryTrack every step of the funnel. Use a tool like Mixpanel to get this done. In an A/B test, there’ll be two variations of the same funnel. The changes can be audience targeted, colors used, or even the copy used. Both variations should be tracked separately and compared. 

Once all the above data is available, it will be very easy to prove the ROI of your A/B testing program. It is better to calculate ROI at the end of each quarter. Some experiments will fail while some will succeed. The overall improvement in revenue should be greater than the cost of your A/B testing program. If that’s not the case, reconsider your testing strategies in place

George Pitchkhadze, CMO atThrive Cuisine, outlines his company’s process below:

Calculate the cost of your experiment.

To calculate ROI, you need to know your costs. This should include direct and indirect costs, e.g.:

-The cost of software/services used.

-The cost of man-hours used.

-The direct costs (e.g. the cost of renting out an email list or PPC ads).

For example, let’s say I’m A/B split-testing two ad platforms. I should look at our PPC ad costs — but also at how many hours it takes to run each ad set and the price of analytics services we’re using to inform our ads.

Calculate your return

This part is straightforward. Calculate the direct returns that come from an effort. What matters here is using modern analytical tools to connect sales and revenue with things like emails sent, ad sets on FB/Google, etc.

Calculate ROI

Now that you have an accurate idea of returns and investment, you can easily calculate ROI. Just remember: calculate the cost of the experiment thoroughly and connect returns to specific campaigns and messages deployed

Other Ways to Measure A/B Testing Impact

There are other ways to measure the ROI of your A/B testing program.

Convert’s Head of Integrations, Data Privacy and Information Security Officer, Dionysia Kontotasiou, shared 4 more methods to gauge the impact of your A/B tests.

1. How Many Experiments Are Concluding?

If a high percentage of a company’s experiments are inconclusive, it might be a sign that they’re setting experiments up incorrectly. Perhaps they’re writing ambiguous hypotheses like “This change will make the app better” rather than explicit ones like “This change will increase monthly sign-ups by five percent”.

2. Product Team Feedback

Does the product team have more user insights and data than before? Are their product launches growing more successful? If yes, this is a good sign that A/B tests being conducted are successful.

3. Experiment Velocity

Testing velocity is the measure of how many tests you are performing over a certain time period. This is an operational benchmark to gauge how fast you can design, develop, test, analyze, and launch your tests. While some larger tests will take longer and are important for producing a bigger impact on results, smaller ‘quick wins‘ shouldn’t be discounted.

4. Projected vs Actual Costs to Run the Program

Track hours or budget devoted to running your program. You can also monitor your projected budget vs. your actual spend on your testing projects to assess the value of your A/B testing program.

If you’re using Convert Experiences for your A/B testing program, you have access to complex reporting (like in the screenshot below) that include your goals and each test variation’s performance — so you can not only maximize ROI but prove it too!

Conclusion

The value of A/B testing is both in finding improvements that lead to an increase in revenue or provide a better experience for your customers on your website and mitigating risks that may arise. Using revenue and other measures, you can conclusively prove the ROI of your A/B testing program to the C-Suite in your company.