How to Choose an A/B Testing Tool in 2024

The A/B testing tool market is bursting at the seams in 2024, with over 50 options to choose from. And that number is only growing. The recent sunset of Google Optimize has created a void in the market, prompting a surge of new tools as vendors eagerly step in to offer their solutions.

You’re spoilt for choice. With all these new tools popping up left and right, how do you separate the wheat from the chaff and find ‘the one’?

You don’t need to break the bank or go with the category leader to have your testing needs met. Consider what kind of optimization challenges you face on the regular—the best A/B testing software for you should match your needs and budget.

In this blog post, we’ll outline the factors you should consider when making your decision and share our top picks for the best A/B testing tools of 2024.

Let’s help you out!

But first…

Understand the Optimization Puzzle

This State of Experimentation Maturity report identified 4 pillars of experimentation maturity. These are core aspects of experimentation that helped organizations across various industries succeed at experimentation while scaling their programs.

It’s interesting what we’ll learn here. Those pillars are

1. Process and accountability: This includes the process for ideation and prioritization, experiment design, how success is measured, etc.
2. Culture: What’s the organization’s general attitude towards experimentation? How much C-suite support is present?
3. Expertise: Time, resources, and skill sets. And finally…
4. Tools and Technology: The tech stack powering your experimentation.

Although this research was conducted a few years ago, the fundamentals remain unchanged. Tools are but a very small piece of your optimization puzzle.

If you’re reading this to find the best A/B testing tool that can singlehandedly take your testing program from 0 to 100 or okay to fantastic, then consider this a reality check.

What then should be your focus?

Arguably, people and processes take the cake when it comes to experimentation success. The brands and agencies at the top of their CRO game take training very seriously. And so should you too:

You should invest in your team more than just as a best practice in A/B testing; it’s often an expert-prescribed priority for organizations who want to take their optimization game to the next level.

This training goes beyond basic A/B testing tutorials and also includes research, data collection and analysis, critical thinking, experiment design, statistics, QA, stakeholder communication, and so much more.

Let’s put a pin on this idea because we will touch on it a bit more later.

Consider employing an A/B testing vendor as part of your education mix when searching for the appropriate tool for your team.

The right tool will provide education that corresponds to understanding what practices and frameworks offer the most efficiency for high test velocity (which is a program objective that many of the best minds in experimentation, including Ronny Kohavi & Lukas Vermeer recommend) in the context of that particular testing platform.

Since people and processes make or break your program with a greater degree of influence than your choice of A/B testing software, an idea that is gaining popularity in the industry is to spend as little as 20% of the optimization budget on tools and the rest on education, recruitment and Ops.

For example, agency WeTeachCRO believes that tool spend can and should be far less than 20%:

Conventional wisdom says not to spend more than 20% of your total CRO budget on tools, keeping 80% spare for people – people after all are the ones that will make the real difference to your programme, whereas the tool is merely an enabler. But putting that “wisdom” aside, our experience is that most clients can (and should) spend far less than 20%.

For the majority of our clients, their tool costs are <£500 / month with a people cost 10-15x that, putting them more in the 5-10% bracket. Starting out with a free tool like Google Optimize should get you off the ground, with a view to graduating up to something that is a few hundred pounds per month once your test complexities exceed its capabilities

Excerpt from 52 Questions Every Experienced CROer Had To Ask Once by WeTeachCRO

Now that we have established the appropriate mindset, let us proceed to the next vital factor in purchasing an A/B testing tool, particularly if this is your initial acquisition.

If not, you can skip to outlining why you need a new tool.

Do You Have Enough Traffic to Test?

A/B testing doesn’t work for low-traffic websites. Before you begin testing, your website must have enough traffic to send to each variation of your A/B test. Without adequate traffic, your experiments will not reach statistical significance and may run for a long time.

In a typical A/B test, you need to drive thousands of visitors to each variation. At Convert, we recommend sending at least 5K visitors to each variation and around 1000 conversions to produce conclusive results of which variant is the clear winner.

Head over to Convert’s A/B testing calculator to find out how much traffic you need to run your A/B test. In the event you lack the required traffic to the A/B test, head over to our guide on testing with low traffic.

Once you’ve sorted out the required traffic to successfully run an A/B test, outline what’s prompting you to get a new A/B testing tool.

Start With the Problem

Determine the specific need for a new A/B testing tool. It is crucial to have a systematic approach to accomplish this. Here is the process:

Map Out Why You Purchased Your Existing A/B Testing Tool

If you already have a testing tool in your stack, what led you to it?

Take note. It’s your Point A that will help you reach point B, your next ideal tool.

What specific features attracted you and your team to this product? What features kept you as a user? What aspects of this vendor did you particularly appreciate?

Don’t shy away from digging deeper. Understandably, there’s a strong emotional aspect in our decision-making process, but for the sake of a solid optimization program moving forward, you need to drill into your current choice.

Where Was Your Program When You Started?

Were you starting out in A/B testing and just wanted to get your feet wet? Did you hear about website optimization on a growth marketing podcast and had this tool recommended to you? Or were you on a free tool like Google Optimize and wanted to amp up your testing beyond its limits?

What Guardrail Metrics Were You Hoping to Move with the Tool Acquisition?

Guardrail metrics are meant to keep you aligned with your goals and objectives. They alert you when things aren’t going quite as planned.

So, what specific metrics prompted you to acquire the tool? Was it to improve revenue or enhance UX metrics such as bounce rate and order rate?

Did You End Up Achieving Your Goals?

If not, then why? Did your testing program pivot? Or was the tool—GASP—a bad buy?

Answer these honestly and you’ll have a clear map of the problem. That’s your point A.

You didn’t find any problem at all? Or you don’t have the answers to some or all of these questions? Who are the actual buyers and the heavy tool users? Having an informal chat with them will get you the answers you need.

Because you don’t want to change tools just because you have the budget and someone wants a fancy new feature.

Features hardly ever solve problems. They can, at best, help in the execution of solutions. And the shift should make sense for you:

• Adoption-wise
• Vendor support-wise: If you have a great rapport with your existing vendor, this means something. It is having a partner on the journey.
• Roll out and ramp up time-wise: Days lost adjusting to a new tool are the days you won’t be testing, either at all or at capacity.

Bottom line: Decide to shift if and only if you know ‌the move will tackle all the elements that make up the problem you’ve identified, as well as plug holes and bring additional value to your people and processes.

The tech will catch up to whatever bells and whistles you think you need if those bells and whistles enable a real opportunity and not trends. Don’t rock the boat for one feature.

Ask the Right People

Once you have established that there is a real problem that can be solved by experimentation or by changing platforms and not something either frivolous that won’t be worth the effort or a chimera (the ghost of operational and procedural chaos manifesting as tool trouble), then move on to building your consideration set.

To construct the consideration set, we must equip ourselves with accurate data to make a well-informed decision that encompasses all aspects. And as with everything, the pertinent data are your challenges, not subjective viewpoints.

You’d also want to avoid a common mistake most people make—going for the category leader. Category leaders are rarely the best fit for everyone. Neither should you default to the same tools you evaluated last time.

Because experimentation is complex. You need a platform that will support your specific program objectives and your experimentation workflows. While fitting perfectly into your culture, since all the pillars of experimentation maturity have to work in tandem.

Be open to choices. As Peep Laja says, they instantly improve the situation.

So, where to begin?

What B2B Buyers Say About Buying Tools—Here’s Where They Start

We ran a quick poll on LinkedIn. We asked B2B software buyers, “What is the first thing you do when buying software that costs over $5,000 a year?”

They answered:

• 57% do a Google search
• 21% research known brands
• 14% start on G2 or TrustRadius, and
• 7% ask for referrals

Additional opinions included those who will do a combination of all four. Though not a significant sample, it is fairly representative and gives us a glimpse into buyers’ preferred method to begin their tool search.

So, pick the tool with the strongest SEO? Not exactly. We’re taking this data with a massive pinch of salt. It illustrates that when we don’t have a clue about something, we’re kind of wired to turn to Google first.

While that’s a springboard for your search—giving you a list of A/B testing tools to consider—you need a more informed strategy. One that comes from the bottom of this poll: Ask the right people.

What We Recommend

Consult people who have used A/B testing tools extensively, your team, and also see what competitors or other players in your niche use.

Ask People in Your Network Who Test Regularly

Start with

• LinkedIn: Look into groups for conversion rate optimization pros. Some are more specific for SaaS, marketing professionals, and Shopify. Or make a post with the right hashtags (ex: #CRO, #abtesting #experimentation, and #optimization), tag the frequent A/B testers you know, and publicly ask for recommendations.
• Slack communities: Request recommendations on Slack communities for marketers such as CRO Growth Hacks, Growmance, and Women in Experimentation.

You’ll want to avoid Google results: Avoid Google search results for specific A/B testing platforms or to make a decision. You want a tool that will suit you, not the tool with the best SEO.

You can also…

Use BuiltWith to Look At Other Brands in Your Space

This works nicely if you are in a niche industry. The oft-recommended platforms may not have integrations with your tech stack. Or they may not support experimentation to improve aspects and metrics that matter to you.

For example, there is a lot of conversion rate optimization in the Shopify space, but most brands accept that struggling with revenue tracking is part and parcel of setting up an A/B testing tool.

Do you want to struggle with anything that matters to your experimentation program?

That’s why the A/B testing tool you’re considering should have specific provisions to accommodate your testing needs. Like how Convert offers a custom Shopify app to get rid of the revenue tracking problem.

To find what brands in your space are using, all you need is their website address to run a search on BuiltWith.

Or you can search the tools you’re considering and see who’s using them in your space. There are limits when you use a free account.

Tap Your Team

If you have a team that regularly conducts tests in your organization, they are one of the best sources of information for your consideration. They are actively involved in the process and will have a good understanding of the situation.

Get their opinion. Ask them why they wish to go with specific tools. What features are non-negotiable and which are just nice-to-haves?

What past experience trumps fancy feature releases? The best vote of confidence is one that comes from a tester who has succeeded in the past with a particular platform.

Gather this information and structure it. If you find a recommendation, do a quick check against your problem to ensure that the suggestion has merit.

Build Your Consideration Set

At this stage, you’ve shortlisted 3 or 5 tools that you’re seriously interested in. That’s a solid starting point. But right now, your decision may be based on intangible factors, referrals, and brand recognition.

When you build your consideration set, you will create a matrix of these 3-5 A/B testing tools and hold them against the components of a problem. That is, how they measure up in solving the major problems you’re looking to fix with the right A/B testing tool for you.

Let’s illustrate:

PROBLEM: Delays in experiment deployment.

If you’ve pinpointed more than one essential problem, you’ll want to have another consideration set for those as well. But with the same tools, only different components of the problem.

Note: Since all problems have different levels of priority, ensure that you’ve weighted them accordingly.

So, how do you know what to fill into a consideration set like in the example above?

Filling in the Gaps

• Reviews
  
  Start with popular software review sites like G2, Capterra, and TrustRadius. Remember: Sponsored profiles may be displayed more. Yet, this has nothing to do with how good a tool is at solving your problems.

• Cons
  
  Aim for those 2-4 star reviews. The cons there are usually well thought out. Dig through them for issues that might be devastating to your goals. See something you’re trying to avoid? Put an “X” there on the consideration set.

Take critical reviews with a pinch of salt, though. What went sideways for someone else isn’t certain to go sideways for you too. But if you notice a common denominator in 3-4 reviews, then there’s a pattern worth paying attention to.

• Comparison Tables
  
  Some of the work is already done for you. Look at comparison tables—both from vendors and from unbiased 3rd party sources like Speero or HubSpot and CXL.

We have a tool to help you compare and contrast Convert Experiences, Google Optimize, Optimizely, VWO, AB Tasty, Qubit, and more.

Interested in reading how CRO experts purchase A/B testing tools?

Here is what Ruben De Boer has to say about choosing an A/B testing tool:

First, of course, you have to make sure you have sufficient traffic for A/B testing, that at least a part of your company supports testing and that development has enough resources to implement winning experiments.

Next, take your current and future needs into account. Start by assessing the current situation, like your goals, needs, and who is involved. For example, if IT is fully involved, you could choose a server-side solution. Whereas if only marketers are involved, client-side is most likely the way to go.

Then, instead of only looking at the current situation, consider where you want to be in a few years. What needs do you have then? Will you scale up the number of experiments? Have more team members use the tool? Run different kinds of experiments? Discuss the possibilities to ramp up with sales reps of the various vendors.

Finally, conduct a free trial with the two-three tools that best match your current and future needs. During the free trial, check for the tool’s user-friendliness, (targeting) options, and other things applicable to your business (i.e., does it work with your single-page application).

Also, a personal connection is important, as you might need their customer support. For example, is there help with onboarding? If so, how happy are you with it? Also, contact their customer support. Besides checking the available channels (chat, mail, phone), see how quickly they respond (are they in the same time zone?) and how helpful they are. Finally, check with IT the tool’s impact on your site performance.

Ruben De Boer, Lead Conversion Manager at Online Dialogue

And this is Max Whiteside’s take. Max is the Content Lead at Breaking Muscle:

Define the testing scope, as doing so will enable you to restrict the testing capabilities of your A/B testing tool. Do you intend to conduct only A/B tests, or do you need a tool that can handle multi-page and multivariate testing as well?

Each tool operates in a unique manner and offers distinct test types. So, look for one that has all the features you need, so you don’t have to switch tools as your testing capacity grows.

Secondly, the budget A/B testing is an iterative process, which makes budget a crucial consideration when implementing a testing environment. You may need to repeat a test multiple times before achieving the desired result, at which point you can proceed to the next iteration. It necessitates both time and resources, and your budget will dictate its upper limits. Find a tool that won’t make a significant dent in your pocketbook.

Some testing tools may have a negative impact on the speed of your website while running a test. A slow-loading page can increase the bounce rate and irritate users, leading to fewer conversions. In such a scenario, neither the control nor the variation would perform optimally, rendering the A/B test results unreliable even if the minimum sample size is met.

Moreover, the flicker effect is minimal. The flicker effect occurs when a user briefly views the original web page before being redirected to the variant. Flicker can hinder user experience and skew test results. To maintain the reliability of your A/B tests, it is best to select a tool that does not produce a flickering effect. Determining whether the A/B testing tool slows down your website is therefore essential.

Max Whiteside, Content Lead at Breaking Muscle

Make Contact with the Tool

Up to this point, you’ve been a secret admirer of the tools you’re considering. Now’s the time to overcome your shyness and make contact.

And the best place to start is a free trial or demo.

Start a Free Trial & Book a Demo

You need to see the tool in action.

How does it perform in various A/B testing applications? As a CRO for a Shopify (or any other ecommerce) store, for instance, you’ll want to know how it performs from testing CTA copy for pop-up offers to product image tests.

A demo shows you how the tool solves your problem while a free trial reveals if the platform backs up the claims with the goods (i.e. features, pricing, service, roadmap).

Note: Most enterprise-level A/B testing tools won’t get on a demo with you before they’ve pre-qualified you. And they do not offer free trials. This can be a blind spot in your evaluation process.

Gauge the Culture

You might ask, “Why does culture even matter in the tool I’m buying?”

When you use an A/B testing tool, the vendor becomes part of your experimentation culture. That relationship works out best if you’re similar in business practices and values.

Don’t worry, this won’t narrow down your search to where you only have 1 or 2 tools to consider. A good culture is actually not rare. You just want to avoid friction that can get too uncomfortable and prompt a tool change too soon.

So, how do you gauge an A/B testing tool’s culture? You can look at Glassdoor reviews. People who work or have worked there can give a fairly transparent (although anonymous) insight into the culture.

Another way to assess an A/B testing tool’s vendor is their interaction with the industry. Do they truly educate the space?

What type of content can you find on their blog and social media pages? How do other players in the industry interact with them?

via GIPHY

Are they looking beyond their tool to help you with improving your processes? It’s more than just posting “our tool is great” content. You can clearly spot a brand that’s invested in the success of its audience. It’s hard to hide such a bright light.

Request Information You Are Missing

Yes, you can ask the A/B testing tool vendor to fill in the gaps for you with Request for Information docs.

RFI documents are similar, so a template like this would work. What’s going to be unique in yours, besides the introductory information about your organization and the goals you want to achieve, are the questions you ask. You’ll want to know

• The vendor’s track record
• The success story of customers similar to your organization
• Pricing
• References, and more

But also importantly, get the name and contact info of the person responsible for answering the RFI.

Pre Purchase Gear Up

Part of the effort in making a successful A/B testing tool purchase is in the pre-purchase gear up—the transition from no tool or current tool to fresh software in your tech stack. What do you need to do to migrate smoothly?

First, reveal your pick to your team and get buy-in. If you’ve not been working alone so far, and you’ll use the new tool as a team, they need to be on the same page with you. Let them know your choice and explain the reasoning behind it.

This is why having a structured approach helps. Now you can show the meticulous considerations you went through to land on this decision so that it makes perfect sense to choose this tool. With everything done right thus far, buy-in is easy at this point.

Discuss your decision as a team. How does it connect with the current problems in your experimentation journey? How does it support your goals for growth? What are other users saying about the tool? What did you learn from your research? And how did it hold up in your consideration set?

Also, focus on the “why” behind choosing the vendor. Beyond the sheer capabilities of the tool, are you going to be working with a vendor that supports your testing goals? Do they provide the educational backing your team needs to make the most of the new A/B testing tool? What are the vendor’s goals for their users?

Then, draft up a timeline for the transition and prepare to roll out. Migrating needs to be planned carefully to prevent hiccups and costly delays.

This is why you’ll want to work with a vendor that supports you. Will the vendor assist with the migration? Can you shift your data to the new vendor? Is there a robust onboarding process? Find out what expertise you need to get that done and how long it’ll take for each stage.

Finally, zero in on your goals for the optimization program broken down by 3-6-9 months. That way, you have quarterly checkpoints to assess the decision you made. Is the chosen A/B testing tool and vendor keeping up with the expectations you started with?

Careful here: Expectations need to be realistic. Make sure you don’t gauge your vendor on outcomes that they can’t influence.

Let’s make this an actionable checklist you can take with you.

A Handy Checklist of Factors to Keep In Mind As You Migrate

You want to migrate from good to better; smart to genius; amazing to blissful. That doesn’t happen by chance.

Tick all the checkboxes below so you can be sure the A/B testing tool you’re migrating to performs beyond your expectations and helps you achieve your experimentation goals.

• Cost — Does the cost of the A/B testing tool justify the results you’re expecting? How does the cumulative investment in the tool measure up to your expected ROI?

• The Learning Curve — You don’t want to spend valuable experimentation hours trying to figure out how to do basic tasks with your chosen testing tool because of a UI cluttered with tons of features you’ll never use. Read reviews to learn about the learning curve of the tool.

• Flicker — Flicker occurs when users see the original version of a page flash on their screen before the A/B testing software presents them with the variant. This can mar the results of your test.
  
  Some tools like Convert Experiences have developed anti-flicker technology, so this doesn’t happen. But many free and open-source A/B testing tools don’t yet have that capability.

• Out of the Box Integrations — Your A/B testing tool isn’t a lone wolf. It has to team up with other tools in your tech stack to bring you maximum value. Check out the number of out-of-the-box integrations your CRO tool comes with. Does it integrate with vital tools you need it to work with?

• Google Analytics — Ensure your tool has a smooth integration with Google Analytics. When setting up, this is one of the first tasks you’ll have to complete. Find out if you might need a developer to help with that tool.

• Goals — Goal setting is an integral part of your A/B testing. It adds context to your tests and illustrates the impact of your test results. How does your new tool handle this? Can you import goals from analytics?

• Revenue Tracking — When migrating, you want to ensure your revenue tracking is properly configured for both one-time and multi-revenue events. Some tools require manual setup. You don’t want that.
  
  Convert Experiences integrates with GA4 to pull values of transactions, revenue, and ordered items to inform the tests.

• A/A Test — Does the new tool pass A/A tests? If you have free trial access to an A/B testing platform, run this test. That’s how you know your tool is primed to handle A/B tests without biases—or not.

• Collaborator Settings — Particularly important if you work in a team. Can you control access to what your collaborators can view and make changes to?

And these 2 additional considerations to remember:

1. What Level of Customer Support Does the A/B Testing Tool Provide?

Another point of consideration in your assessment is the level of technical support available to you as a customer.

Buying an A/B testing tool may be an enormous commitment. You need to be sure that the tool provides enough technical support to your team in the event of difficulties.

The level of support offered by the top A/B testing tools on the market varies. With some research, you can find out if the testing tools you’re considering offer all or a combination of the following:

• Live support: offered by phone or video call
• Email support: offered by a dedicated email address.
• Live chat: using an inbuilt messaging app
• Documentation: an online repository of articles that provide solutions to common problems.
• Automated support: a set of pre-recorded responses.

Once you have confirmed the level of support available, you need to find out if the people stationed in the various support channels are CRO experts. It is especially important that you choose a tool with CRO experts for support personnel, as they can better provide the real-time help you need if you run into technical issues.

It is also worth noting that some A/B testing tools offer dedicated customer support as an add-on which you need to pay for or upgrade to a higher plan to access for free. Paying for dedicated technical support or upgrading to a plan that far outweighs your testing needs to access better support will increase the overall cost of the A/B testing tool.

At Convert, our team of developers and CRO experts provides technical support to our customers for free. Our support takes the form of live chat, online documentation, email support, and phone support. We also offer Convert Assistant, a ChatGPT plugin, that offers real-time answers to your questions by accessing our support documents and blog posts.

2. Does the A/B Testing Tool Impact Your Site Speed?

Another important consideration when buying A/B testing tools is their impact on website speed. Some A/B testing tools can add seconds to your website speed and increase load time. Slow website load time affects user experience and may affect your traffic.

Let’s get into the mechanism of how some of the top A/B testing tools on the market can impact your website speed. A/B testing tools usually fall into two categories:

1. Client-side A/B testing tools
2. Server-side A/B testing tools

Client-side A/B testing tools use a JavaScript snippet stored on a visitor’s browser to alter the content your server shows the visitor based on your targeting instructions. Server-side A/B testing tools deliver the exact content with alterations from the server to the user without using touching the user’s browser.

Server-side testing tools seem like the best option, but they can be expensive and not relevant for many use cases. You can delve more into the differences in our client-side vs server-side testing article.

Many of the top A/B testing tools out there are client-side testing tools. Client-side A/B testing tools load in two ways:

1. Synchronously

This is when the JavaScript snippet of your A/B testing software loads fully before your web page loads. This way of loading the scripts prevents flickering. But may cause your page speed to slow down, as the code snippet from your A/B testing solution load time adds to your overall page speed.

2. Asynchronously

This is when the code snippet from your A/B testing platform of choice and web page loads simultaneously. Tools that use this method of loading may be fast but are prone to flickering.

As I mentioned earlier, flickering occurs when the visitor sees the original content before the variation loads.

Convert is one of the fastest A/B testing software on the market. This is thanks to its Smart Insert Technology^TM which reduces its impact on your site speed.

3 A/B Testing Tools We Like

Where do you start in your search for the perfect A/B testing tool to buy? Right here with our favorites.

We’ve considered everything we talked about above and created a list based on what we know about our audience.

Let’s dive in!

1. Convert Experiences

2. Kameleoon

3. Optimizely

Key Takeaways

Finding the right A/B testing software for your needs isn’t as easy as it seems. Complex factors like your testing needs, pricing, integrations, available skillset, set-up, training, and more, need to be taken into consideration.

When you start with the problem, ask the right people, build your consideration list, and build a consideration set based on a careful analysis of the crucial factors that can make or break your experience with your A/B testing tool, you’re already halfway there.

You can choose to start your test with the tools we’ve listed above. But if that’s not your cup of tea, then we’ve given you all you need to go out there, research over 50 A/B testing tools, and find the right one for your unique needs.

Because the ultimate decision lies with you.