Changing Your A/B Testing Tool? Here are 12 Important Factors to Consider

Changing a core part of your marketing stack is never easy. Balancing product features, pricing, and great customer service makes choosing a new A/B testing tool challenging.

The push to change tools may come from outgrowing your plan on your current tool, but upgrading will cost an arm and a leg. Your optimization stack may have grown more complex with tools that need to integrate with each other, and your testing tool hasn’t kept up. Your testing tool may be causing flickering and ruining the experience for site visitors. Or maybe you are tired of dealing with unresponsive customer support.  

Whatever the push behind changing your A/B testing tool is, the fact remains that you need a new testing tool. The best A/B testing software is one that sits right at the intersection of great pricing, features and customer support.

Here are factors to help you select the best A/B testing tool for you.

The Cost and Pricing Model of the Tool

It is common practice to find that CRO tool providers no longer list pricing on their websites. This makes choosing a testing tool harder, as you’d have to contact their sales team to get a quote. 

Even with the hassle of calling the sales team for each tool on your consideration list, you still have to factor in the pricing model of these tools. Some tool providers use a pricing system that forces customers to either upgrade and pay more or risk being phased out of their tool.

In 2018, Optimizely took down the pricing information on its page and pivoted to providing services to only enterprise accounts, leaving many businesses who could not make the minimum investment of $50,000, later $100,000, in the lurch.

In 2019, VWO went the same route and removed the pricing page from its website. Then they got rid of the monthly pricing for small businesses. Their new annual pricing started at $12,500 with 100,000 tested users per month. Compare that with the previous monthly plans billed annually at $6,000 (billed at $500/mo) with 200,000 tested users monthly and $8748 (billed at $729/mo) with 300,000 tested users monthly, respectively. Many businesses lack the budget to accommodate such an increase, as they will get less value for their money.

When researching testing tools, pay attention to the provider’s pricing practices. Do they offer great grandfathering in case of pricing changes? Can you find pricing information on their website without contacting their support?

At Convert, we believe in transparent pricing and respectful grandfathering. You can review our pricing page to see our different plans. We offer both monthly plans (billed monthly or annually) that you can opt out of with ease.

It is true that you get what you pay for with CRO tools, but that isn’t always the case. The most expensive tools may not offer the best value.

Impact of the A/B Testing Tool on Page Speed

A slow page will cost you traffic. Traffic you can send to your tests. Internet users expect webpages to load in mere seconds. A 2017 Google study analyzing mobile page speeds says the probability of a bounce increases by 90% if your website takes up to 5 seconds to finish rendering. Because Google is in the business of providing amazing user experience, this study spurred action. Google announced it will now use page speed as a ranking factor for mobile searches in 2018. The search engine already uses page speed as a ranking signal for desktop searches.

Many internet users use their smartphones to browse websites. In 2019, the number of smartphone users surpassed 3 billion people. This means a part of your visitors are coming from mobile searches or using their smartphones to access your website. This should send shivers down your spine when you consider those Google study findings and how the search engine may ding your webpages in the SERPs because it loads slowly.

Do you know what could cause your webpage to slow down?

A substandard A/B testing tool.

Because many A/B testing tools change the source code after the page has loaded to serve the test, it can cause a slight delay in page loading time.

Now that you’re switching tools, consider the impact of your new tool on site speed. Your new testing tool should have minimal impact on how fast your site loads. This CXL study on popular A/B testing tools and their site speed impacts is a great place to narrow down your search for new testing software that has little to no impact on your site speed.

The Tool’s Integration into Your Current Marketing Stack

As your company grows, so does your marketing stack. You may have gone from using few Martech like analytics software to having quite a lot of software tools that need to integrate with each other to make marketing magic happen. Your current A/B testing tool may have failed to keep up with the changing times. It sticks out like a sore thumb. It may also hold you back from upgrading your stack because it may not integrate with new software you need.

As you take the step to change CRO tools, consider its integration capabilities. How does this new tool fit into your current market and its impact on your future growth? Does the tool you’re considering add new integrations often?

Your A/B testing tool should be able to anticipate your needs by adding new integrations that will make life easier for your business. Convert Experiences comes with over 80 integrations including Hotjar, Google Analytics, Formstack, Active Campaign, etc. And we keep adding more!

When you choose a testing tool without considering its integration capabilities, you will end up where you started: still looking for a new CRO tool that actually works for you.

Is the A/B Testing Software Inclusive of all Technical Ability Levels?

In an ideal world, all the members of your team know their way around a code editor and can change any elements needed for tests. Sadly, the world is not a testing utopia.

Because many marketers lack technical knowledge, they still need a developer to set up complex tests.

But what about the “simple” tests?

Many CRO tool providers tout how great and efficient visual editors are. But when you use them, you discover that isn’t the case. You still need the services of a developer to set up all the tests. Some CRO tools treat the WYSIWYG editor as an afterthought. Hijacking the company developer to help create tests every single time is untenable in the long run for the average marketing team that typically lacks a dedicated developer.

CRO tools with a robust visual editor enable you to run tests with no developer.

The new tool you’re considering should be easy enough to use for both the less technically knowledgeable users and the developers on your team. Its visual editor should be robust and easy to use for non-technical members of your team.

The Level of Support Provided by the CRO Tool

Support is a crucial part of any complex piece of software. And testing tools are no different. Given how much they cost and the complexity of the function they perform, proper support is an important consideration when choosing A/B testing software.

Many CRO tools offer support in the form of live chat and calls, email support, documentation and automated responses. But not all support is created equal. Long wait times to get a response to queries that may delay tests and less than knowledgeable customer support agents are constant complaints in the industry.

Before you switch tools, examine the support the tool offers.

Do you get 4x faster support response than the industry average (as you get with Convert Experiences)?
Do you have easy and quick access to CRO experts who can answer all your questions when you run into issues?

Is there clear and thorough documentation for you to refer to?

Consider the robustness of support you get after onboarding before you commit to any tool!

Can the Tool Handle Unusual Pages and Unstructured URLs?

One vital factor to evaluate when choosing A/B testing software is how it handles unusual things like pop-ups and hovers that rarely change the page’s URL. How does the tool handle testing with pop-ups, especially when you’re testing different variations of them?

Does the tool test unconventional pages like shopping carts and checkouts whose final form depends on user behavior? What about testing on unstructured URLs which you find on many ecommerce websites?

For example, Convert Experiences recognizes the same user session across multiple tabs which enables us to carry out tests on carts and checkout pages. Our tool also uses regular expression capturing to identify parts of URLs to make bulk testing simple.

If you run an ecommerce website or any business that uses pop-ups, it is important to ask how any CRO tool you’re planning to switch to handle these curveballs.

The A/B Testing Software’s GDPR Compliance

Privacy has become a hot topic in the technology space since the GDPR passed in 2016. When the compliance deadline rolled around in May 2018, the discussion around privacy intensified. With the Facebook-Cambridge Analytica data scandal fresh on internet users’ minds, it is imperative that you use tools that respect the privacy of your visitors.

Many CRO tools are not GDPR-compliant and this could be a problem for your business if you get any traffic from countries in the European Union. The EU heavily fines non-GDPR compliant companies even outside the EU. Fines can get up to 4% of annual sales or 20 million euros depending on which is higher.

A GDPR compliant tool will save your business a lot of headache in the long run. Some A/B testing tools like Convert Experiences are GDPR compliant. At Convert, we bucket visitor data in large groups and don’t store individual data records. This makes it difficult to trace any data to one person. This handy infographic by Convert CEO, Dennis van der Heijden, will aid you in picking a GDPR compliant tool.

The Testing Type the Tool Runs

Every A/B testing tool runs tests differently. Tools may run one-tailed or two-tailed tests. Each type of test has its own pros and cons. Tools that run one-tailed tests require less traffic and reach significance faster. If your website has low traffic volume, this may work for you. Note that one-tailed tests are prone to give inaccurate and biased results.

Two-tailed tests, like the Z-test two-tailed testing engine at Convert, produce more accurate results. This test also mitigates against Type I errors (false positive) and cognitive bias errors. The downside to two-tailed tests is that they require a larger traffic volume to declare winners and take more time to reach significance. Remember, testing is not a 100-meter dash but a marathon.

As you review each tool, decide on what trade-off you’re willing to accept: faster results achieved with less traffic (which may be inaccurate) or accurate results you can trust that need more time and traffic.

Its Revenue and Third-Party Goal Tracking Capabilities

Your business may need to track revenue, third-party goals or both. A testing tool that not only tracks your revenue but integrates with your analytics software is invaluable. This way you can have all your data, including revenue, in one spot.

Again it comes down to integration with your analytic software. Many companies use Google Analytics, for example, as their analytics solution. If your CRO tool does not integrate with Google Analytics or its integration is buggy, it will be difficult to track your revenues in one dashboard.

Another area where some testing tools fall short is third-party goal tracking. You can test your lead generation efforts across third-party and affiliate websites, which may be a huge deal for your company. The unavailability of this feature will stymie your efforts to test your lead generation process.

Convert Experiences automatically integrates with Google Analytics (both Universal and Classic versions) to track your revenue in one dashboard. We also offer third-party goal tracking. A simple code placed in the header, body or footer of third-party sites increases your testing potential several folds!

Any tool that doesn’t offer these features which are a core part of your business may not be right for you.

The Targeting and Segmentation Capacity of the Tool

The targeting and segmentation capacity of any A/B testing tool is an important factor you need to think carefully about when switching tools. Your tool should offer advanced segmentation and targeting to enable you to serve tests to specific groups based on the data you collect. 

For example, your site has a running shoes product page that gets the most traffic in the afternoons on weekends. With a great testing tool, you can segment this traffic according to what they search for, whether they clicked on an ad, the time and day they visit and more. Send them to tests you’re running on the product page and gather valuable insights that will drive improvements and conversions.

Segmentation allows you to divide traffic according to specific traits like geolocation, time of day, etc. Behavioral targeting analyzes visitors’ behaviors and serves them test variations based on that information. This targeting provides a window into understanding a customer’s behavior and needs based on traits like traffic sources, time on page and more.

At Convert Experiences, we have an advanced segmentation and targeting engine that lets you divide your traffic based on location (up to city deep), language, time of day, weather and more. Any tool you consider should offer you targeting and segmentation that takes your tests to the next level like we do.

Does the Testing Tool Cause Flash of Original Content (FOOC)?

Blinking is a scourge that plagues many CRO tools. Blinking happens when a user sees the original page before the test variation loads. Flickering happens when your testing tool loads its script simultaneously with others on your webpage. Sometimes, the original content on the webpage loads before the test variation and this causes the flickering.

FOOC can turn visitor user experiences into negative ones. It also affects your tests as you won’t be certain of your results unless you isolate and eliminate the flicker effect. Certain CRO tools load their script asynchronously to reduce page load times, but this causes blinking. This provides a bad experience for users.

Other tools load their script synchronously: this means their scripts load fully before the webpage loads. This may cause a second or more increase in page loading time.

Unlike other synchronously loaded scripts, Convert Experiences is super fast. This is thanks to the Convert’s SmartInsert™ Technology that loads the Convert Experiences script as a layer between the browser and website, thus preventing flickering with no negative effects on page speed.

When evaluating a new CRO tool, you should consider how the script loads: think synchronous vs asynchronous. This will enable you to make a decision that provides a great user experience to visitors and get accurate test results.

Other factors

Changing your A/B tool doesn’t have to be daunting. Before you start your research, consider other factors like:

• Is the reporting robust? Can you send export reports as PDFs or spreadsheets? How about integrating with your analytics dashboard so you can keep your data in one place?
• Can this testing tool handle server side testing?
• Will you be able to test on single page applications? Will this tool handle testing in SPA like a champ?

… and more.

Once you’ve made your pick, don’t forget to put your chosen tool to WORK. Here are 6 ways to use your A/B testing tool for optimization success.

Your Turn Now

Now that you know what to look for in A/B testing software, why don’t you take Convert Experiences for a free 15-day spin?

That’s right, sign up for a free 15-day trial and experience the Convert tool for yourself! Best of all, easy opt-out and no credit card required.

Happy testing!

SIGN UP FOR A FREE TRIAL