Using Google Optimize & Other Free A/B Testing Tools? Shift Before September 30th, 2023…

Can an A/B testing tool truly be free?

No, not really. You may think that paying for the service is just a matter of handing over your credit card information but have you ever considered what else you might be giving up in exchange if somebody asks nothing from you upfront?

Sure, there are quite a few free and open source A/B testing tools out there.

But have you ever wondered if the reason you’re not paying for a service that costs money to build and maintain is that you are paying in some other way?

When you’re starting out, free tools are great. They help you understand and apply the fundamentals of experimentation.

But as you advance, there are things that’ll make you want to rethink your preferred A/B testing software. And that’s not because free tools don’t get the job done, but because there are limitations you’d want to shoot past.

Let’s show you…

Is There Any Such Thing as Free A/B Testing?

If money is the only factor you’re considering, then A/B testing can be free.

There are many tools out there that cost $0 to run tests and give you results.

But if you’re looking at it from a business point of view, factoring in other resources that contribute to success or failure, then you’d probably want to look deeper.

Many of these free tools are designed for first-time testers to get their feet wet in marketing experimentation. When these marketers want to test further or implement more advanced strategies, they quickly find that their free tools can’t perform these.

And if it was a freemium tool, you’ll see your path to advanced features blocked by a “sign up for a pro plan” notice.

But that’s not all.

For most free tools, you’ll be sacrificing more than just advanced features.

Free vs. Freemium A/B Testing Tools (With Examples)

One of the best free A/B testing tools is Google Optimize — the successor of Google Analytics Content Experiments.

Depending on when you read this blog post though, Google Optimize may no longer exist. Google announced that its optimization tools would be retired in September 2023 (as we reported here first).

Google Optimize was nice while it lasted. It allowed entry level testers to get started with:

• A/B, A/B/n, Multivariate, and Split URL testing
• Delivering personalized experiences to your web visitors
• Creating variants of your site with a responsive WYSIWYG editor
• Editing the HTML, JS, and CSS of your site in a basic code editor
• Integrating with Google Analytics for extracting valuable insights, and more…

Freemium tools, on the other hand, take it from where free tools stop. You start with a free plan that gives you basic features or use pro features for a limited time, and then you have to pay.

But it is usually worth it. These are common features you can expect from using them:

• No limits on the number of tests you can run at the same time
• No limits on your number of goals
• Ability to plan and execute more complex tests
• Run tests on apps and websites (not just websites)
• Integrate with many other marketing apps you use
• Customer support

And much more depending on the tool you’re using.

One example of this is Nelio, an open-source A/B testing software for WordPress and WooCommerce sites. It is free for 500 page views per month. If you have up to 5000 visitors, you’ll need to pay the starting price of $29/month. It comes with unlimited experiments and technical support.

Also, you should know there’s another type of free tool you can opt for — Open source A/B testing tools.

What’s different about them?

Free vs. Open Source A/B Testing Tools (With Examples)

Open source A/B testing tools provide some premium benefits while being free.

Some examples are:

1. Wasabi

This is a totally free open-source A/B testing tool by Intuit. So what’s the catch? It is no longer under active development and there’s no support. Also, you need some technical knowledge to deploy this 100% API-driven experimentation platform.

2. Vanity

Vanity also lets you run experiments on your websites and apps. But just like Wasabi, you may need a developer to help install and configure Vanity to run your A/B tests.

3. Mojito

It’s an open-source experimentation stack that lets you build and run tests with git.

4. Split

Split is a rack-based A/B testing framework that allows for maximum customization and extensibility.

But of course, there’s no support team. And that’s tough because deploying these tools usually requires a strong technical background.

It’s just you, the tool, and your experiments. If anything goes wrong or you don’t know how to take a certain action, you have to hope it’s in the documentation or there’s a help community/forum to run to.

Also, no one is obligated to get rid of any bugs you may find.

When Is Using a Free A/B Testing Tool like Google Optimize Okay?

Besides being a fantastic introduction to the world of website A/B testing software, free A/B testing tools like Google Optimize are great for:

• Proving that a new design idea works on your site
• Making minor changes toward improving your website conversion rate
• Getting assurance that a proposed idea will not lower your site’s current performance, and
• Generally, helping you make informed decisions that boost your bottom line with your existing traffic

But ultimately, optimization teams eventually grow past this.

Lorenzo Carreri, Conversion Rate Optimization and Experimentation Consultant, said:

The Google Optimize Debate

What Is Google Optimize?

Update January 20th, 2023: Google Sunset Officially Announced!

Google Optimize lets you run A/B tests to improve your website performance for free. It has a simple interface that anyone can easily figure out for their first test — and it integrates with Google Analytics perfectly.

It’s also compatible with Google Ads, Google BigQuery, Accelerated Mobile Pages (AMP), etc. And apart from A/B tests, you can run redirect, personalization, multivariate, and banner tests on Google Optimize.

Other great things about Google Optimize are that you can:

• Easily troubleshoot on your own with little technical knowledge,
• Define a detailed audience segmentation without stress, and
• View test reports in an easy-to-understand interface and terms.

This is perfectly suited to A/B testing for small businesses. But when you’re actively working on growth, you’ll quickly find that you’re limited in so many ways.

Since Google Optimize is a free tool, they don’t use a lot of resources. For example, you can’t upload a picture directly to the platform. It has to be hosted somewhere else.

When your needs grow past a certain point, Google Optimize is no longer able to handle your tests.

Other limitations are:

• You can only 5 tests at a time
• You cannot set more than 3 goals
• You can only test websites, not apps
• Your tests can’t run for more than 90 days
• In multivariate testing, the maximum number of variables is 16
• You cannot write more than 10,240 characters of code in Google Optimize’s code editor
• And if you’re not using Google Analytics, it can be a hassle to integrate Google Optimize with your “alien” analytics tool

Google Optimize wasn’t designed to be the beginning and end of your A/B testing. That’s why they offered the advanced 360 version at a custom price. Besides, looking at both Optimize and Optimize 360 as one service shows that, in essence, Google Optimize is a freemium tool.

Now that its sunset has been announced, switching to a paid option seems a good idea.

6 Reasons to Shift from Google A/B Testing Tools

As convenient as no-cost tools like Google Optimize are for A/B testing, there comes a time to make a shift.

We recommend making the switch when the 6 following issues start to impact test velocity or introduce opacity in how the results of critical tests are calculated:

1. A Too Generous Interpretation of Bayesian

In Optimize’s documentation, Google states that they’re using the Bayesian inference “to better answer the questions that marketers actually have.”

Here’s a bit of background on it:

Bayesian inference is a method of making educated guesses by using previous information to update the probability of an event taking place.

Here’s an example to illustrate this in simpler terms.

Imagine you were blindfolded and asked to pick from a basket of 3 oranges and 1 apple. The chances that you’ll pick an apple are pretty slim, right? 1 in 4, or 0.25, so a 25% chance of picking an apple.

What if you were told that the last 3 people who were blindfolded and asked to choose, all picked an orange? How does that make you feel about your chances of picking an apple now? Pretty confident, right? That’s because you know that there were only 3 oranges, and if 3 people picked them already, you have a 100% chance of picking the apple.

So you are mentally adjusting the probability of the event of picking an apple being true based on knowledge of prior events.

And that’s the framework Google Optimize uses — And many would argue that it lacks rigor and is too opaque, especially when it comes to “prior” selection!

Here’s what I mean.

In the documentation, their answer to “What is ‘probability to be best’?” goes:

Note that this is predictive and not based on final results. This is in addition to the fact that prior assumptions/beliefs are factored into your data. And Google Optimize doesn’t let you know what these prior assumptions/beliefs are exactly.

On the other end of the spectrum, we have a frequentist-based statistical methodology used by other testing tools.

According to Wikipedia,

Frequentist inference allows you to make clear “go” or “no-go” decisions after running well-defined random experiments with minimal risk of influencing the results with biases.

This is what Convert Experiences uses for its tests.

Google isn’t interested in offering all of its testing capabilities to everyone. This would be counterintuitive and lead to wasted money on resources such as servers which cost real dollars. Without feature limitations on the free Google Optimize, lots of people would be using it, putting immense pressure on the company.

2. Feature Restrictions

So it just doesn’t make sense for Google, which is a for-profit business, to do that.

If you’re just starting out, this doesn’t affect your optimization goals that much. But when you’re growing, the feature restrictions of Google Optimize will stifle your growth.

Like we mentioned above, these restrictions include a maximum of:

• 5 tests
• 3 goals
• 90 days runtime
• 16 variables, and
• 10,240 characters in the code editor

And there’s more…

You:

• Cannot test complex features
• Cannot add goals while the test is running
• Cannot customize a Google Analytics audience

Also, there’s no feature to push winners to personalizations. This allows you to validate these winners by attempting to replicate the lifts you got from the experiments without permanently changing things immediately.

Convert Experiences has that feature.

3. Lacking Support

The truth is, no business-minded developer will pay for a dedicated support team to serve non-paying customers.

So as far as support goes, the best you can get from Google Optimize is their resource hub.

On the plus side, it is very detailed. You get everything from tips, Google Optimize video tutorials, community, onboarding tips, etc. But there is no live support.

No one to call, no live chat, no email support. You can only get that when you opt for the paid option — Google Optimize 360.

4. SEO Impact as a Result of Testing

There are certain best practices for running A/B tests without negatively impacting your SEO.

But, if you run A/B tests using Google Optimize, you risk hurting your SEO anyway.

That is because of flickering — which is what happens when a user visits the page you’re testing and sees the original version of that page for a split second before their variant is displayed.

Not only does this impact the quality of your test results, but it also hurts the user experience.

And Google is big on user experience with respect to SEO. 

To solve this problem, Google Optimize has an anti-flicker snippet you can add to the page.

This anti-flicker snippet works by making your website elements transparent (or, invisible) until the variant from Google Optimize’s JavaScript loads. So, instead of seeing a split second of the original, the page remains white until the variant is ready.

But there’s a problem with this snippet that even Google Optimize recognizes with this warning on the “install the anti-flicker snippet” help page.

They say: 

Because it prolongs the page loading time. So, the users viewing the variant have a poorer experience than those who get the control.

So despite your effort to remove flicker with the snippet, your SEO still takes a hit.

Convert Experiences serves your variations fast, without any annoying flickering.

5. Tested Visitor Count Inflation

Your visitor count is the number of unique monthly visitors coming to your site and participating in your active tests. This count is not how many times (sessions) a certain visitor has visited your site, only how many of these unique visitors there are.

Google Optimize has a way of logging your visitor count that negatively affects your test results.

This Google A/B testing tool uses the Google Analytics rule where a session is measured as 30 minutes and can sometimes be stretched to 4 hours. What this means for you is that if someone visits your site at 9 AM and then returns by 2 PM (5 hours later), Google Optimize logs that as 2 unique visitors.

This makes your conversion rates lower than they actually are, making your results less accurate and impressive.

Why is this so? Google Optimize is a free testing platform. To minimize the use of server storage for keeping user data, they have to keep these sessions short.

So, if you need better results, consider a paid alternative that’s worth every buck you invest in it.

6. The Debate Around Sampling

Google denies any sampling occurs in Optimize, Optimize 360, Analytics or Analytics 360. It claims instead that it pulls unsampled data from Google Analytics data tables and doesn’t perform further sampling within the optimization app itself.

Here’s the official statement from their support team:

Optimize data is pulled from the underlying Google Analytics data tables, and is not sampled. And, Optimize does not impose any sampling in the Optimize interface. This means that any data you see in Optimize is unsampled, regardless of whether you utilize Optimize or Optimize 360, or Analytics or Analytics 360.

That said, seasoned optimizers are noticing a worrying trend of sampling on high-traffic sites (think millions of visitors).

Silver Ringvee, CTO of Speero, has found sampling to be an issue their team comes across on a daily basis:

At Speero, most of the sites we work with have a rather large number of visitors, generating millions of hits every month. Naturally, most of those sites have Google Analytics and not all of them have the premium GA360 version.

This means that sampling is a problem we deal with on daily basis, especially when analyzing the results of A/B tests using custom segments. Be it Optimize or some other tool, sampling is still a problem when it comes to analyzing this data in Google Analytics.

That being said, the numbers you see within Google Optimize’s reports are not sampled but those don’t allow for any advanced analysis. While data in Google Optimize reports isn’t sampled and it doesn’t say it clearly in its ToC, I believe that this data is still being affected by the data collection limits mentioned in Google Analytics ToC. This belief is supported by the fact that Optimize is based on the same dataset as is Google Analytics and we’ve seen some rather weird numbers in Optimize for sites that receive massive amounts of hits (100 million and more a month).

Why does sampling happen in the first place? It’s a way to save money on infrastructure or make reports load faster. However, it reduces the accuracy of results because you’re only looking at a portion of data from the original dataset.

Silver Ringvee of Speero has found a solution to circumvent sampling issues:

For any business-critical decision (like deciding the winner of an A/B test), I’d always wait longer and get my reports based on 100% of the data. Unfortunately, the free version of Google Analytics doesn’t give us this option. Sites that are receiving lots of visitors will have to deal with sampling. Especially when using custom segments that are very common for analyzing the results of A/B tests.

So, sampling may result in skewed data from your A/B tests that can lead you down the wrong path. If you get a lot of traffic on your website, consider moving to a paid tool that won’t mess up your data and give you accurate reports instead.

How to Choose An Alternative to Free A/B Testing Tools

You will hit natural stopping points with free solutions. And this can impede the results you get from your experiments.

But that’s fine. This only means you’ve advanced to the point where you’re no longer a starter.

And even if you’re not at an enterprise level yet, you can find the right tool that’s suited for that sandwich position you’re in.

Here’s how:

1. Figure Out Your “Why

Why are you choosing to switch over to a paid tool?

When you understand what limitations you’re looking to leave behind and the features you want to start using, it’ll help you make a great decision.

Are you trying to get rid of flickering? Are you looking to run more concurrent tests? Or are you searching for expert support as you scale your business?

When you’ve found the features that are fuelling your migration, the next thing to do is evaluate the available tools for certain criteria.

2. Do an A/B Testing Tools Comparison

Not all tools are created equal. Of course, features are one differentiating factor. But there’s more to this.

Even if they offer the same features for the same price, there are other below-the-surface characteristics that may matter to you and are worthy of evaluation.

You may have questions such as:

• What’s their stance on user privacy?
• What pricing model are they using?
• Do their features make sense compared to the cost?
• Are they showing care for the world?

3. Shortlist Your Preferred Tools

At this point, you may have found 3 or 4 tools that fit your idea of an ideal tool.

For some tools, your initial evaluation may hit a stone wall when you’re checking out pricing. Many A/B testing tools only reveal a custom price after many talks with sales reps.

When you’ve figured out your shortlisted preferred tools, test them with a free trial. And then, find your winner.

As you’re making the shift from Google Optimize (or any other free A/B testing tool), you may be walking into a more complex environment. Paid tools are more equipped and can be complicated compared to free alternatives.

That’s why your winner should come with great customer support. Whether it is live chat or 24/7 phone and email support, you’ll need reliable support you can always fall back on so that your work doesn’t slow down because of any challenge.

Go for a tool whose users speak of the support team this way:

• “Customer support team is fast, knowledgeable and solution oriented.”

• “Support is always friendly and helpful.”

• “Support is top-notch and very responsive.”

The chosen tool should have the right support team to help you make a smooth switch from free to paid A/B testing tools and scale your testing program.

The Shortcut to Finding an Alternative for Free A/B Testing Tools

If you need to find an alternative to a free A/B testing tool but don’t want to shell out the cash for it yet, check out our affordable Basic plan. The trial gives you free access for 15 days so that you can evaluate how well it fits your needs before making any purchase decisions. You can integrate with over 90+ tools and run A/B tests on your terms. No limits.

Conclusion

Free A/B testing tools like Google Optimize are ideal for kickstarting your experience with testing UI, UX, websites, apps, etc. But naturally, these are designed to have limits since they’re not bringing in revenue to the owners of the tools.

What this means for you is that when you need to do more than basic experimentation, the features of these free and open source testing tools will limit you.

To cross over this barrier and take your optimization to the next level, you have to go for the paid GO alternatives. The amazing thing is that you can get an affordable & powerful tool like Convert Experiences which delivers limitless features with a dedicated support team to help you maximize your ROI.