A/B Testing in 2020 & Beyond: What is Changing and What Should You Do

The A/B testing landscape changes fast.

Everything — right from what goes into the testing mix to the process of testing and the technology powering it — has a tendency to evolve rapidly.

You’d recall how just a few years ago, concepts like personalization, advanced server-side testing, testing via platforms such as Google Tag Manager and others were mere predictions.

But they’re all here, and shaking up the status quo.

In some time, AI could get mainstream as well. You already have Microsoft Clarity in the making.

So what pushes these changes?

Sometimes, it’s the market.

Sometimes, it’s the experimenters.

And sometimes, it’s the CRO tech solution providers.

So let’s look at how the A/B testing landscape is changing and what you can do in 2020 and beyond to adapt to it!

A/B Testing: What Lies Ahead

A/B testing has been the go-to way for conversion optimization.

You can attribute its popularity to the fact that it gives decisive results even with relatively simple experiments, as compared to the other CRO techniques like multivariate or funnel testing.

But this is slowly changing as businesses become more aware of the “segments” in their audiences.

Let me explain…

Earlier, marketers would be happy to test two value proposition options for their home page.

A classic unique value proposition A/B testing experiment.

Period.

But now they want to do more. Much more.

They want to find out if the winning version performs well across every audience segment.

For example, if a university website offers both graduation and post-graduation degrees, then in-house marketers would want to know the best homepage headline for each audience segment.

This is nothing but personalization.

And data driven marketers are looking to A/B tests to find results that’ll up the ROI of their marketing efforts in the form of personalized experiences.

David Boronat, CEO at Multiplica and Business Angel, predicts how personalization will be the key trend powering proactive web experiences in the future: 

The most important trend I can foresee is about personalization and how we are going to design proactive websites. Sites that become relevant, accurate and timely. Sites that actively search for their users according to their behavior and their micro-moment of the purchase journey.

We can see a proactive site when it tries to geolocate its users and shows a product according to their location. Or when it recognizes its users and offers what they need according to their profiles or past behaviors.

Contextualization, recognition, profiling, customization, segmentation, recommendation, proactivity, personalization, inference or multi-channel experience would be some of the strategies we would use to make our sites commercially smart.

What does a/b testing have to do with this? It is going to be easier to implement those tactics on our sites using platforms such as Dynamic Yield, Monetate or Evergage, easy-to-use tools that allow marketers to create, test and deploy an unlimited number of personalized digital experiences with limited need for IT or consulting resources.

This trend would impact a/b testing platforms that don’t allow you to create audiences, apply business rules or run campaigns and personalized experiences.

David Boronat, CEO at Multiplica and Business Angel

Michal Parizek, Head of Product at Rohlik.cz., author of Conversion Rate Optimization Maturity Model, and a passionate growth marketer, also tells how those who are confident with running A/B tests will forge into advanced targeting and personalization:

Peers who have already successfully launched several A/B tests will try to leverage it and will experiment with advanced targeting and personalization. In case of a lot of e-commerce websites there is a huge potential in personalization and a lot of companies will dig deeper into the topic.

It will be interesting to see how the tools market will change. Google Optimize aims to disrupt it and control it as much as it has done with Google Analytics several years ago. How will Convert.com and many other react? One thing is sure, there is one subject who usually benefits from fierce competition. And that’s you, a customer.

Michal Parizek, Head of Product at Rohlik.cz., author of Conversion Rate Optimization Maturity Model

At Convert Experiences, as the makers of a world-class testing tool, we saw this coming many years ago.

In fact, we moved away from being just an A/B testing tool to a full-blown personalized experiences solution because we knew personalization was here to stay, and that A/B testing would be used along with personalization down the lane:

This is not a simple rebranding of an A/B testing tool to an Experience tool, we are rebuilding all of our elements from Audiences to Segments and will be adding more enterprise integrations to other marketing tools. With today’s release we are committed to this exciting path and implementing Personalization to the over 5000 websites we currently collaborate on.

Dennis van der Heijden of Convert Experiences, on the move toward powering personalized website experiences (July 2016)

Ever since the switch, we’ve actively developed our personalization prowess because we’ve had our customers wanting to try ambitious ideas.

Today even with support for personalization-based on real-time behavioral targeting, DMP profiling, provisions for easily managing and prioritizing overlapping audiences, and more, we still feel like there’s a long way to go.

But experiments like A/B tests and personalizations don’t just have winners and losers, they also have an impact on other marketing areas such as SEO.

Sure, there are more questions than answers here, but these growth tactics don’t work in isolation.

And so there’s a need to monitor how such experiments influence a website’s SEO performance and more.

Duncan Heath of Fresh Egg, who is a conference speaker, author, trainer and consultant on the subjects of UX and conversion rate optimisation and the organizer of the very popular BrightonCRO conference (now in its 6th year), explains how clients often consult him on the impact of such CRO campaigns on SEO:

I’m increasingly being asked by clients about the impact that conversion optimisation and personalisation have on SEO.

These three disciplines continue to converge as search engines are able to capture ever more data on the experience users have on website (e.g. bounce rate, time on site, pages per session etc.). This is not just happening on an aggregated level, but an individual level as well – many people are logged into Google Chrome whenever they’re browsing, even if they don’t know they are.

Search engines have long said they reward sites that have good user experience with better rankings and we already see them index and rank test variations and even personalised pages.

Good optimisers are already mindful of how their split test variations might influence search visibility (both positively or negatively) and site changes should certainly only be rolled out after considering both. Personalisation however poses a bigger challenge for optimisers over the next few years since it’s an always-on experience that differs for different audience segments and individuals. Thus, the question for search engines becomes –

Which version of should page should be indexed and where should it rank for different people searching?

Consider that the ‘Sports’ page of a news website is personalised by featuring sports content the site knows to be relevant to a particular user (based on previous browsing history). When this user searches for ‘sports news’ in Google, the sports page on this site is possibly the most relevant on the web. For another user however, who has never visited that news site before, the landing page is not personalised and therefore does not contain content that is very relevant.

Should Google rank the sports page mentioned above differently for different people with the same search term? Should Google index multiple versions of the page? How would Google have visibility of the personalised content delivered to specific user groups or individuals anyway?

These are questions that will need to be answered because personalisation is not going away. Econsultancy found it was the third highest priority for marketers last year.

Search engines will therefore have to get better at indexing personalised pages/content and gathering the data required to understand how this impacts which pages to show different users, and when. In the meantime, optimisers will have to continue to closely monitor the impact their test variations and personalisation are having on search results.

Duncan Heath of Fresh Egg

Understanding That Testing is a Marathon and There are Stages to CRO Maturity:

While some companies that started with A/B testing early on are in the more advanced levels of CRO maturity, most of them are new to this game.

Shanelle Mullin (Conversion Rate Optimization at Shopify) points how CRO is still a young industry and stresses that this year will be more foundational than evolutionary for most companies who’ve started with CRO:

CRO is still a young industry. According to The 2018 State of Conversion Optimization Report, 62.4% of marketers have been working in the CRO industry for less than 4 years.

Despite the fact that there’s been plenty said about A/B testing, a lot of marketers are still getting it wrong. The result? People are making major business decisions based on inaccurate results from improper tests.

I think this year will be more foundational than evolutionary for most companies as they continue to define (and refine) their processes.

Shanelle Mullin – Conversion Rate Optimization at Shopify

Mullin also shares a general 7-step CRO process that most companies adapt as per their needs:

1. Conduct qualitative and quantitative research on your unique site to identify problem areas.
2. Use your research to come up with test and experiment ideas.
3. Prioritize those test and experiment ideas using a prioritization method like ICE or PXL.
4. Begin running the highest priority test or experiment.
5. Analyze the results of the test or experiment.
6. Record or archive the results of the test or experiment.
7. Use the insights from your most recent test or experiment to come up with smarter test and experiment ideas.

Shanelle Mullin – Conversion Rate Optimization at Shopify

From those who are now experienced experimenters, Parizek expects a better focus on “… the quality of testing hypothesizes (and less of stupid button color tests, please)… ” His CRO Maturity Model does a great job at explaining the different maturity models a company’s CRO program can be at.

As CRO programs of most of the companies are at the initial levels, they mostly only run trial-and-error campaigns. Without streamlined CRO processes, these companies often end up running experiments on an ad hoc basis with no long-term optimization strategy in place. Besides, there’s little learning from these experiments.

The Growing Importance of the Culture of Experimentation

A few years ago, stories like “The X best things to A/B test on your website” got published routinely. Everywhere, everyone was talking about changing button colors, adding the word “FREE” to the CTA, and using a different header image.

Marketing blogs were also flooded with articles on getting triple digit “lifts.”

But such content is now being replaced by the type focusing on developing a culture of experimentation rather than looking for quick wins.

This change reflects a big shift in the mindset of the marketers as they understand that these tactics don’t work toward long-term CRO success. And they do nothing to encourage data-driven thinking.

As a result of this realization, more and more marketers are committing to investing in a culture of experimentation.

They’re also recognizing CRO as an inherent part of the marketing/revenue mix.

Shlomo Trachtenberg, the Director of Conversion Optimization at WebMechanix, expects even more businesses to adopt a conversion-focused culture as he sees it as a sure way of achieving profitable cost-per-acquisitions, given the riding ad costs and competition:

Ad costs are rising, and competition online is becoming fierce as more businesses expand their budgets for digital marketing. The cost-per-click and conversion rate directly impact online marketing more than ever. So while cost-per-click’s are rising due to competition, conversion optimization becomes a huge lever and opportunity to achieve profitable cost-per-acquisitions and cost-per-leads. I believe we’ll see more and more businesses adopting a conversion-focused culture.

Shlomo Trachtenberg – Director of Conversion Optimization at WebMechanix

Tzvi Harel, Global Digital Marketing Manager, EFI Optitex points out that it’s only by getting everyone onboard and by building a culture of experimentation that a company can truly succeed with CRO. He explains how CRO is “cross organization,” and no more a single person’s job:

For many businesses out there CRO is still the realm of a single-person’s set of skills, instead of being a cross organization, backed-up effort. Yes – some businesses already adopted the “always be testing” culture, but these remain the minority.

The competition is fierce, users have high expectations, and human attention span is at its lowest ever. Top it with challenging attribution modelling (to say the least), and you can see why this year will be no different.

In other words, companies planning ahead cannot afford to drift and stay behind. Now, more than ever, everyone needs to play their part in the optimization game – product, marketing, creative, R&D, retention, support (you name it!). This must be a team effort with an ALL IN attitude – working hard on generating ever-increasing results, and working together to improve ROI.

The emphasis on meaningful & insightful data analysis will continue to be at the forefront. Let’s not forget automation, chatbots (AI), and voice search. Companies will start adopting new creative ways to use them, and these will continue to make a lot of noise in the industry.

Don’t get me wrong, automation, personalization and AI are very important and are here to stay and grow. But, for the long tail, the focus should continue to be on mastering the fundamentals of the trade and understanding how to connect the right dots.

At the very core of it, this means doing 4 things: 1. finding out what’s not working 2. prioritizing expected impact 3. hypothesizing the optimal fix, and 4. Iterative testing.

For some of you this might be trivial but I’ll say it again. Forget “best practices,” the new ’50 things you should be testing’ articles or the obsessiveness with testing colors. The WHY part is where we all keep failing and where we actually need to focus.

Every business should know its audience, make and test bold changes and fight to maintain accuracy. With this mindset, most teams will be able to spot valuable and repeatable opportunities and leave their mark on the bottom line.

Tzvi Harel, Global Digital Marketing Manager – EFI Optitex

Echoing the same idea, Parizek also sees companies investing in in-house experimentation as opposed to outsourcing as it helps drive innovation: “I also believe that there will be less outsourcing of A/B testing to agencies and external contractors. Innovation is a core process of every successful company and therefore in-house experimentation teams provide huge value.“

(And) Then There are The Changing Pricing Models of Testing Tech Providers

The biggest talking point in the testing tech space has been the changing pricing models of the top solution providers, with most of them looking at just two audiences: Enterprise and non-enterprise.

Many providers have squarely shifted focus to Enterprise customers.

Customers who are:

1. Inclined to spend upwards of several thousand dollars a year
2. Capable of affording yearly contracts
3. Already invested in CRO and in it for the long haul 

The tech and service needs of their highly intensive CRO programs easily justify the five/six figure annual enterprise contracts.

But because grandfathering isn’t so common in the CRO space, this is often a deterrent to companies getting phased out. Mostly, companies at the first, second, and even the third levels of their CRO programs get hit by such price hikes.

Unfortunately the trend of “going exclusively enterprise” is something that is here to stay. This is why businesses need to choose tech partners who respect grandfathering and provide them with the support they need to continue their programs, without the pressure of upgrading.

All Said & Done, Changes Are Challenging…

Every change in this ever-so-changing A/B testing landscape will have you work harder and faster to keep up.

Not doing so, isn’t an option.

At Convert, we always strive to empower optimizers like you stay ahead of the curve.

Whether it’s with our testing solution that comes with advanced personalization capabilities and affordable plans (for all kinds of optimization programs) to our transparent pricing policies, we love seeing businesses win big with experiments.