From Series A to Series C, With Serious A/B Testing.

A/B testing is a decision tool, or methodology, based on the statistical foundation of hypothesis testing. It gives us the confidence to implement site (and product) changes with the understanding that: 

• The adjustments that are made in Conversion Optimization, that A/B testing is part of, have the potential to improve site experiences and thus boost conversion rates and revenue for brand specific use cases. 
• With the changes incorporated, the resulting experience is indeed different from the existing experience and thus an effect will be felt in the observed metrics. 

There are a number of reasons why controlled online experiments (or A/B tests) have emerged as a popular way to prevent gut instinct fueled decisions from wreaking havoc on business KPIs. In fact, their role is just to deny or confirm them! They act as a safety mechanism, and ultimately as a decision tool.

If you’ve ever felt that the “boss” or the HiPPO a.k.a. The “Highest Paid Person’s Opinion” is used as a tie breaker to tip the decision to implement in favor of ideas that aren’t guided by data, then properly executed A/B tests are an unbiased and powerful way to show that it is almost impossible to predict how buyers will behave, on the fly. And the only source of an educated guess is the buyers themselves. 

In short A/B testing is: 

1. The gatekeeper. A/B testing takes time and on the face of it may delay releases and launches, but it drastically cuts down the time and the cost to reach an experience that is optimal for both the user(s) and the business. In other words, it provides a sure-fire way to climb the mountain of success online, by advancing one experiment at a time, and making sure that the new changes do not degrade any important metrics, and instead improve them.

2. The compass that takes you where you want to go. This is the “optimization” aspect of experimentation. It allows marketing and product teams (and ultimately the entire company) to iteratively improve key experiences and touchpoints in a way that leads to compounding returns. And without untested disruptions that may inadvertently move the needle backwards. 

3. The guide to the mountain you wish to climb. This is the exploratory side of A/B testing. The one interested in innovation. Keep in mind that A/B testing isn’t just good for protecting what already works, and optimizing it. With the right kind of attitude, top down buy in for consistent experimentation, and a solid learning repository, it is also a spotlight. One that unerringly falls on potential hidden in your business and your funnels… giving you ideas to try novel approaches to existing business problems. 

Research into 35,000 startups has conclusively proven that a culture of experimentation has tangible benefits – namely more product launches, more VC funding, and rapid growth. See for yourself!

A/B testing during experimentation is largely a scientific process. 

Here are the key steps that you can’t miss, if you are A/B testing: 

1. Defining the business problem you’ll solve. This acts as the North Star metric for your optimization program, and not just a single test. You can definitely run one-time experiments. But it’s like task checking. Finishing one can move you forward, but it does not finish the project. The true value of A/B testing is realized by setting up a flywheel. 

2. Assembling a team of experts who know what they are doing. Don’t start with the tool. This is a classic mistake. The tool isn’t a silver bullet. You must first have the “consumers” or “users” of that tool in place. And then they help define the specs of the A/B testing solution you should invest in, based on the specs (goals, test velocity, roadmap) of the experimentation program. 

3. Defining the tech. This isn’t restricted to just the testing platform. It includes all the analytics tools needed for a functioning data layer that feeds a user friendly knowledge layer. Just because you’ve been collecting interactions across every possible touchpoint for years, doesn’t mean you have the quality of data needed to inform hypotheses. 
   
   This leads us back to the business problems you wish to solve. Once you know what you want out of your optimization and A/B testing program, it is relatively easier to specify entities, interactions, events of import, and the best way to collect and process this data for a true picture of what isn’t working as expected. 
   
   CAVEAT: Don’t A/B test a solution. A/B test to better understand the problem on your way to seeking the right solution, especially if you have traffic to spare. 

4. Thinking critically with quantitative and qualitative data! This includes treating data with respect and not shoe-horning it to fit the narrative in your head. 
   
   There are lots of biases to contend with. A code of ethics for data handling is non-negotiable. We’ve written at length about it. From here, you can start to frame hypotheses. Lots of frameworks to help you do that. 
   
   This is Convert’s hypothesis generator (which has the option built in to consider the impact of your hypothesis on the people who are participants in your “tests”). 

5. Coding up and deploying your experiments. This is the stage where the contributions of your tool matter the most. Please bear in mind that the complexity (or the simplicity) of the process you go through to get to the phase of rolling the experiment out does not have a significant bearing on how the deployment will go. 
   
   A simple hypothesis may involve specific triggers and convoluted actions (goals) that are hard to define and code on a client side A/B testing platform. 
   
   And a hypothesis that took weeks to isolate may need a simple change made to site copy through the visual editor. This is the reason why it is best to invest in an A/B testing solution that gets the basic pillars of experimentation right. The bells and whistles are great to have but they do not add value to actual test deployment. 
   
   Don’t forget the aspect of Quality Assurance. A broken test will yield broken results. The delicate process of A/B testing spends site traffic as currency. Most websites don’t have endless traffic. Spend it on experiments that work as expected. 

6. Learning and Re-iteration. If you aren’t learning from your A/B tests in a way that the insights make sense and appeal to 3 groups of stakeholders, you are leaving money on the table: 
   • Experimenters. These are the people who are testing, every day. A growing repository of artifacts summarizing the learning from previous efforts is crucial. Not to replace A/B tests with the repository. But to improve their understanding of your audience to create increasingly mature hypotheses. 
   • C-Suite. These folks may be excited about winning tests. But they are worried about implementing winning solutions. Share how A/B testing solves key company problems. Either through direct impact on conversions (with a revenue potential attached) or through insights that help the whole business take better decisions. 
   • Cheering Squad. People who are not in the day-to-day grind of A/B testing, but who have the ability (and clout) to translate the statistics and the reports into progress – for the entire company to understand, cheer for, and embrace. Marketing has had to win a seat at the proverbial table to be taken seriously, beyond PR and keywords… It’s time for experimentation to emerge as a key player in revenue and innovation.

Here’s a compilation of real world A/B testing examples that have impacted important numbers for businesses. Get inspired.

Testers today are spoilt for choice when it comes to choosing A/B testing tools and platforms. 

We’ve created an in-depth guide on how to vet and choose an A/B testing solution that fits the needs of your company. 

But here we would like to focus on the most important considerations, ones that are generally not covered in the resources found online. 

1. The rhythm of your experimentation program. How much testing do you want done? How many goals do you wish to attach to each experiment? Because here is the thing – you can get by with one goal (it may be enough to indicate an uplift or a decrease), but do you wish to stop there? Or do you want multiple secondary goals to validate and corroborate the impact detected by the primary experiment goal? In fact, do you want to track interactions for learning? The framework of running A/B tests should ultimately dictate the kind of features you need and the limitations that you can live with. 
   
   No A/B testing tool is perfect. No single A/B testing tool out there has the perfect combination of features and benefits … Every investment is in some way a compromise. What are you not willing to compromise on, given how you want to run your program? 
   
   Think about it! 

2. Your team structure. This one is important. Are experimenters going to code and deploy tests? Or do you have developers, who know what it is like to code for an experiment, in your team? The latter can make life really easy. Under such circumstances, you definitely want an A/B testing solution that lets you get your hands dirty with the code. And tap the potential of the tool to the fullest. 
   
   Do you want to work with an agency? And if that’s the case.. Do you want to start on an agency purchased plan or buy a tool of your choice and let the agency manage your account? This is a key consideration because yes, some tools allow users to shift accounts from agency plans to independent operation. 

3. Your website. This one is obvious. For example, SPA A/B testing (like testing with React, Vue, or Svelte) poses specific challenges. Shopify A/B Testing comes with its own set of technical limitations, that some tools like Convert.com solve. Again, your A/B testing solution doesn’t need to be the best at everything. It just needs to be the best at handling the nuances of your unique tech stack. 

A side by side A/B testing tools comparison is useful.. But only when you are on the verge of taking a call on a particular platform. 

A lot of big picture thinking happens in the background before that.

There are 4 features or capabilities an A/B testing platform needs to equip its users for success. 

Before we dive into what they are, a reminder. 

An A/B testing tool can never replace an experienced team with an understanding of processes and a strong sense of accountability. 

However this talent should not be hamstrung by a limited tool either. 

So here it goes: 

1. A/B Test Targeting and Triggering: A vital aspect of robust A/B test design. An experimentation platform should allow its users to target the right people (because you are estimating behavior of a much larger group based on a small sample. The closer the sample is to the real persona you wish to serve with the solution, the better the results) and serve the variant in the right site section, in the correct sequence, and at the right time. Triggered earlier than needed, or later than needed … the behavior elicited from the test sample won’t be an accurate reflection of the behavior that would be elicited in response to a properly timed stimulus. 
   
   Plus, you don’t want to inadvertently expose your sample to information that can bias behavior and outcomes. 
   
   The importance of targeting and triggering can’t be stressed enough. This is why Convert Experiences comes with 40+ filters in its targeting engine. Serve almost anyone you intend to. Bring your hypotheses to life. 

2. Experiment Data Tracking and Goals. There are goals. And then there are goals that give your optimization program the maturity to go beyond templates. Templates are convenient. But A/B testing is about respecting the unique nature of your business and use cases (That’s where the sentiment of A/B Test everything comes from). 
   
   How can you hope to serve possibly unprecedented actions if all you have are editable templates? An A/B testing platform should do a double whammy – templates and advanced goals. With the ability to add multiple goals to every test. 

3. Debugging and Quality Assurance of A/B Tests. We’ve touched on this already. A/B testing spends your site traffic as currency. You can’t afford to incur a debt by displaying broken tests to visitors. Most importantly – the idea is to have an experience that is not perfect, but as close to the actual experience (or feature) you’ll roll out, as possible. This is what ensures the best data for insights and the most reliable result. 
   
   QA is overlooked. But it literally makes or breaks experiments. This is the reason why Convert Experiences comes with both a Chrome debugger and the convenient, real time Live Logs. 

4. Integrations. This is for users who know that all-in-one platforms rarely ever live up to the hype. Either the extra features don’t work to support the scientific process of A/B testing (where “good enough” isn’t really an option). 
   
   Or the clutter from the bells and whistles erode adoption and usability. 
   
   It’s much better to invest in tools that have a singular focus. So that each aspect of your data collection and analysis stack is excellent, ever evolving, and maintained by experts. A backbone to trust for your A/B tests! 
   
   Integrations bail you out of the all-in-one quagmire. They make data available in a centralized location. They gift you the freedom of assembling a stack that serves your exact needs, without you having to pay for features you’ll never use. 
   
   Shifting away from tools that do “everything” is a mark of a mature A/B testing program. One that understands “everything” isn’t the key to success, the right things are!

Most of the information online has traditionally been provided by vendors. So it is no small wonder that there is a narrative of A/B testing solutions or platforms being the savior – the silver bullet. 

But we beg to differ. 

1. Decide on the specs of your optimization program (what kind of business challenges you’ll solve and how) 
2. Get your team onboarded. Take stock of their skills and preferences. 
3. Then invest in a tool that lives in the intersection of non-negotiable features and user must-haves. 

Have a budget that will allow you to access an app that meets most of your genuine requirements. But you don’t win any medals for the most exorbitantly priced A/B testing tool. If there is a surplus .. invest that budget in up-skilling your team.

Here are 4 great reasons to onboard an A/B testing agency or a CRO agency: 

1. Assistance with A/B testing build-outs. Most experimentation programs do not have dedicated developers who focus on test coding and deployment. This is where an agency that specializes in doing the heavy lifting, coding wise, keeps up test velocity. Their experts not only have the bandwidth to start emptying out your backlog, they also understand how to walk the fine line between a perfect experience and an experience or a feature needed for testing and thus one that is a close enough reflection of the real world ideal. 

2. Assistance with Telemetry. Setting up proper data collection pipelines is a challenge. While data storage and processing is a vital skill set that must be cultivated in-house, if the expertise is not available, growing it can take a lot of time and add to the chaos of broken tables, ill-defined entities, random events, and more. 

3. Assistance with Customer Research. There is no such thing as too much qualitative data. Especially if it is collected in response to unbiased questions asked in the right way, at the right time. Most brands may not have the bandwidth to run experiment specific Cx research sprints. An agency fills this much needed gap. They also bring their learning documentation chops to the table, giving an experimentation program the invaluable gift of processes and frameworks that work! 

4. Evangelizing Experimentation. Remember the “Cheering Squad”? An agency can kickstart the evangelizing process by helping in-house stakeholders: 
   • See results (Results =/= Wins) 
   • Understand the part A/B testing plays in solving problems they care about 
   • Present learning 
   • Even “Prove ROI”

Convert Experiences possess a suite of enterprise grade A/B testing features at self service prices. 

The best way to understand how Convert Experiences can have a positive impact on your experimentation program (especially if you’ve been grappling with the BIG 4 issues we’ve laid out) is through a custom demo. 

But that takes 30 minutes. And you may not have them to spare. 

Here’s a 5 minute recorded demo of the app that gives you a teaser of what you can expect: 

Watch Demo Right Away [5-Min]

Great question. 

There are very few A/B testing solutions meant for teams just starting to outgrow a free tool like Google Optimize. This is a crucial stage in their experimentation maturity where they need to test more and test increasingly ambitious ideas to remove experimentation from the confines of “CRO”. Testing may not have the leverage needed (yet) to demand 150k USD a year up-front – just for a tool. 

This however doesn’t mean that the teams should keep wrestling with restrictions and limits. 

Convert Experiences is created to help these eager teams realize the true potential of A/B testing, and then shift over to a platform that will do justice to the evolving skills of its experimenters and the changing needs of the business and the program.

In many cases this next step is a function specific A/B testing tool – like ones that are especially created to serve product teams. Or it may be a custom solution coded up in-house to handle use cases which out of the box apps simply can’t.  

This is the reason why we are priced transparently, and the way we are. 

We want more businesses & brands to A/B test – the right way and with the right team cause this is a success lever most companies aren’t pulling hard enough. 

This is also why we only work with Conscious Businesses, growing which will do the world a good turn.

At Convert, we use simple and proven statistical methods in our A/B testing reports, the classical Frequentist Z-test, with two tails. 

Thanks to our Convert Calculator, you can also plan your test the right way, but choosing a minimum effect of interest, also called minimum detectable effect (MDE), which will give you the amount of visitors you will need per variant before you can end your test.