A/B Testing on Shopify: Top Challenges & How to Overcome Them
A/B testing is the bane of existence for Shopify stores that want to scale. But despite a large amount of literature available, many e-commerce businesses still get it wrong.
Most advice is oversimplified, making A/B testing out to be a magic pill that doesn’t require much foresight, is super easy to implement or a one-and-done affair.
But if your methodology doesn’t include a testing strategy that requires you to iterate in almost real-time, this results in poorly designed tests that don’t account for the business problem or solve the user’s burning issues.
Base your business moves on the inaccurate results, and you’ll likely end up making detrimental changes. The best way to avoid this and run better A/B tests is to understand the top challenges and ways you can quash them.
What is A/B Testing?
A/B testing is a randomized experiment that pits two versions of marketing assets against each other simultaneously to determine the best performer.
“A” refers to the original version and is also known as the control while “B” is the variation that is meant to challenge the original and thus is also called the challenger.
You can A/B test your CTAs, page headings, pricing, images, messaging, web page layouts, and more on a Shopify store.
To assess the best-performing version, traffic is split at random so one segment lands on Version A while the other sees Version B. The winner is decided through statistical analysis.
Why Should You A/B Test on a Shopify Store?
More traffic = more revenue.
For most Shopify & Shopify Plus seeking rapid growth, this simple equation holds true. But this linear growth hits the point of diminishing returns for the following reasons:
- Changing Paid Ads Landscape: E-commerce store owners have long depended on paid ads to fuel their growth. But privacy updates like Apple’s ATT and Google’s cookie-phase out have changed the landscape drastically. The inability to target audiences effectively has caused the ROAS to plummet across the industry.
- Inaccurate Email Marketing Data: Apple’s privacy updates can also potentially reduce the ROI of your email marketing campaigns. You can no longer rely on open rates or location-based targeting, or segmentation based on previous activity.
- Store-wide Poor UX: The “don’t fix what’s not broken” logic is dangerous when applied to Shopify stores. You might assume that since orders are coming through, nothing is broken. But you may not realize you have a leaky bucket costing you sales.
A/B testing is the answer to overcoming all these challenges.
Even a simple A/B test where you remove unnecessary friction from the buyer’s journey or introduce elements that nudge shoppers along can result in increased conversion rates. And that’s just scratching the surface.
Note: CRO or conversion rate optimization is not the best use of experimentation. If you were to remove all the fields from your form, conversion rates would shoot up, but you’d end up with low-quality leads. Aim for retention through your experiments to offer a better customer experience and drive revenue.
Here are three more reasons A/B testing is the right way to increase revenue for your Shopify store:
1. Supports Classic Growth Levers
You can propel your SEO and paid media efforts by A/B testing your messaging, heading, or improving page UX and make the most of your existing traffic. A/B testing complements your ongoing growth levers helping you achieve more with less.
2. Lowers the Cost of Implementation
Since you can compare variations without needing to invest in extensive dev resources or build-outs if you’re performing client-side A/B testing, the cost of innovation is low. This means you can explore new retention strategies on your Shopify store without any qualms.
3. Validates Ideas & Offers a Science-Backed Sneak Peek
Instead of emulating your competitors on a whim, A/B testing helps you validate ideas and estimate expected uplifts. Even your “failed” experiments can teach you what you need to know to create winning experiments.
A/B testing on a Shopify store can help CRO agencies and e-commerce entrepreneurs stop mindlessly chase more traffic and focus on building a personalized hyper-growth playbook instead.
Common Shopify A/B Testing Examples
You now know that you should run A/B tests but what should you be testing?
First, you need to develop a hypothesis. The premise should be built on existing data like heatmaps surveys, user research, site analytics, or interviews.
The hypothesis should be attached to an e-commerce KPI so that you know the changes you intend aren’t frivolous and will impact the business metrics.
Since you can tinker with each element on your Shopify store, you can potentially create thousands of variations. So while Google Optimize may suffice for a simple test, you should use a conversion rate optimization software like Convert Experiences to create a true culture of experimentation.
Here are five common Shopify A/B testing examples to inspire your next experiment.
#1. Reduce Cart Abandonment Rates
According to the Baymard Institute, the average abandonment rate is a whopping 69.82%.
Shoppers abandon carts for several reasons—unnecessary friction in the checkout process, limited different payment modes, inconvenient return, and refund policies.
Integrate Convert Experiences into your Shopify website, set up goal tracking via Google Analytics, and use heatmaps and visitor replay to identify potential issues to start running your A/B tests.
#2. Optimize Landing Pages
HubSpot’s 2021 Not Another State of Marketing Report revealed that only 17% of marketers A/B test their landing pages in a bid to improve conversions. That’s a big missed opportunity.
Discover low-hanging fruits, build your hypothesis, and use Convert Experiences to launch it.
#3. Test Pricing Strategies
From removing the $ sign to choosing prices with fewer syllables, you can run many pricing experiments. Use behavioral analytics and revenue tracking to choose the pages for the A/B test.
#4. Boost Form Submissions
Should you have fewer form fields?
Conventional wisdom says yes.
But this case study from Unbounce’s former conversion optimizer Michael Aagaard reports otherwise. Aagaard removed fields from a client’s form, resulting in a shocking 14% drop in conversion.
That is why you need to test everything (even universally accepted insights) because what works for them may not work for you.
You can use Convert Experiences to design forms with several variable changes and find the best combo for conversions.
P.S. Convert Experiences offers over 100 integrations so you can seamlessly integrate with your current Martech stack. If you’re a Zuko user, you can get even more insights into why users are abandoning your web form or checkout.
#5. Test Different Shopify Themes
Your theme isn’t just a visual representation of your brand. It can also significantly impact page loading speed, conversion rate, and trustworthiness.
Install the Convert Tracking Code in both the control and the variant theme and create a split URL test in Convert Experiences.
Shopify vs. Shopify Plus: Why is Shopify Plus Better for A/B Testing?
The main difference between Shopify & Shopify Plus is that Shopify Plus allows for theme optimization, cart customization, and gives you greater control over the checkout experience by allowing you to run third-party scripts.
You also get access to detailed analytics, which is crucial when running A/B tests. So, switching to Shopify Plus is the smart choice.
Top Challenges When A/B Testing on Shopify (And How to Get Over Them)
We turned to experts to share the top challenges they’ve faced, and how they overcame those so you don’t run into these roadblocks.
Mobile QA of Tests
Clickjacking allows hackers to insert a hidden UI layer on the device screen, tricking users into clicking on actionable content on a decoy site.
Shopify employs anti-clickjacking technology to prevent clickjacking, but this prevents some A/B testing tools from performing optimally when you try to switch from desktop to mobile viewport.
The test still runs on mobile but prevents you from editing the test on mobile unless you paste your code.
“Thus, you need to use a google chrome extension called “Ignore X-Frame headers” and add the site you’re optimizing to ensure that you can build tests across devices.”
– Amrdeep Athwal, Performance Analyst at HM Revenue & Customs.
Pushing the Test Winners Live
Matt Beischel, Experimentation Associate Consultant at Speero, has performed A/B testing on several Shopify sites. One of the main issues with the A/B testing process isn’t the actual test but rather an effective implementation.
“If you’re executing tests on the front-end, then the platform doesn’t really matter since tests get executed client-side on the finished site render. The problem comes in when trying to implement changes.
A test result is meaningless if it can’t be soundly and efficiently implemented.
Often, implementing a test result means customization; it’s not something an off-the-shelf app or plugin will be able to accomplish. Also, an excess of plugins can lead to code bloat, conflicts, script errors, etc., which contribute to site slowdown, which lowers SEO score and user experience sacrificing any potential gain the test found in the first place.
Finding, investing, and working with a competent developer to implement test results is critically important; don’t be penny-wise/pound-foolish.”
But you may not always have the luxury of waiting for your dev team to get involved. While you should eventually hardcode your A/B test results, being able to fashion an elegant build-out fast is key to a streamlined Shopify Plus store.
That’s where Convert Deploy comes in. Deploy allows you to push A/B test winners live and manage content and design updates without you having to pause for a dev to step in.
(Often) Sticking to Lower Complexity Tests
Standard Shopify stores don’t have access to most of the Shopify Plus stores’ features. For example, you don’t have access to the checkout template files, so you can’t split test Shopify themes.
This limits what you can test, and you get stuck with lower complexity tests that are most likely to deliver low impact rates.
That’s why Andra Baragan, Founder of OnTrack Digital, advocates for Shopify Plus. But Baragan also points out Shopify Plus stores have a larger website which means a testing roadmap is crucial.
The challenge for Shopify Plus stores is following the roadmap and waiting for the results when we live in this instant gratification world.
“There is usually a lot of pressure to implement things directly, rather than test them, as it’s such a fast-paced environment and business owners want results now. That’s where our expertise as CRO specialists or experimenters needs to come into place and educate stakeholders on the importance of A/B testing and running a scientific optimization program.”
And if you’re looking for an A/B testing tool that allows you to quickly integrate with Shopify Plus, try Convert Experiences. Packed with robust code and visual editors, it makes testing on Shopify stores a walk in the park.
Even though Shopify places limitations on deploying complex tests, Convert’s granular targeting options and advanced goals make it easy to design and deliver tests to the right audience.
There are over 40 conditions you can apply to your tests, to segment visitors based on traffic sources, average time on page, visit duration, visit time, browser and devices types, as well as page types, product SKU, product price, or product name.
Get a trial of Convert Experiences to figure out what works on your Shopify store and what doesn’t.
You can use it to test different versions of your landing pages, category pages, and product pages. You can also use it to test new referral initiatives, pricing changes, or subscription systems, and identify which tactics bring in the most orders and revenue.
3 Prioritization Frameworks to Help you Rank Shopify A/B Test Ideas
The most common challenge you’d have when A/B testing on Shopify is identifying which ideas are worth testing first.
Should you run with the low-effort idea or try solving a more significant UX issue? Here are three prioritization frameworks you can use to answer that question:
- ICE (Impact Confidence Ease) – Each factor gets a 1-10 ranking. If you perform the test without dev resources, you may rank Ease an 8.
- PIE (Potential Importance Ease) – Like ICE, each factor receives a 1-10 ranking. If the A/B test affects 80% of your traffic, you may rank Importance an 8.
- PXL – CXL’s prioritization framework is different and customizable, asking you objective questions and an ease of implementation question. If you answer yes, it gets a 1. If not, it gets a 0.
Between identifying your unique challenges and carrying out A/B tests via a CRO testing software like Convert Experiences, you’re all set to hit your revenue goals.
Written By
Sneh Ratna Choudhary
Edited By
Carmen Apostu
