The Challenge: Free Shipping Trimming the AOV
Through quantitative analysis of their ecommerce client’s transaction data, Sculpt Digital found that uptake of express delivery was too low.
More buyers than should be expected made their purchases using free delivery.
Also, analyzing the user recording showed that users didn’t notice that there was an option to receive their orders faster and were often defaulting to free shipping.
This meant that two critical metrics, average order value (AOV) and the revenue per user (RPU), were in the red.
Sculpt Digital then figured that express delivery should be a more obvious choice for users.
Hence the hypothesis:
If the express delivery option is promoted on the checkout page, shoppers will opt for it, which will increase the average order value since many prefer a faster delivery time.
But they didn’t stop there. They also explored another option that will increase AOV while making the checkout process more frictionless: Adding Stripe as the preferred payment method because of its lower processing fees compared to PayPal.
PayPal was the old default, and it cost the client more.
Making Stripe the preferred payment option would bump up the AOV by reducing the cost of each purchase transaction.
The Test: Promote Express Delivery to Shoppers on the Checkout Page
A version of the checkout page implementing the hypothesis was tested against the control. The goal was to see how the changes impacted revenue per user, AOV, and the change in payment method and delivery option chosen.
All visitors to the website experienced the test, which ran for 35 days.
Control
Variation
These are the test stats:
| Testing Parameter | Value |
| Number of Variants | 1% |
| Statistical Significance Threshold | 95% |
| Power | 80% |
| Traffic Split | 50/50 |
| Minimum Detectable Effect (MDE) | 1% |
Tech Stack
- WooCommerce: The ecommerce platform hosting the site.
- Convert Experiences: For running and managing the test and reporting.
BrowserStack: For quality assurance.
Convert’s native anti-flicker technology was of critical importance for this test [because], understandably, the client had concerns when testing checkout that any experiments wouldn’t hinder the purchase journey at such an important stage.
Convert’s technical flexibility, QA’ing facility, and debugging, such as the live logs, came in handy when setting up the experiment. The native GA4 integration works really well for post-test analysis as well to look at the broader impact of the test on metrics not originally defined in the setup of the experiment.
— Jamie McGrath
Results
| Metric | Pre-Test | Post-Test | Improvement |
| Revenue Conversion Rate | 43.50% | 48.36% | +11.15% |
| Revenue Per User | 26,305 GBP | 30,517 GBP | +16.01% |
| Average Order Value | 60.46 GBP | 63.10 GBP | +4.36% |
| Express Delivery Conversion Rate | 2.15% | 4.87% | +127.10% |
Learnings from this Test
- Promoting express delivery in the checkout presents an opportunity for increasing AOV: It’s possible that some buyers will miss the express delivery option if it’s part of a drop-down menu where the default entry is “free shipping.” You can make it more prominent with a banner, so it is easy to opt for with one click.
- Provide more than one payment option: It makes checkout more seamless by accommodating various buyer preferences. It’s also an opportunity to improve your AOV (even though it’s marginal) with lower transaction processing fees. This benefit stacks up quickly for stores with a high volume of sales.
- Don’t be afraid to optimize checkout flow: Regularly test different elements of the checkout process, such as messaging, payment options, button placement, address auto-fill, etc. It can reveal improvements in user experience (UX) that translate to better sales.
Ensure technical reliability during testing: When testing critical parts of the shopper’s journey, such as the checkout flow, it’s vital that you ensure the test doesn’t negatively impact the UX. Debugging tools like BrowserStack and Convert’s live logging ensures everything works great. Also, using an A/B testing tool like Convert that doesn’t flicker will keep the experience clean for shoppers.
Sculpt Digital is a digital agency with expertise in CRO, social media, SEO, PPC, and more. They focus on compounding digital growth through mastery of data, insights, and ecommerce tech.
At Convert, we’re on a mission to create meaningful, sustainable experiences for today’s users and future generations. As the only Climate Neutral Certified A/B testing platform, we’re proud to offer a robust, affordable, and feature-rich platform that’s been an industry favorite for over a decade. But we’re not just about optimization for profit. With a Conscious Business approach, we’re dedicated to making a real difference and driving positive change within the CRO space and the world.