Don’t Put Your A/B Testing on Ice This Holiday Season

If you’re planning to halt your A/B testing program for the holidays, you may be missing the most wonderful time of the year to capture customer insights.

While it’s true – the holiday season is very different from rest-of-year, and shifts in traffic and customer behavior can pollute your existing A/B tests (which should wrap up ahead of Black Friday), there’s still great opportunities for testing specific hypotheses during the holidays.

But instead of FOMO (fear of missing out), it’s FOMU (fear of messing up) that makes merchants tremble in their snow boots.

There are 3 common reasons merchants avoid testing during the holidays:

1. To keep bugs out of their code
2. To improve web performance
3. To avoid the risk of a losing experiment

Let’s unpack these reasons and explore ways to move your testing program from “on ice” to “very nice.”

Why Merchants Avoid A/B Testing During the Holidays

Code Freeze

It’s prudent to protect production code from problems that can poison peak season experiences.

That’s an alliterative way to say – keep bugs out of the backend.

But your A/B testing application can handle a lot of testing requirements using client-side CSS and Javascript without adding code to your commerce platform.

Similarly, you can also set up test elements through your CMS, merchandising tools, no-code experience builder, or even with section blocks inside your Shopify theme editor safely.

Many of these systems enable you to modify site content, layouts, and even business logic through application settings and don’t add code to your core system in the same way that new feature development does.

Unless you’re on an ecommerce platform that still requires a full stack developer to handle every single front end change, there’s a lot you can do without fear of breaking stuff.

Web Performance

If you’ve ever run a Page Speed Insights or Lighthouse report on your webpage, you’ve likely seen scripts for your analytics, tag manager, and yes – A/B testing tools – flagged for their Javascript execution time.

We all know that slower page load speeds can impact bounce rates, conversion rates and revenue.

But it’s important to remember that properly implemented tracking scripts are deferred to execute after more important content.

It’s slow loading content that impacts user experience and drags conversion KPIs.

While scripts are calculated in your total page weight, it doesn’t mean they have to impact user experience or perceived page speed.

Instead of turning off testing, consider:

• removing unused or unnecessary third-party apps,
• look for opportunities to optimize the loading of primary content,
• properly compressing media,
• lazy loading below-the-fold content, and
• eager loading above-the-fold content.

Fear of Failure

Many merchants don’t want to risk the revenue loss of serving a losing variant during the most critical time.

But this means putting full faith into the gut feel decisions you’ve made. You’ll never know if serving 100% of visitors to your Control costs you conversions.

Even if you have carried over insights from previous holidays, every season has its own nuances.

Everything from the state of the economy, ad spend and competition in search and Meta, product trends, the number of days between Thanksgiving and Christmas – even the weather can shake things up.

The key is choosing the right experiments and testing strategy. The holidays are not the time to deliver new functionality or radical redesigns, or even to fix usability issues.

It’s the time to maximize holiday sales – so focus on the “low hanging holly berries” suggested below.

Pause Holiday A/B Testing If…

Before we dive into ideas to test, consider why A/B testing during BFCM and the holidays is not a fit for your team.

…You’re Short on Team Bandwidth

Even if you’re using a consultant or agency to run your CRO program, if your internal team won’t be able to keep up with regular meetings or interim communication, pausing testing makes sense.

…Your Biggest Promotion Ends on Cyber Monday

Running a test through two business cycles is standard practice, but BFCM is a 2-4 day window of hyper-shopping activity. Your holiday season may span four weeks, but it’s really a single, highly erratic business cycle with a massive influx in traffic.

You can still fit in holiday-specific tests within a 2-week span, but consider that the BFCM weekend itself may impact this testing cycle if behavior or marketing activities change dramatically after Cyber Monday.

The BFCM weekend will likely have large swings in traffic, conversion, average order value and revenue per visitor that will impact your “2-week business cycle” plans.

However, you can still run specific tests during BFCM weekend (4-day range) and determine at the end of the data collection period whether to accept any results as valid based on traffic levels and conversion events.
While you won’t be rolling out winners this year from this method, you may learn important insights to apply to future seasons and flash sale events.

…You Have Frequent Planned Changes

Also, consider if your merchandising strategies are planned to change frequently and significantly through the season, they can throw monkey wrenches in your test conditions.

For example, many merchants run daily flash deals on specific departments or featured products, with aggressive email and ad pushes around each micro-campaign.

Consider how website cutovers (changing discounts, creative or messaging) might impact test variables.

Low-Hanging Holly Berries

So if you’ve warmed to the idea of holiday A/B testing – what should you be testing?

Cater to Observed Behavior

Because holiday behavior is atypical and is highly influenced by real time competitive offers and trends, traditional user research is less helpful for forming hypotheses.

Instead, you can start by analyzing and comparing last year’s holiday analytics, heatmaps and session recordings to rest-of-year trends.

If there are glaring differences in things like top entry pages, home page scroll depth, navigational clicks, review engagement, search terms, cart abandonment or visits-to-purchase (for example), these can give you clues on where to focus.

Test Populated Content

Think about what creative and copy can be tested within existing sections like:

• promotional strips and text scrollers,
• hero banners and media grids,
• category page grid banners, and
• cart drawers.

Instead of testing new designs and UX features, you’re not fundamentally altering your site templates with this approach – and you can learn what verbiage and imagery persuade holiday shoppers most.

You can even experiment with how you order home page banners from top-to-bottom, which category links to prioritize, or which filters to promote higher in faceted navigation.

And focus on holiday-specific anxieties like discreet gift options, return policies and on-time delivery.

Here are some more smart holiday testing angles, courtesy of Alexandre Suon on LinkedIn:

Test Merchandising Strategies

Most search and merchandising applications enable you to A/B test merchandising logic like:

• changing a category’s default sort order (e.g. by Bestselling, Newest, Average Rating or Price),
• boosting and burying products by attribute, and
• pinning featured products to specific grid slots.

You can also experiment with changing product recommendation logic. Remember that holiday shoppers are often gifting for others and may not want your usual cross-sells. It’s far easier to cross-sell a one-size accessory than suggest a second pair of jeans or shoes. (At least, that’s the hypothesis!)

So, don’t let FOMU freeze your testing program this holiday season. With smart planning and low-risk opportunities, holiday testing will be your way to insights that keep giving all year long.