A/B Testing Pricing – Is It Legal And Ethical? Can It Help You Find The Right Pricing Strategy?

Remember this dress that nearly broke the internet?

Were you on camp “black and blue” or “white and gold”?

Yeah, what does it have to do with A/B testing pricing?

Here’s the thing: CRO experts are just as divided on the matter.

Some say heck yes while others vehemently oppose the idea. And while the dress mystery was ultimately resolved (it was black and blue), pricing tests have no concrete conclusion.

So in the spirit of true experimentation, consider the facts and both POVs before you make up your mind.

Pricing Strategy – What It Is & How to Choose One For Your Brand

A pricing strategy refers to the processes you use to set prices for your products or services.

But having an effective pricing strategy isn’t about crossing one of the 4 P’s in marketing—it’s an underutilized growth lever. Your pricing strategy conveys value, convinces customers to buy, and gives them confidence in your products.

Your pricing is the exchange rate on the value you’re creating in this world.

Patrick Campell, Founder & CEO of ProfitWell

Ideally, you should already have one. But if you’re unsure about which strategy to choose or haven’t thought about it, here’s a quick rundown of the most common pricing strategies:

• Value-based pricing: Charge customers based on what they think your product is worth.
• Competitive pricing: Set prices based on what the competition is charging. Good if you’re just starting out but remember there isn’t a ton of room for growth.
• Price skimming: Start off with high prices that will still have takers and lower your prices over time. You skim the top of the market and then lower your prices to reach everyone else. Exercise some caution if you choose to try this.
• Cost-plus pricing: Take the production cost and slap a percentage to it. This is ideal for physical products.
• Penetration pricing: Offer products at a lower price than your competition if your industry is very competitive. This can get you customers and higher sales volume but remember to also build trust and possibly a community that will rally around you when you inevitably raise your prices.
• Economy pricing: Price your products cheaper than the competition and make your money back through the increased sales volume. Ultimately, you will have to switch to a different strategy.
• Dynamic pricing: Charge based on the current demand for the item. Also known as surge-based pricing, you may charge different prices to different groups or based on the time.

How Do Consumers Feel About These Pricing Strategies?

Pricing can convince your reader to make a purchase or scare them off. When customers perceive a price in a negative light, they’re considering the monetary sacrifice they would have to make to acquire a product or service.

“The pain of paying” is quite literal. Spending money activates the areas of the brain associated with physical pain. (Zellermayer, Ofer. (1996). The Pain of Paying.)

And as you might’ve guessed, consumers aren’t huge fans of dynamic pricing. In fact, it seems discriminatory.

So are they looking for the cheapest possible products?

Again, no. The pricing conveys value and confidence. According to a 2019 study at the University of Texas at Arlington, consumers equate high prices with high value. But several factors that affect price perceptions and generalizations don’t have a place in data-driven decisions.

The Role of Experimentation in Determining the Right Pricing: Pricing Optimization

Like with everything else that warrants a second look, the best way to know for sure is to test it.
Instead of going off what the experts have to say or what your competitor is cooking up, let your audience tell you what they think your products are worth and adjust your prices based on cold, hard data.

How Does Price Testing Work?

When A/B testing, you pit a variant against a control or original and present it to your visitors to see if the change has had the desired result.

If it has, you want to be certain the impact can be directly attributed to the change. So you’d do a thorough review of the key experiment parameters like sample size, conversions, statistical significance, power, and more.

The process for A/B testing prices is the same but it’s not that simple.

For the test results to remain untainted, users should remain unaware of what is being tested. The knowledge interferes with their natural response rendering the test unusable.

But this means when you test two different price points for your products or services, some customers may pay more than others.

A/B testing tools split the traffic to the control and variant at random. But you still have to exercise caution to eliminate bias and choose the right targeting conditions (i.e. who sees the different prices) based on factors that don’t give rise to inadvertent discrimination. You run the risk of undercharging or overcharging based on ethnicity, social class, or gender.

We don’t have to spell it out for you. It’s not fair to your customers nor a good look for your brand. And a PR nightmare if word got out.

Is Price Testing Illegal?

Price testing can be unfair and possibly damage your brand’s reputation but is it illegal?

Some of these pricing strategies technically have the same definition as price discrimination:

There are 3 types of price discrimination—first-degree, second-degree, and third-degree price discrimination.

• First-degree price discrimination: Also known as perfect price discrimination, first-degree is rare since it means you charge a different price for each unit sold.
• Second-degree price discrimination: Charging different prices for different quantities like quantity discounts for bulk purchases.
• Third-degree price discrimination: Offering a different price to different consumer groups. Think offering discounts to students or frequent travelers or increasing the prices at peak hours. This is the most common type of discrimination.

For price discrimination to occur, these criteria must be met:

• You must have a monopoly to some degree.
• You must be able to recognize differences in demand.
• You must be able to identify different market segments
• You must have the ability to prevent arbitration or resale of the product.

What does the EU say about price discrimination?

The laws in the EU state that consumers cannot be charged more than other EU countries or prevented from buying something. The only exception is when you factor in the cost of the delivery but even those costs must be made known to the customer upfront.

Consumers are entitled to have access to the same prices as local buyers. These pricing rules apply to offline and online sales as well as travel tickets.

You may think it makes for a good customer experience to automatically redirect users to the country they live in but EU forbids automatic redirects. You can suggest a different site but consumers have to give explicit permission. This is done so that consumers can check what kind of offers and prices are offered in other countries.

EU’s GDPR laws can also prevent you from using data to offer personalized pricing. Users have to explicitly give their consent and companies have to let them know the specific reason for collecting the data.

What does this mean for you?

You cannot A/B test prices based on nationality. This can trigger an official complaint against your e-commerce business.

If you do plan to collect data to run experiments to A/B test personalized pricing, you have to be transparent with visitors. This can lead to tainted data which renders the test useless.

Maciej Wilczyński, Partner & Founder of Valueships, is a fan of A/B testing but doesn’t believe you should run A/B testing on pricing pages:

Another skeptic of A/B testing pricing is Maciej Kraus, Partner at Movens Growth Equity. Kraus posits that statistical significance is rarely possible for pricing and that A/B testing is relative, not absolute.

But Kraus agrees that it can work in some cases. An A/B test for an app subscription compared pricing and purchase screen layouts gathered some promising data.

The US and Price Discrimination

The FTC says,

Price discrimination can be illegal if the reason for the difference is based on a “suspect category”—race, religion, nationality, gender, or the like.

The Sherman Antitrust Act

The Sherman Antitrust Act states that “[e]very contract, combination in the form of trust or otherwise, or conspiracy, in restraint of trade or commerce among the several States, or with foreign nations, is declared to be illegal.” It also makes monopolization illegal.

Price fixing and monopolies affect consumers in a negative way. They would have no option but to pay whatever price is set by companies.

Since the law was the first of its kind and covered broad strokes, it was amended by the Clayton Act in 1914.

The Clayton Antitrust Act

There are several provisions of the Clayton Antitrust Act but we’ll stick to the one that covers price discrimination.

The Act bans price discrimination i.e. businesses cannot charge different customers different prices for the exact same product or service. This was later amended by the Robinson-Patman Act.

Other provisions of the act prevent the creation of monopolies and price fixing.

The Robinson-Patman Act

A federal law passed in 1936, The Robinson-Patman Act is supposed to prevent “unfair” competition. This prevents distributors from charging different prices to retailers.

So if you’re in the B2B space, you cannot sell products or services at a lower price to large-volume buyers putting small volume buyers at a discount.

However, this law has been criticized by several economists and legal scholars citing that price competition is a necessary tenet of business growth.

What does it mean for you?

The complexity of the act and tensions around it with regards to price competition and other aspects of antitrust law prevent it from being enforced and gathering support. Charges have to be brought forth by private plaintiffs on the grounds of

• Price discrimination on at least two consummated sales from the seller to two different purchases
• Sales must cross state lines
• Commodities must be of “like” grade and quality sold in the US.
• It should result in lesser competition or tend to create a monopoly

The Competition and Consumer Laws in the UK

The competition law prevents businesses from price fixing, carving up markets, price discrimination when there are no supply chain issues, and teaming up to create monopolies.

The Consumer Protection from Unfair Trading Regulations 2008 (CPRs) is designed to prevent unfair practices such as withholding information, invoking FOMO, and aggressive practices that don’t allow consumers to make an informed purchase decision.

What does it mean for you?

The Financial Conduct Authority (FCA) notes that “price discrimination is not in itself an unfair practice.” But, many questions are being raised about how and when is it fair.

Some examples of when it is deemed unfair include:

• Loyal customers getting a bad deal than new customers
• Customers paying more because of their email address, gender, or the ethnicity of their name
• Big data being used to charge personalized prices based on a customer’s profile.

What Do Consumers Believe?

The FCA pointed out in a 2018 Research Note that consumers regularly pay different prices for the same product.

However, not every instance of price discrimination leads to public outrage. For instance, companies will often provide a student discount or a lower price for non-profits and this is widely accepted. Or customers are happy about discounts they receive through loyalty programs.

But other forms of price discrimination illicit a stronger and negative reaction.

A 2005 report from the Annenberg Public Policy Center of the University of Pennsylvania termed “Open to Exploitation” looked into online price discrimination.

The study found that nearly two-thirds of adult Internet users incorrectly believed that price discrimination was illegal. And that’s not it. The survey participants also believed that if it wasn’t illegal, it should be. 87% of people strongly objected to personalized pricing.

Cut to now and consumers still object to price discrimination when they find out they’re not getting the best price but different generations handle this information in their own way.

Boomers are more likely to object to any kind of personalized shopping experience while Gen X sees this as an opportunity to game the system. Consumers search for flights from an internet cafe instead of their living rooms or add goods to their cart and intentionally abandon their cart to get a better deal. 

Millennials are split on the issue. Some have no issue handing over their data while others don’t like being pitted against each other.

Is Price Testing Unethical?

Okay, so price testing is largely legal. But just because it’s not illegal, does it mean it’s okay to run these pricing tests?

Craig Sullivan, Optimizer in Chief at Optimal Visit, helps businesses increase conversion and improve the UX of their multi-channel and multi-platform businesses. Sullivan recently spoke about the ethical conundrums caused due to the mass democratization and access to A/B testing tools on an episode of the UX podcast.

Sullivan talks about the perverse incentives in experimentation that are harmful to the user experience.

Sullivan argues that businesses don’t have to grow at the expense of humans.

LinkedIn explains how this can be put into practice through “fairness of design.”

Traditional A/B testing focused on the average user. LinkedIn recommends going beyond the average and tracking the inequality impact of these tests. 

The Case Against A/B Testing Your Prices: What’s the Real Objection?

If A/B testing prices is legal and ethical with some caveats, why aren’t there more people that favor that middle ground?

Let’s take the 3 major objections against A/B testing pricing and see if they hold up:

1. Statistical significance is rare for pricing: for B2B but not for DTC
   
   Sidebar: What is statistical significance?
   When you run A/B tests, you want to make sure that the results are legitimate and not a product of chance.
   
   This means you need a lot of visitors who are potential customers to consider A/B testing pricing and a massive number of visitors if you’re testing different personas.
   
   For B2B businesses, this may not be in the realm of possibility but is this an issue for B2C brands, especially those in the DTC space?
   
   You could argue that even with thousands of visitors on a DTC site, you may not be able to identify visitors and leads correctly. Or that you have to track these visitors over a month or more to get accurate data but price sensitivity shifts dramatically and can hamper those results.
   
   While this may be true for B2B businesses since sales cycles are long and the purchase intent isn’t always crystal clear, DTC businesses have extremely short buying cycles. Often a purchase cycle is completed in a single session which means your experiments can reach statistical significance easily and also allow tracking of direct revenue per variant (RPV) instead of dealing with proxies of revenue generated.
2. Price anchoring will upset your potential customers: Maybe B2B customers but not DTC
   
   Sidebar: What is price anchoring?
   Price anchoring is a practice of establishing a price point that customers use as a reference point when making decisions. For example, if a $250 product is marked down to $125, the original price is the price anchor for the discounted new price.
   
   Price anchoring is complex in the B2B world. There are multiple decision-makers and you can’t use dynamic pricing because businesses expect consistent monthly or yearly expenses.
   
   But B2B and DTC customers are inured to price sensitivities to a degree.
   
   
   • Visitors see different prices on affiliate sites and marketplaces like Amazon compared to the main store and prefer to shop around for the best deal.
   • Customers are aware that VIP and loyalty programs will grant them a better discount
   • Shoppers know and look forward to sales when prices drop dramatically and wait for these occasions. Think Cyber Monday and Black Friday.
   • Buyers are also aware that personalized pricing exists in e-commerce and that brands may offer different discounts to different audience segments during the same sale.
3. A/B testing is relative, not absolute: Short-term gains vs. long-term benefit
   
   Yes. Pricing should be a strategy. It’s not about testing a price point in isolation or trying to ascertain which pricing drives the completion of the most number of transactions.
   
   Especially when you look at B2B, you want to set prices that will support the product value so that you build long-term relationships with your customers and not trigger a one-time purchase. That helps your business thrive as a financially healthy organization where the unit economics makes sense and you also have the funds for R&D for continued growth.
   
   All A/B tests do is tell you which price point is more appealing to a cohort at that time aka finding that sweet spot. But this information (with a fair degree of certainty) can be a gamechanger for making short-term decisions like filling out a form.
   
   People generally don’t regret doing that. And you don’t have to mull over the potential impact of one less form field on how your prospects view the value your product generates.

So How Should You A/B Test Pricing, If At All?

Context is everything.

Why are you A/B testing pricing? What impact do you hope it will have?

And more importantly, how are you running these experiments?

Your pricing strategy should come from critical thinking through your goals, market conditions, your business model, and audience research. A/B testing isn’t a replacement for this exercise.

Instead, use A/B testing to understand how you can best present the value generated by the product. Especially in B2B, you want to tie price to the product value and the problems its solves.

So don’t A/B test prices. A/B test the pricing page.

Take Instapage for example. They ran two experiments on their pricing page with some surprising results.

Instapage’s pricing page experiments

Experiment #1: Side by side layout

Instapage tested a layout variation that displayed the feature comparisons between self-serve and enterprise plans and used a 70/30 split (70$ to the original and 30% to the variant) to allow more visitors to interact with the variant.

They tracked a goal tied to conversion actions to conclude whether the redesign was worth it.

Here’s the original:

And the variant:

Instapage came up with an initial hypothesis that a redesign would make it easier for visitors to see which features were available on each plan and see if that would increase free trial signups and more inquiries about the enterprise plan.

But they found the original outperformed the variant for all user segments.

Experiment #2: Feature comparison side-by-side

The second pricing page experiment wanted to find out if seeing all the features in a long list would increase conversion rates. The original layout was tested with a variant that had all the features revealed as soon as the page loaded.

The same 70/30 split was followed and the goals were the same. Instapage expected the original to outperform the variant because they believed users would be overwhelmed with all that information but the variant narrowly outperformed the original.

To confirm the result, they analyzed additional metrics like bounce rates, time spent on page, and the conversion rate.

RJ Metrics pricing page A/B tests boosted conversions by 310%

RJ Metrics, a big data analytics provider, suffered the same conundrum many SaaS companies do—how to give users enough information without the paradox of choice and how to cater to the core demographic while still showing other industries they can help.

They built a calculator for their pricing page which users loved but it dramatically decreased the sign-ups. Through heatmaps, they understood that users weren’t scrolling to where the sign-up button was anymore.

Design is riddled with assumptions. Getting good at optimizing a website is largely an exercise in getting good at spotting your own assumptions.

Janessa Lantz, data-driven marketer at RJ Metrics

Through conversion research, RJ Metrics found that users were more interested in seeing two things—how much would this cost them and where could they sign up.

The second variant crushed the original by 310%.

Two simple tests you can run on your pricing page today

Use the concept of price elasticity like Apple does with its iPhone models. By deliberately pitting a higher-end model and a lower-end model and demonstrating the product value, they get more consumers to buy the higher-end version.

The second test you can run is to rearrange the order of your pricing tiers. Instead of following the usual order to arrange the least to most expensive from left to right, place the most expensive pricing plan on the left. This sets the reference aka price anchor. Plus. you can add a callout to the plan that’s most popular to get more buyers to choose that plan.

In the DTC context, try A/B testing discounts vs. no discounts, especially during seasonal sales like Black Friday where herd mentality drives purchases.

You’ll have an idea of exactly how many more purchases and consequently revenue you can drive via the hype and discounts. This will help you justify your efforts or scrap the plan completely.

A good rule of thumb to follow in the e-commerce space is if you wouldn’t do something to a customer in a brick-and-mortar store, you probably should not do it online either. So make sure the difference between price points isn’t unconsciously steep.

Your experiment shouldn’t be aimed at squeezing as much money as you can from your prospect to the point they balk and leave. Instead, A/B testing on Shopify should help you gauge how a sampling of buyers will behave when you expose them to two different price points that are both fair for the same perceived value.

Again, don’t try to let the results of A/B tests guide your pricing strategy. Restrict your extrapolations to buyer behavior and not how much $$$ you can extract.

Testing Different Price Points on Shopify With Convert Experiences: Quick Guide

With Convert Experiences, you can test different product price levels on e-commerce platforms. Here’s a quick step-by-step guide:

1. Clone the product you want to A/B test. Modify the price or other elements you want to test it against. Note: Hide the different version from your product list so it doesn’t show up on your site.
2. Create a Convert Split URL experiment with the original product URL and the cloned URL
3. Add your goals—conversion, revenue, sign ups
4. Start the experiment
5. See which product converted better

You can also run a similar experiment if you’re testing sets of products. Here’s how:

1. Create a Convert Split URL experiment and configure it with a regex expression to transform the original products’ URLs into the new ones.
2. Create a normal Convert A/B experiment and code the redirection on the Variation Custom JS as explained here.
3. The convert.redirect() function will redirect users to the new variation and enable tracking.

4 Shopify Price Testing Apps in 2022: Try these Ones First

This selection is based on user reviews and average ratings, and factors in how responsive the support is, compatibility with multiple themes, effect on site speed, and transparent pricing.

Note: There are dedicated price testing apps and all-in-one CRO (conversion rate optimization) apps that also allow you to test prices.

#1. Dexter: The Price Testing App

#2. Leaflet ‑ Price A/B Testing

#3. Prisync ‑ Dynamic Pricing

#4. Elevate Product A/B Testing

Other price tests you may want to try

These two apps don’t have enough reviews and ratings to be included in the main list but can be considered as an alternative if you’re shopping around.

• Intelligems Price Testing
• Dynamic Price A/B Testing

A/B Testing Pricing Is Legal* and Ethical* but Can’t Replace a Pricing Strategy

We can’t guarantee A/B testing pricing will work for you. (If someone does, run!)

But what we can tell you is that A/B testing prices isn’t illegal or unethical provided you put users first and don’t indulge in deliberate manipulation.

Implementing pricing experiments by launching new pricing, optimizing existing pricing, or innovating with a new model can also be a powerful tool for product-led growth. By testing and iterating on pricing strategies, companies can identify the optimal pricing structure for their product and target audience, which can lead to faster revenue growth and a more sustainable business model.

If you do decide to run tests to experiment with your prices or pricing page, make sure you have a solid WHY and HOW to prevent getting caught up in the latest new trend.

And if you still have qualms about it, start a live chat with one of our experts to understand if there are any legal implications and how you can design ethical tests.