Ethics in A/B Testing. The Good, The Bad, The Future

Have you ever thought about ethics in A/B testing?

We all have our own ethical compass guiding us to what’s right and wrong. And as digital businesses, we should have our user’s best interests at heart, right?

Well, that’s where the waters get a little murky. A/B testing is all about growth, we both know this. And as such, it’s easy to overlook the effect our tests have on the end user. We probably just want to grow our revenue and make a profit.

Sure, we don’t set out to do anything unethical.

We may even try to stay away from the things we believe are frowned upon, like countdown timers that aren’t real, or stock counts that aren’t linked to the back end.

But what if our product genuinely can change people’s lives? What if our product can make people happier, healthier, or wealthier? We can argue that if our product does do one of these, we should do anything in our power – use any marketing or sales tactic – to get our product into the hands of the end user.

Like I said, it’s murky waters we’re swimming in. Business KPI’s and user interests are not always aligned. In fact, I’d hazard a guess to say that they’re definitely not aligned for the majority of companies.

So let’s take a deeper look at how we can navigate these ethical waters, keep our customers happy, and grow our revenue.

How Do Ethics Fit In with A/B Testing?

I was recently speaking with Bart Schutz, a behavioral psychologist and an expert in A/B testing. On our call he gave the perfect example of an A/B test that on the surface looks like a winner  but had huge ethical implications for the end user.

Imagine going to a booking website and seeing the cleanliness of hostels emphasised. You’d probably think it was a great idea. As you can imagine, the hostels with the best cleanliness score would get the most clicks. And they did. Sales increased, everyone was happy, and with no ethical issues in sight.

That was until their lead psychologist found an article that researched the association between cleanliness and feelings of safety, especially among women. And by emphasising the cleanliness of hostels, they could induce implicit feelings of safety, possibly driving more women to what could be actually unsafe hostels.

On the surface this A/B test looked to benefit both the business and the customer. The business benefited by an increase in bookings, and the customer seemed to benefit by seeing the cleanliness of the room.

However after doing the experient and analysing the data, it turned out to create a big ethical dilemma. What was in the businesses best interest wasn’t necessarily in the customers, especially their female customers, best interest.

This a very specific example and might not relate to your business at all, but it does show how A/B tests can hugely impact the end user.

And how without proper data analysis and research by psychologists, this link might never have been found.

So What Does Ethical A/B Testing Look Like?

We can guess all we want about the predicted results of our A/B test experiments.

But until we’ve conducted the experiment, got back accurate data, and done the right analysis… we simply don’t know the ethical implications.

Sure sometimes it’s obvious, and if it is, then it’s our duty as humans to speak up and let our voices be heard. Whether that’s from the client, the agency, or whoever is doing the testing. But as we’ve seen from the booking site example, sometimes the ethical implications of experiments are very hard to see.

And that’s really the message here.

Human behaviour is so complex that we’ll often overlook the far ranging effects  our experiments can have. Because when we’re unaware of it, we’re not analysing it. And if we don’t analyse it, we could be pushing tests that hurt people’s wealth, health and happiness.

What Should You Consider When Approaching A/B Testing?

The first step is to ensure that our data is completely trustworthy, then start experimenting and analysing based on this trustworthy data, ensuring trustworthy outcomes.

And the only real thing we can do to be ethical, is to be rigorous in our analysis of the possible negative or harmful effects of our testing.

The next time you make an A/B test, ask yourself the ethical question. Right now, with all the trustworthy data and knowledge you have, could this test have negative implications for your end user? And if you choose to experiment, make sure company interests and user interests are aligned.

Although this is a tough one to do. We must weight up short term gains with long term KPI’s, taking our end user into account.

The fact is, businesses drive more revenue, customer lifetime value and margins and they do it quicker when they experiment.

Being experiment driven can also save a lot of time and money validating ways to  best prioritize IT resources based on the value they bring

We just need to make sure that our success metrics include both business and user interests. And we should always make an effort to research and analyse the ethical implications of our actions.

The Future Of Ethics And User Interest As A Business Metric

More and more companies are seeing the importance of asking themselves the ethical questions and aligning business interests with the interests and happiness of their users and customers.

Certainly It’s never black and white, especially in the murky waters of ethical A/B testing and driving business growth.

And It’s never the tactics that are unethical, but the intention and the cost/benefit analysis for those who are ultimately subjected to the testing.

The fact is, we first need trustworthy data – based on proper research, with large data sets.

The next step is making sure that our experiments and result analyses are done accurately.

Furthermore, when we take into account the end user and base some of our metrics on whether a test is good or bad for them, we help create businesses that do good in the world and stay around for the long term.

Let’s wrap this up with some action items you can work on right away.  What can you do to make sure you’re being ethical with your testing and are aligned  with your customers’ best interests? Here are some points that can give you better direction:

1. In an ideal situation you’d have behavioural psychologists looking through research -analysing and identifying correlations in the data. But if this isn’t possible, then really dive deep into the data and the possible effects of the experiment on your end users to the best of your ability. This is how you’ll catch tests that on the surface appear to be ethical but actually may cause harm.
2. Ask yourself ethical questions before the test begins and encourage team members to speak out if they have issues with testing. We’re all human and have differing levels of ethical sensibility. So what crosses the line for one person may not for another. That’s why an environment of open discussion and freedom to speak out can be key.
3. Guessing and opinion can only take us so far. The only way to know the result of an a/b test and it’s ethical implications is to run the experiment and gather accurate data. In the words of Nike, “Just do it.”
4. Sure we can ask our users to give permission in some way to the testing, or even ask their opinion on the tests and whether they feel manipulated by them. But keep in mind that how people view ethics in relation to commerce can vary wildly from country to country.
5. Look at your KPI’s and see how you can align business goals with the health, wealth, and happiness of your end users. ‘Time spent on site’ is a good metric for media companies wanting to monetize their platform. But ‘time well spent’ is a much better way to look at monetization in the context of the experience end users have with your brand. Users who are happy and healthy and derive value from your products in the long term will always lead to an increased lifetime value and reduced churn.

As end users ourselves, we’ll always gravitate towards the businesses that have our best interests at heart. And it’s these businesses that become brands which flourish in the modern world of increased accountability and transparency.

So let me ask you a question, how are your business metrics taking your customers best interests into account? And if not, how can you make sure they do moving forward?