Testing Mind Map Series: How to Think Like a CRO Pro (Part 18)

Interview with Siobhan Solberg

Our guest today is Siobhan Solberg, a measurement marketing and optimization mastermind, founder of Raze, and podcast host at Marketing Unf*cked. She’s also an advocate for sustainability, privacy and ethics in marketing.

Needless to say, she knows a thing or two about harnessing data to build a winning experimentation program. For her, there is no “winning” or “failing” in an experiment – it’s all about learning and making good business decisions.

Here’s how she suggests you maximize your efforts:

1. Make sure you are measuring and tracking your data correctly so you can set a solid foundation for your experimentation program.

2. Focus on extracting insights from your data, rather than just looking at wins and losses.

3. Use experiments to learn about your customers and provide a better user experience, rather than just “optimizing” your website.

Let’s dive in…

Siobhan, tell us about yourself. What inspired you to get into testing & optimization?

It happened organically while I was still a musician. I was playing with data and eventually wanted to find solutions to the problems and pain points that I found.

How many years have you been optimizing for?

8 years.

What’s the one resource you recommend to aspiring testers & optimizers?

It’s hard to limit this to one source. I would say stay curious and keep reading/listening about everything, not just CRO. Psychology, Statistics, Marketing, etc.

Answer in 5 words or less: What is the discipline of optimization to you?

Curiosity!

What are the top 3 things people MUST understand before they start optimizing?

1. Basic Statistics
2. Research
3. Analytics/Measurement

How do you treat qualitative & quantitative data so it tells an unbiased story?

Treat it like gold!

But really, just don’t treat it other than removing clear outliers.

More importantly, don’t even collect it if you are not sure that it is an unbiased collection method.

For example, when you are requesting qualitative data via surveys – don’t ask a leading question – it messes up the data before you ever get it.

Another way I love to mitigate being biased is to challenge, question, and counter the assumption I’m going into the data with – this forces me to look at it from another angle.

What is the most annoying optimization myth you wish would go away?

The concept of “winning” or “failing” an experiment. It’s not about winning and losing, it’s about learning. It’s about enabling us to make good business decisions.

We are all stuck on winning so much that we forget that a failing experiment tells us that we were doing pretty darn good before (providing it’s not a bug within the experiment) and that it has given us the opportunity to learn even more about our journey, customers, etc.

We hope Siobhan’s insights will help guide your marketing strategy in the right direction.

What advice resonated most with you?

Keep an eye out for our next interview with a CRO expert who takes us through even more advanced strategies!

And if you haven’t already, check out our interviews with CRO legends Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, and our latest with Bhavik Patel.