Testing Mind Map Series: How to Think Like a CRO Pro (Part 42)

Interview with Melanie Kyrklund

A bona fide expert in experimentation, Melanie Kyrklund has an impressive 14-year career behind her. Fair to say she knows her stuff.

According to Melanie, the best way to navigate an ever-changing business landscape is to continuously refine your knowledge and experimentation process in order to make the best possible “next decision”.

And before embarking on your journey, Melanie recommends arming yourself with a robust understanding of statistics and measurement principles, learning to connect the dots between your efforts and strategic KPIs, and getting to know the tech landscape you’re operating in. ‘Cause at the end of the day, it’s not just about the tool, but how you use it.

Now let’s take a look at what she has to say…

Melanie, tell us about yourself. What inspired you to get into testing & optimization?

I was interviewing with a large global broadband/TV service provider based out of the Netherlands in 2009. I narrowly missed out on that job but was offered a freelance contract to launch their optimisation program instead. I took the risk and have been richly rewarded ever since, experimentation is a wonderful industry to be in and it has been interesting to see it evolve into a more mature discipline.

As my move into optimisation was driven by circumstance, I can only speak to what has inspired me tostay in experimentation. I have always had an affinity for both science and the arts so the broad remit of this work suits me well. I also love the “applied” nature of this work, where insight is translated into practical solutions for validation. Finally, experimentation has taught me how to be resilient and pragmatic in the face of continuous challenges and change – an essential business skill.

How many years have you been optimizing for?

14 years

What’s the one resource you recommend to aspiring testers & optimizers?

It’s hard for me to answer this question with authority as I joined the industry in its nascent phase. I predominantly learnt by doing and by delving into subjects as necessary to deepen my knowledge. I find it difficult to just pick one. However, if I could pick a quality I look for in aspiring and more seasoned optimizers, this would be self-reliance.

Experimentation requires a multidisciplinary skillset that continually needs to be refined, so it’s important to remain curious and take ownership of your learning trajectory. If you couple that with complex businesses and ever-changing circumstances, you need to be able to draw on your knowledge and a strong experimentation process to confidently make the best possible “next decision.”

Answer in 5 words or less: What is the discipline of optimization to you?

Skilfully navigating uncertainty

What are the top 3 things people MUST understand before they start optimizing?

1. Statistics and the principles behind a robust experiment – hypothesis creation & testing, basic statistics, pre-test calculations, defining a measurement plan, data analytics
2. Opportunity identification – how to tie your efforts to strategic KPIs relevant to the business you operate in and anchor your experiments in well-researched knowledge of your user needs
3. Platform knowledge – understanding the tech ecosystem you are operating in alongside frontend development and UX basics will help you uncover feasible and pragmatic solutions to test

How do you treat qualitative & quantitative data to minimize bias?

When defining the problem/opportunity space, confirmation bias is a risk. This occurs when data is sought to back up an idea or hypothesis. Being aware of this bias (and others) to begin with is important in giving us the awareness of when we may be prone to going down that route.

Using qualitative and quantitative together helps to build a solid evidence-based picture of what a problem is. However, on occasion, analytics may be your only source of insight. You will need to use your creativity, gut feeling and experience to jump from insight to hypothesis because the “why” is not as apparent. I think it’s difficult to control for biases in this process, but I’m fine with that. The objective is to garner sufficient evidence for an experiment.

When evaluating experiments a couple of thoughts spring to mind – the importance of checking for outliers in the data and once again, not resorting to confirmation bias when analysing the results of a hypothesis you are particularly “attached” to.

What is the most annoying optimization myth you wish would go away?

I remember writing a Linkedin article about this back in 2016: the expectation that experimentation would be easy. The truth is experimentation is very hard to do well and consistently at large companies. Scaling programs and creating a culture of experimentation takes years, it’s a continuous process. We need more honest accounts of the journeys we are all on.

Our thanks go out to Melanie for taking part in this interview! To our lovely readers, we hope you found the insights useful and encourage you to apply them in your own optimization efforts.

Don’t forget to check back twice a month for more enlightening interviews! And if you haven’t already, check out our past interviews with CRO pros Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, Tim Mehta, Rommil Santiago, Steph Le Prevost, Nils Koppelmann, Danielle Schwolow, Kevin Szpak, Marianne Stjernvall, Christoph Böcker, Max Bradley, Samuel Hess, Riccardo Vandra, Lukas Petrauskas, Gabriela Florea, Sean Clanchy, Ryan Webb, Tracy Laranjo, Lucia van den Brink, LeAnn Reyes, Lucrezia Platé, Daniel Jones, May Chin, Kyle Hearnshaw, and our latest with Gerda Vogt-Thomas.