Testing Mind Map Series: How to Think Like a CRO Pro (Part 46)

Interview with Oliver Kenyon

Oliver is the Co-Founder of ConversionWise. They provide solutions that skyrocket your business and make high conversion rates achievable for anyone! Since founding the company in 2013 Oliver and Andy have built a team of 30+ conversion rate experts and have worked on over 3000+ projects for the likes of Unilever, SlimFast, Lotus Biscuits, BodyBuilding.com, Rob Lowe, Manny Pacquiao, T-Pain and so many more. They currently manage ongoing CRO for some of the biggest brands in the world including, Lionel Messi, GFUEL, and Ezra Firestones Boom by Cindy Joseph.

How many years have you been optimizing for, Oliver?

Almost 11.

What’s the one resource you recommend to aspiring testers & optimizers?

Our conversion rate academy and CXL.

Answer in 5 words or less: What is the discipline of optimization to you?

Test everything!

What are the top 3 things people MUST understand before they start optimizing?

1. If you have less than 100,000 UV per month, then focus on conversion design.
   Before diving into A/B testing and other forms of optimization, businesses with less than 100,000 UVs per month should focus on the fundamentals of conversion design. This involves crafting a user experience that is intuitive, engaging, and encourages conversions. Factors like website layout, call-to-action buttons, and even color schemes can significantly impact user behavior. Ignoring these basic elements and jumping straight into optimization tactics can lead to inaccurate or misleading test results.

2. Not all tests win.
   Many people believe that every optimization effort should yield positive results, but that’s far from the truth. Optimizing is as much about learning what doesn’t work as it is about confirming what does. Failed tests are not wasted efforts; they’re opportunities to learn more about your audience and improve your strategies. It’s crucial to manage expectations and prepare for both wins and losses in your optimization journey.

3. Big results don’t come from one test, they come from constant compounding winning tests.
   Optimizing your conversion rate isn’t a sprint; it’s a marathon. A single test, no matter how successful, won’t transform your business overnight. Sustainable growth and significant improvements come from consistently running tests and implementing small changes that compound over time. It’s this iterative process that allows you to refine your strategies based on what you’ve learned and thus achieve significant long-term gains.

How do you treat qualitative & quantitative data to minimize bias?

Minimizing bias in the treatment of both qualitative and quantitative data is essential for accurate and effective optimization. This is especially important as conversion rate optimization (CRO) isn’t a “one size fits all” strategy.

Qualitative Data: For qualitative insights gathered from interviews, surveys, or user feedback, it’s essential to have a structured methodology for interpretation. By categorizing and coding responses, you can begin to see patterns and trends that are less influenced by subjective opinions. It’s also helpful to have multiple team members review the qualitative data to cross-verify interpretations.

Quantitative Data: When it comes to quantitative metrics like bounce rates, time on page, or conversion rates, it’s crucial to apply rigorous statistical methods to ensure the reliability of your findings. Make sure to use a sufficiently large sample size and run the test long enough to reach statistical significance. Additionally, use control groups and randomization to mitigate variables that could introduce bias.

Combining Data Types: Balancing both qualitative and quantitative data provides a more holistic view of what’s happening. While numbers can show you patterns and trends, qualitative data can help explain the “why” behind those numbers. When you ‘test everything,’ as the mantra goes, it’s important to let these different types of data complement each other. Give your tests enough time to collect a robust set of data so that the results speak for themselves, reducing the chance of biased interpretations.

What is the most annoying optimization myth you wish would go away?

The most irksome myth that persists is the idea that “CRO is a one-time thing.”

This notion misguidedly treats optimization as a task you can check off a list, rather than an ongoing, iterative process. Conversion rate optimization is not a “set it and forget it” activity; it requires regular attention, testing, and refinement. Markets evolve, consumer behavior changes, and website performance fluctuates—all of these factors necessitate continuous optimization. By thinking of CRO as a one-off task, businesses risk becoming complacent, missing out on valuable insights, and ultimately failing to maximize their conversion potential.

Our thanks go out to Oliver for taking part in this interview! To our lovely readers, we hope you found the insights useful and encourage you to apply them in your own optimization efforts.

Don’t forget to check back twice a month for more enlightening interviews! And if you haven’t already, check out our past interviews with CRO pros Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, Tim Mehta, Rommil Santiago, Steph Le Prevost, Nils Koppelmann, Danielle Schwolow, Kevin Szpak, Marianne Stjernvall, Christoph Böcker, Max Bradley, Samuel Hess, Riccardo Vandra, Lukas Petrauskas, Gabriela Florea, Sean Clanchy, Ryan Webb, Tracy Laranjo, Lucia van den Brink, LeAnn Reyes, Lucrezia Platé, Daniel Jones, May Chin, Kyle Hearnshaw, Gerda Vogt-Thomas, Melanie Kyrklund, Sahil Patel, Lucas Vos, and our latest with David Sanchez del Real.