Testing Mind Map Series: How to Think Like a CRO Pro (Part 50)

Interview with Shirley Lee

This week, we’re excited to feature Shirley Lee, a seasoned Experimentation Specialist and CXL Certified Optimizer who’s on a mission to challenge the status quo and free the world from the HiPPO syndrome.

She offers a refreshingly candid take on the challenges of experimentation, drawing a parallel with the legal world. Lawyers and experimenters can meticulously prepare for success, yet neither can predict the outcome.. In experimentation, preparation is crucial, but immediate huge uplifts are in short supply.

Shirley stands firmly in favor of a data-driven approach, skillfully mining insights from analytics and customer feedback and making smart use of statistics. She’s all about making every test count and every insight actionable, in order to move towards genuinely effective optimization practices.

Join us as we delve into Shirley’s journey…

Shirley, tell us about yourself. What inspired you to get into testing & optimization?

Beginning as a management consultant, I observed large companies relying on the opinions of the highest-paid persons and using data to reinforce their biases rather than challenging the status quo. This drove me to seek a more data-driven role.

Furthermore, I am passionate about data, (digital) products and consumer psychology, and I’d love to work at their intersection. Fortunately, I quickly discovered the world of Experimentation, and once I found it, I knew I had found my dream career. 🙂

How many years have you been optimizing for?

I’ve fully specialized in experimentation for 2 years! 😇

Yet in a broader sense, I’ve accumulated about 4 years of experience as an optimizer through previous roles, where I worked on digital optimization and instilling organizations with ‘test and learn’ mindset.

What’s the one resource you recommend to aspiring testers & optimizers?

For theoretical knowledge, I highly recommend CXL courses, the podcast “From A to B,” and the YouTube channel “Treachery of Data.”

Nevertheless, the most effective way to learn is by getting “hands-on” experience. Every time you engage in executing an experiment or conducting research, you’ll encounter nitty-gritty challenges (such as technical complexity). Once you overcome these challenges, you’ll become a more skillful practitioner.

Answer in 5 words or less: What is the discipline of optimization to you?

Process for better decision-making

What are the top 3 things people MUST understand before they start optimizing?

1. Get clarity on the company strategy and KPIs: Ask the stakeholders upfront what targets and KPIs the product needs optimization for. Begin your process with this step, so as to prevent heading down the wrong path and wasting time.
2. Don’t skip CRO research: Rather than relying solely on “best practices” for test ideas, derive them from your analytics data and customer research. By doing so, you’ll be testing against real user bottlenecks and specific opportunities tailored to your website. This approach will make your program more effective.
3. Understand statistics and incorporate that in your process: For example, ensure you calculate runtime and estimate MDE (and why you should do this) during your pre-test planning; and be cautious about SRM (and what that is) when monitoring an experiment. To learn about statistics for experiments, check out: Ronny Kohavi and Georgi Georgiev.

How do you treat qualitative & quantitative data to minimize bias?

Focus on reducing bias and noise when handling data, hence practical tips:

• Consider what data to collect that would align with your goal; and how to track it to match real user behaviors closely.
• Start with a solid hypothesis and stick with it throughout. Don’t tweak it just to make the experiment “suddenly” win.
• How you treat your data depends on how big the risk is for the decision. For high-stakes decisions, you can collect a bigger volume of data from multiple sources. For lower-risk decisions, sometimes a good experiment or qualitative research will do the trick, depending on the context.

What is the most annoying optimization myth you wish would go away?

Myth: CRO guarantees ROI.

The term CRO, or Conversion Rate Optimization, can create false expectations about what it can achieve. ‘Experimentation’ might be a better term, as it implies we are solving broader problems.

Experimentation doesn’t guarantee a specific ROI or overnight results. Instead, it offers a process for making better decisions. It enables teams to make data-informed decisions, leveraging insights from experiments and research.Think of experimenters as lawyers. Lawyers prepare their clients for a trial, but they cannot guarantee a win. Experimenters offer organizations a discipline to foster their data-driven culture, but they cannot promise immediate (huge) uplifts.

Our thanks go out to Shirley for taking part in this interview! To our lovely readers, we hope you found the insights useful and encourage you to apply them in your own optimization efforts.Don’t forget to check back twice a month for more enlightening interviews! And if you haven’t already, check out our past interviews with CRO pros Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, Tim Mehta, Rommil Santiago, Steph Le Prevost, Nils Koppelmann, Danielle Schwolow, Kevin Szpak, Marianne Stjernvall, Christoph Böcker, Max Bradley, Samuel Hess, Riccardo Vandra, Lukas Petrauskas, Gabriela Florea, Sean Clanchy, Ryan Webb, Tracy Laranjo, Lucia van den Brink, LeAnn Reyes, Lucrezia Platé, Daniel Jones, May Chin, Kyle Hearnshaw, Gerda Vogt-Thomas, Melanie Kyrklund, Sahil Patel, Lucas Vos, David Sanchez del Real, Oliver Kenyon, David Stepien, Maria Luiza de Lange, and our latest with Callum Dreniw.