Testing Mind Map Series: How to Think Like a CRO Pro (Part 56)

Interview with Edd Saunders

Every couple of weeks, we get up close and personal with some of the brightest minds in the CRO and experimentation community.

We’re on a mission to discover what lies behind their success. Get real answers to your toughest questions. Share hidden gems and unique insights you won’t find in the books. Condense years of real-world experience into actionable tactics and strategies.

This week, we’re chatting with Edd Saunders, Optimisation Consultant & Strategy Specialist at LeanConvert, a fast-growing customer experience, analytics, and engineering company, helping some of the world’s biggest brands drive digital transformation.

Edd, tell us about yourself. What inspired you to get into testing & optimization?

For as long as I can remember, I’ve had a “See what happens” mentality, meaning I enjoy testing things, experimenting, trying new approaches, peeking under the hood to understand how things work, exploring the unknown.

My first job after university was as a recruitment consultant, where a large part of the work was cold calling. I hated the inefficiency of this approach (a very small number of those who you call will answer, let alone agree to be put forward for a job) so I had to search for the other tools at my disposal to get candidates interested in jobs. Email was my preferred approach because a large problem I discovered with hiring software devs pre-covid is they typically worked in open-plan offices, which were not conducive to taking interviews.

I decided to test different email subject lines and measure the response rates. This evolved into testing email content, frequency, etc.—all logged in my trusty spreadsheet. Absolutely 0 statistical rigour, but the approach helped me open up a much (significantly 😎) larger pool of candidates.

Researching how to improve my approach opened my eyes to the world of digital marketing, analytics, and eventually CRO.

How many years have you been testing for?

It’s been around 8 years since I launched my first A/B test! My approach is far more methodical and statistically robust these days, but the enjoyment of “seeing what happens” still remains. My experience spans digital marketing and product analytics, and I’ve been consulting for the last 4 years.

What’s the one resource you recommend to aspiring testers & optimizers?

The CXL blog is a good starting point and helped me a lot when I was getting started. There are lots of interesting communities popping up these days which are worth joining, such as Experiment Nation (I helped put together the CRO certification, check it out!) and the CRAP Talks community, if you’re London based and like in-person meet-ups. (Sorry, that was 3.)

Answer in 5 words or less: What is the discipline of optimization to you?

Finding & solving real user problems.

What are the top 3 things people MUST understand before they start optimizing?

1. Remember that you’re influencing the behaviour of real people who are using a digital product. Your work has the potential to impact people all across the globe.

2. A customer journey doesn’t focus on just one page. Think wider. How does the change you’re proposing impact the user on their onward journey?

3. Don’t chase ideas.  By all means, look for inspiration, but test ideas will be the natural output of your research work. Spend time doing your primary research, follow a solid process, uncover your assumptions about what problems users are facing and the ideas will flow easily.

How do you treat qualitative & quantitative data to minimize bias?

Map your customer’s journey using quantitative data, identify problem areas, then layer on the qualitative to hone in on specific problems.

Making assumptions is something we all do, so embrace them. An assumption is the conclusion we draw based on the data available to us. Recognise them but don’t discount them (there’s a fantastic framework called assumptions mapping which you can use to help you identify the good ones). Better yet, collaborate with someone else to bring out the best ideas.

How (to you) is experimentation different from CRO?

Experimentation is a mindset, whereas CRO is process oriented.

CRO revolves around improving specific KPIs on a digital product, is statistically rigorous, and designed to inform business decisions. 

Experimentation is trying things to explore new outcomes, testing with the intention of learning.

(Side note: I don’t like the argument around whether we should call ourselves CROs or not, but if I had to choose my contribution would be Internet Scientist.)

Talk to us about some of the unique experiments you’ve run over the years.

I enjoy the ‘MVP-minded’ experiments as they have the biggest bang for buck, i.e. the simplest solution to solve the identified problem such as changing copy or serving nudges, prompts & messages using intent-based triggering rules.

On the other end of the spectrum, a complex campaign that sticks out involved building a custom CDP to solve a specific personalisation problem, to overcome a platform limitation.

Prior to my son being born, I experimented with ways of improving my deep & REM sleep, such as not looking at screens 2 hours before bed time, magnesium supplementation, intermittent fasting. I recorded the data using a Whoop band and logged the results in a trusty spreadsheet to identify trends.

Our thanks go out to Edd for taking part in this interview! To our lovely readers, we hope you found the insights useful and encourage you to apply them in your own optimization efforts.

Don’t forget to check back twice a month for more enlightening interviews! And if you haven’t already, check out our past interviews with CRO pros Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, Tim Mehta, Rommil Santiago, Steph Le Prevost, Nils Koppelmann, Danielle Schwolow, Kevin Szpak, Marianne Stjernvall, Christoph Böcker, Max Bradley, Samuel Hess, Riccardo Vandra, Lukas Petrauskas, Gabriela Florea, Sean Clanchy, Ryan Webb, Tracy Laranjo, Lucia van den Brink, LeAnn Reyes, Lucrezia Platé, Daniel Jones, May Chin, Kyle Hearnshaw, Gerda Vogt-Thomas, Melanie Kyrklund, Sahil Patel, Lucas Vos, David Sanchez del Real, Oliver Kenyon, David Stepien, Maria Luiza de Lange, Callum Dreniw, Shirley Lee, Rúben Marinheiro, Lorik Mullaademi, Sergio Simarro Villalba, Georgiana Hunter-Cozens, and our latest with Asmir Muminovic.