Testing Mind Map Series: How to Think Like a CRO Pro (Part 63)

Interview with Tim Donets

Every couple of weeks, we get up close and personal with some of the brightest minds in the CRO and experimentation community.

We’re on a mission to discover what lies behind their success. Get real answers to your toughest questions. Share hidden gems and unique insights you won’t find in the books. Condense years of real-world experience into actionable tactics and strategies.

This week, we’re chatting with Tim Donets, a Growth and Experimentation specialist with over 12 years of expertise in Product management of software products, specializing in Product Growth strategies.

Tim, tell us about yourself. What inspired you to get into testing & optimization?

During my many years as a Product Manager, I was always leaning towards the commercial side of things. It always felt fun and rewarding to work with funnels, user engagement metrics, qualitative and quantitative research, behavioral psychology, etc. At some point, I started looking for a fitting job and landed in Urban Sports Club, a Berlin-based sports and wellness startup, where I was responsible for the user acquisition journey from the Product Management side. My team had a CRO manager embedded, and we ended up testing quite a few things. It made me realize that I like this quite a lot, but my job as a Product Manager did not allow me to spend significant time doing Experimentation.

Later, as I started consulting, I got a chance to work in a CRO agency as a CRO consultant, and this is where it clicked. I enjoyed my time there thoroughly and made a decision to shift my career towards Growth PM’ing and CRO.

How many years have you been testing for?

Adding up the years when I was taking part in Experimentation with those when I ran tests end-to-end myself – roundabout 4.5 years.

What’s the one resource you recommend to aspiring testers & optimizers?

It would be hard to pinpoint ‘the one’, but I’d say CXL for general learning. Their courses are really good and very hands-on. I have finished a couple and was blown away by the amount of useful stuff they are sharing. I’d also give a shoutout to analytics-toolkit.com by Georgi Georgiev, especially his blog articles. For people who want to dive into a rabbit hole of relevant statistical concepts and techniques like sequential testing – it is a goldmine and I have spent hours reading through those articles.

Answer in 5 words or less: What is the discipline of optimization to you?

Understanding and changing things holistically.

What are the top 3 things people MUST understand before they start optimizing?

1. It is easy to start but progressively harder to scale.
   
   Really, with the number of tools, resources and information, any business can start optimizing their digital presence in very short order. As an example, when I joined The Kyiv Independent as a consultant, we were able to launch our first A/B test in 1 week, following a quick integration of a testing tool and setting up some processes. But going from 1 test a month to even 3-4 tests a month and then beyond is a challenge and requires the right people, culture and resources.

2. Always have a hypothesis to confirm or disprove.
   
   Testing for the sake of it and interpreting the results as whatever you wanted to confirm leads to bad results. Always generate a viable hypothesis that not only predicts the effect of the test but also explains why this effect is expected. And generating a good hypothesis requires at least some amount of qualitative and quantitative research. Every time I skimped on the effort to write a hypothesis I would later regret it. Interpreting results becomes an act of fortune telling and you have no clear way forward when there was no solid hypothesis in the beginning.

3. Everything is connected.
   
   While you can start with very tactical CRO, optimizing singular elements or pages, you will eventually have to start looking at things holistically. Truly transformative experimentation happens when you collaborate with others to get a clear picture of a user journey. Which marketing channels are users coming from? What are they seeing in those ads, and how is your homepage / landing page experience matching their expectations? What are your customers doing right after the registration? Which features should they be aware of? What is the best path for every user to arrive at their desired outcomes? Only when looking at things in their entirety can a good CRO team / person move the needle on commercial metrics that matter.

How do you treat qualitative & quantitative data to minimize bias?

I guess any data is inherently biased in some way. With quantitative, you as a person apply your biases to how you interpret it. With qualitative, the respondents can be biased, but the interpretation of their answers can be as well.

Usually, I just cross-check these two sources. If what I am seeing in the qual explains what I am seeing in the quant and vice-versa – this is a good sign. I am also trying to inform one type of research with another – I am looking at this particular behavioral data because some users reported X in the latest set of interviews, or I ask this particular question because I have seen trend Y in the data.

How (to you) is experimentation different from CRO?

Conceptually, experimentation can be many things, as long as there is a hypothesis and a way to prove or disprove it. Doing a “fake door” test is still an experiment although it has nothing to do with optimizing conversions. So for me, experimentation is just a much broader field, rather an approach to making decisions, reducing risks and informing strategy, while CRO is a set of activities focused on improving a specific metric, even if it is not at all tactical.

Talk to us about some of the unique experiments you’ve run over the years.

Once we got to run a test for a full redesign of the functional page, like a wishlist area for an e-com or a marketplace. The complicated part was that the new designs (and there were not one but TWO competing versions plus control) had a lot of flows and elements that did not exist in control but which we had to somehow recreate within Optimizely DOM manipulation toolkit. Figuring out key metrics and how to better understand which change might have affected metrics the most was also a challenge. All in all, it was something I would rather avoid doing in the future; dev and QA took like 2 months. And in the end, the results were exactly the opposite of what we had expected from both versions in terms of user behavior.  

Huge thanks to Tim for generously sharing his expertise with us. This kind of hard-won wisdom is pure gold. To all you optimizers out there, hopefully Tim’s insights provided some useful food for thought as you continuously level up your skills.

We have fresh interviews dropping twice a month, so bookmark this blog and check back often. And if you missed any of our previous convos with CRO pros, now’s the time to get caught up and give these a read: Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, Tim Mehta, Rommil Santiago, Steph Le Prevost, Nils Koppelmann, Danielle Schwolow, Kevin Szpak, Marianne Stjernvall, Christoph Böcker, Max Bradley, Samuel Hess, Riccardo Vandra, Lukas Petrauskas, Gabriela Florea, Sean Clanchy, Ryan Webb, Tracy Laranjo, Lucia van den Brink, LeAnn Reyes, Lucrezia Platé, Daniel Jones, May Chin, Kyle Hearnshaw, Gerda Vogt-Thomas, Melanie Kyrklund, Sahil Patel, Lucas Vos, David Sanchez del Real, Oliver Kenyon, David Stepien, Maria Luiza de Lange, Callum Dreniw, Shirley Lee, Rúben Marinheiro, Lorik Mullaademi, Sergio Simarro Villalba, Georgiana Hunter-Cozens, Asmir Muminovic, Edd Saunders, Marc Uitterhoeve, Zander Aycock, Eduardo Marconi Pinheiro Lima, Linda Bustos, Marouscha Dorenbos, and our latest with Cristina Molina.