Testing Mind Map Series: How to Think Like a CRO Pro (Part 16)

Interview with Ryan Thomas

There are a lot of best practices in the CRO world that people tend to follow blindly. “You must run a test for at least two weeks”, for example, or “you should only run one test at a time on the same page”.

In this article, Koalatative Co-Founder Ryan Thomas challenges people to think about when and why it makes sense to discard those best practices. And to actually focus on what matters most: becoming good at talking to customers and understanding their motivations and concerns.

Ryan believes that test duration or the number of concurrent tests shouldn’t be set in stone, as they are highly contextual.

In the real world, conversion rate optimization has few hard-and-fast rules – so always challenge the dogma and learn by doing!

Now on to the interview…

Ryan, tell us about yourself. What inspired you to get into testing & optimization?

I was vaguely familiar with CRO for a long time since the early 2010s from reading blog posts and watching videos. I tried to apply some of the concepts to the websites I was building, but I wasn’t involved with any projects where I could really put it into practice at scale.

One of the things that inspired me to make a career shift in that direction was reading Steve Blank’s books “The Startup Owner’s Manual” and “The Four Steps to the Epiphany” where one of the main themes is “getting outside the building” to challenge the assumptions and biases of the internal team.

The combination of analytics data, qualitative research, and testing in CRO seemed like a perfect way to do that for websites. And not just for startups, since it seems like organizations can get really big and still operate mostly based on opinions generated inside the “building”, so there’s a lot of opportunity to provide that outside, customer-focused and data-informed perspective.

How many years have you been optimizing for? What’s the one resource you recommend to aspiring testers & optimizers?

I made the decision to specialize in CRO in 2017, and I signed up for CXL’s certification course which led to me being hired by the agency side of their business shortly after I completed it. It was basically a catapult directly to the top of the industry. There are a lot of courses out there but none of them really compare to CXL so I definitely recommend that as a solid starting point for aspiring optimizers.

But, if I had to pick one resource it would be: customers. If you can get good at actually talking to customers and understanding their motivations, needs, hesitations, and how they think, then courses to learn about specific tools and methodologies are just icing on the cake. When you first get used to this idea it almost seems like cheating.

The answers to some of the hardest business mysteries are just a question or two away if you are engaged with the people you are trying to sell to.

Answer in 5 words or less: What is the discipline of optimization to you?

Hill-climbing algorithm for website performance.

What are the top 3 things people MUST understand before they start optimizing?

1. What you see in quantitative data is more likely to be an artifact of the tracking setup than real user behaviour.
2. The case studies you see with 500% uplifts and no information about the test planning and sample size are total BS.
3. There’s a lot of dogma in the CRO world that you absolutely should challenge. Things like “you have to run a test for at least 2 weeks” or “you should only run one test at a time for the same page and/or metric”. Pretty much everything depends on the context, there’s always more to the story, and while there may be a legitimate basis for some of these rules of the trade, none of them should be set in stone. So follow the dogma at first, but as you learn you should figure out when and why those rules should be broken.

How do you treat qualitative & quantitative data so it tells an unbiased story?

Quantitative data tells you what’s happening, qualitative tells you why.

I typically poke around in the analytics, looking at different reports, and then when I see something interesting I drill into it by applying segments and secondary dimensions, and check the tracking setup to figure out if it’s real user behaviour.

Once tracking issues have been ruled out, I review the page(s) involved for clues that explain the finding, and then set up some qualitative data collection to get the user / customer perspective.

That alone is a pretty good start to eliminating bias, but running an experiment to test your hypothesis gets you way closer (you can never eliminate bias entirely).

Another tip for this is what I call a “naive assessment”. Your naivety at the beginning of a project is actually a really valuable resource, so try to make use of it before you learn too much about the website, the customers, and the business. It’s easier to think like a customer when you don’t know any more than they do about the business, so I like to do an initial light audit as early as possible in the project.

What kind of learning program have you set up for your optimization team? And why did you take this specific approach?

Our team is still small, but generally, our new hires will have already taken some courses on CRO. So the initial learning will be based on the content that we produce at Koalatative, which is designed to cut through the noise of all the other courses and content out there while providing a more high-level view on how to approach various topics in CRO.

Once they have the basics of how we think about things and our processes, then the best way to learn is by doing. So we’ll get them into client calls and then let them take ownership of a piece of the CRO puzzle at a time. There are tons of capable and motivated people out there who want to move into CRO as a career and have taken lots of courses, but have a hard time getting any of it to stick because they don’t have a way to apply what they are learning.

What is the most annoying optimization myth you wish would go away?

That CRO is a bad name for what we do and that it should be changed. There are a lot of prominent voices pushing this idea, and in most or all cases it can be traced back to a deliberate strategic narrative to push a specific product or tool, or a tactic of differentiation.

CRO is finally getting more mainstream attention after years of obscurity, and sure there are people out there doing it in a scammy way, but it seems like a step in the wrong direction to just let those people have the term and start all over again to build awareness of whatever the new term is. If you play it forward the same cycle will just continue, so it’s up to the white hats in the industry to promote good practices and educate practitioners on how to do CRO the right way.

Hopefully, our interview will help guide your experimentation strategy in the right direction!

What advice resonated most with you?

Be sure to stay tuned for our next interview with a CRO expert who takes us through even more advanced strategies!

And if you haven’t already, check out our interviews with CRO legends Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski and our latest with Ryan Levander.