Testing Mind Map Series: How to Think Like a CRO Pro (Part 20)

Interview with Rommil Santiago

Rommil Santiago has been championing brilliant voices in experimentation through his work at Experiment Nation for years.

With an impressive career spanning 13 years, Rommil says he’s always been passionate about experimentation – trying out new ideas and seeing how they perform. This approach has served him well throughout his career, as he’s held roles in marketing, product development, growth, design, and analytics.

He holds passionately argued views on why optimization isn’t just about one framework, stats model, or tool. After all, it’s an important part of the journey toward truth. And in order to make people less skittish about it, we need to stop hiding behind fancy terms and frameworks and make it more approachable.

So, if you’ve been finding optimization daunting, then this is the place for you!

Let’s see what Rommil had to say…

Rommil, tell us about yourself. What inspired you to get into testing & optimization?

My career has zig-zagged across domains and industries over the years. I’ve held roles in Marketing, Product, Growth, Design, Analytics, and even Engineering. Throughout my career – the one thing that I always loved doing was experimenting,  trying out new ideas, and seeing how they performed. I got my first taste of experimentation when I was an Analytics and Optimization Consultant for Bell Canada where I developed their landing page optimization practice using what used to be called, Adobe Test and Target. I guess I’ve been testing ever since!

How many years have you been optimizing?

I like to think I’ve been optimizing for my entire life – but if we’re talking strictly about the digital domain, I’d say since 2009, or 13 years.

What’s the one resource you recommend to aspiring testers & optimizers?

ExperimentNation.com is a great resource, haha!

But self-promotion aside, I think one of the most useful books I’ve ever read is Steve Krug’s Don’t Make me Think. It’s not specifically an experimentation book, but it explains how one can do user testing on a shoestring budget. It can be read in a couple of hours – but the lessons have stayed with me for years.

Answer in 5 words or less: What is the discipline of optimization to you?

Make better decisions with evidence.

What are the top 3 things people MUST understand before they start optimizing?

Great question. Hmm, I guess that would be:

1. You will be wrong sometimes, and that’s OK. Experimentation is the path towards truth.
2. You will be right sometimes, enjoy those times but never let them get to your head.
3. Communicate clearly and be gracious. Experimentation is full of fancy and confusing terms. It’s easy to lose people in all the frameworks and stats. But if you really want people to experiment more, you must make it more approachable.

How do you treat qualitative & quantitative data so it tells an unbiased story?

Well, all data is biased – as is all storytelling.

I like quantitative data as I’m most familiar with it, but I always do my best to seek out qualitative data to add clarity to what is going on.

So I guess I’d say they are two halves of the same store.

That aside, perhaps my best advice on avoiding bias is to get a second (or more) pair of eyes to review your analysis. Ideally, someone not as close to the work as you are. Sometimes a peer can call out if you are bringing bias to the analysis.

What is the most annoying optimization myth you wish would go away?

That there is only one way to run experiments. Be it a framework, a stats model, or a tool – practitioners can become so entrenched in their favorite flavor of testing that they attack anything and anyone that says otherwise.

Experimentation has been around a long time before digital was ever a thing and there are countless ways to come to a problem – just with different levels of bias and external applicability. I’d wager that those who run experiments in psychology or health sciences would laugh at the looseness of our rigor.

But at the end of the day, find the method which works for you – where you are constantly learning – and that’s a great start. That said, you should probably double-check your math – just to be safe. 😉

Thanks to Rommil for sharing so many valuable insights! What advice resonated most with you?

Keep an eye out for our next interview with a CRO expert who takes us through even more advanced strategies!

And if you haven’t already, check out our past interviews with CRO legends Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, and our latest with Tim Mehta.