Testing Mind Map Series: How to Think Like a CRO Pro (Part 43)

Interview with Sahil Patel

Today, we sat down with Sahil Patel, CEO of Spiralyze, for an interview you won’t want to miss. Be prepared to be blown away.

First off, Sahil says that just mimicking your competitors’ playbook is a fool’s errand. This is especially true for B2B SaaS. Instead, look more broadly and you’ll discover a treasure trove of insights from companies outside your vertical.

But that’s not all. Sahil also takes a swing at the myth surrounding the statistical rigor required for optimization. It’s less like a clinical drug trial, and more like… well, choosing an outfit. An unexpected metaphor that shows that the stakes of being wrong aren’t as dire as they seem. As trends shift, so too does the dynamic world of A/B testing. It’s not about achieving perfection from the get-go, but continuously iterating and improving.

Get ready for a treat…

Sahil, tell us about yourself. What inspired you to get into testing & optimization?

I was an early client of Spiralyze’s – back in 2015 – I was the CEO of a SaaS healthcare tech company (ER Express).  It opened my eyes to the power of experimentation.  Some tests that I didn’t think would win did, and it changed our design decisions from classic HIPPO (by the way, I was obviously the guilty party) to a more data-driven one.  I joined the company last year (after selling ER Express) because I wanted to help more companies benefit from predictive CRO.

How many years have you been optimizing for? 

Going back to 2015, so it’s been 7+ years.  For context, I’ve been at Spiralyze for one year.

What’s the one resource you recommend to aspiring testers & optimizers?

You mean besides Spiralyze? 🙂

Get data on what test ideas will work – not just anecdotes, not just ideas from a podcast or blog (those are fine as a starting point) – because testing is a cruel game and you need to run a lot of tests to find wins.

Answer in 5 words or less: What is the discipline of optimization to you?

Relentlessly find better outcomes.

What are the top 3 things people MUST understand before they start optimizing?

1. Don’t rely on anecdotes to come up with test ideas.  Copying your competitors is a classic fallacy.  Well, our competitor made this change, and they have the same ICP, so it will work for us.  It probably won’t.
2. Testing is a cruel game.  Once you pick the low-hanging fruit, unless you have information on aggregated winning tests, the win rate is 11%.  So you need to run 10 tests to find one winner.  If you run 2 tests per month (average for many organizations), you might get two or three wins in a year.
3. Meek-testing might satisfy statisticians but does not excite check-writers (the execs that fund the program).  The idea of changing a single element at a time might satisfy the purists, but it will not drive your business.

How do you treat qualitative & quantitative data to minimize bias?

Both are really important.  In my experience, many organizations over-index on anything that their competitors do and discount anything that comes from companies outside their industry vertical.  In fact, you should listen to aggregate data.  If a test concept works for 10 other companies, you should run that experiment.  Obviously, it’s better if some of those ten are in your segment, but I’ll take a test that has worked for 10 companies across industries over a test that has worked for a single company that’s my direct competitor.  Really important context here: within the realm of B2B SaaS.

What is the most annoying optimization myth you wish would go away?

That you always need the same kind of statistical rigor that clinical drug trials use – I’m talking about the “well, it’s not 95% stat sig!” game.   It’s a weird kind of loss aversion that really comes down to, “what if we are wrong?”

If you are running a single test, then the cost of being wrong when you choose a winner is very high. 

But if you are going to run a series of tests on that page, your cost of being wrong goes way down.  If you choose wrong, the next test will defeat your previous winner.

It’s like choosing what outfit to wear.  If you have to choose a single outfit to wear every day for the next year, the cost of choosing the wrong jeans today is very high.  What if fashion trends change?  What if I gain weight?  What if no one thinks acid-wash retro is cool anymore?  Wow, it’s overwhelming.

In reality, you’re going to choose a different outfit tomorrow.  So rock those acid-wash jeans today, and tomorrow throw on some dark jeans.  Or baggy jeans.  Or no jeans.  Wear shorts. 

Our thanks go out to Sahil for taking part in this interview! To our lovely readers, we hope you found the insights useful and encourage you to apply them in your own optimization efforts.

Don’t forget to check back twice a month for more enlightening interviews! And if you haven’t already, check out our past interviews with CRO pros Gursimran Gujral, Haley Carpenter, Rishi Rawat, Sina Fak, Eden Bidani, Jakub Linowski, Shiva Manjunath, Deborah O’Malley, Andra Baragan, Rich Page, Ruben de Boer, Abi Hough, Alex Birkett, John Ostrowski, Ryan Levander, Ryan Thomas, Bhavik Patel, Siobhan Solberg, Tim Mehta, Rommil Santiago, Steph Le Prevost, Nils Koppelmann, Danielle Schwolow, Kevin Szpak, Marianne Stjernvall, Christoph Böcker, Max Bradley, Samuel Hess, Riccardo Vandra, Lukas Petrauskas, Gabriela Florea, Sean Clanchy, Ryan Webb, Tracy Laranjo, Lucia van den Brink, LeAnn Reyes, Lucrezia Platé, Daniel Jones, May Chin, Kyle Hearnshaw, Gerda Vogt-Thomas, and our latest with Melanie Kyrklund.