How Can Experimentation Reduce Uncertainty During a Recession?

It’s tough to make predictions, especially about the future.

Yogi Berra

One of the most valuable aspects of experimentation is it can reduce uncertainty in decision making. But it can never eliminate it fully. There will always be uncertainty in business.

Even in “normal” times, there’s no clear path towards the optimal outcome. But in times colored by inflation, an economic downtown, war and pandemics? Things get even murkier.

While I can’t predict what will happen in the coming months and years, and I’ve never built a business during a recession, I’ve thought and read a lot about how to operate during a recession. I’ve thought about this from the perspective of an agency owner as well as an experimentation practitioner.

The good thing is we have the past to learn from and many operators have gone through economic winters before.

We can learn from their lessons as well as apply our own hypotheses to not only survive through a recession but thrive.

Economic Outlook Changes Consumer Behavior

“A nickel ain’t worth a dime anymore.” ― Yogi Berra

There’s a lot we don’t know, but what we do know is that inflation and recession fears change consumer behavior.

All things equal, inflation means a fall in purchasing power. The same dollar spent a few years ago can purchase less now.

With price inflation at the highest level in 40 years, consumers are already adopting new behaviors to curtail discretionary spending, including “eating even more at home, trading down to cheaper products and shopping at retailers that they perceive are doing better at managing prices.”

Certain categories suffer more than others, particularly those perceived as non-essential as well as categories with more malleability or with larger arrays of options.

Anecdotally, my rent continued to increase this year, and I’m going to keep paying it, but I’m cutting down on my DoorDash spending (and cooking more regularly) and cutting out extraneous high cost experiences like my membership at the local athlete recovery lab (I can get my sauna benefits by walking outside in Austin). 

I’m not a macro-economist (or any type of economist), but any event or loss of certainty can cause consumers to reflect on their current budget and lifestyle.

Just like at the start of the pandemic when many city dwellers left their urban environments for more spacious rural lodging, consumers are likely to pause and reflect about where they’re spending money. Price matters, especially if your company sells something non-essential.

Consumers are also likely to start comparison shopping and questioning their current subscriptions and consumption. According to Ipsos, “shoppers are very likely to pick and choose among different retailers and brands even more than they’ve done in the past.”

This is scary, but it’s also an opportunity.

As a brand, you can double down on what’s driving customer loyalty in the first place, making yourself indispensable.

You can also take advantage of the fact that customers are looking to compare and switch to new products, driving new acquisition and adoption.

I’ve written before about the Surround Sound Strategy, which can take advantage of this comparison shopping stage:

Not all companies can absorb cost increases or afford lower margins, however.

If you’re in a tight place economically, this means keeping lean, narrowing the focus, increasing the velocity, and raising your bar of excellence and expectations for results (I’ll cover all the following below).

All of that said, what consumers end up doing is highly speculative, but how you operate during uncertain times is relatively straightforward.

For the rest of the article, I’ll walk through principles for operating a company during a downturn as well as operating an experimentation program specifically.

5 Principles for Operating During a Downturn

There’s a ton of research looking at past recessions. These surveys try to distinguish those who thrived, merely survived, and those who capitulated to the market.

I’ve found the data to be largely unsurprising. Essentially, here’s the advice:

• Those best prepared for the recession before it happens are most poised to succeed during the downturn (obviously).
• It’s not those who cut fastest or deepest, but those who properly reallocate budgets and continue to invest that succeed (though there’s lots of nuance where you should invest)
• Consumer behavior changes. Adaptation is the key to survival in both evolution and economic uncertainty.

The counterintuitive learning, though, is that there are many silver linings to economic uncertainty, especially if you play your cards right.

Assuming you’ve operated well during boom times, or at least that you’re not in panic / survival mode, you’re in a good place with options.

 Now you need models by which you can operate during uncertainty. I’ll cover five ideas here:

1. Increase velocity
2. Raise standards
3. Narrow the focus
4. Get to default alive if not already
5. Take advantage of silver linings

My favorite model for operating, period, is in Frank Slootman’s book, “Amp It Up.”

In it, he describes three principles for driving hypergrowth:

• Increase velocity
• Raise standards
• Narrow the focus

These principles are effective regardless of economic outlook but are particularly important during uncertain times.

For what it’s worth, these are the principles by which I’ve run my experimentation programs as well as how we operate our content agency.

1. Increase velocity

Slootman’s first principle is “increase velocity.” According to him, most organizations run shockingly slowly, and you should amp up the pace to a degree that feels uncomfortable:

The pace has to be profound, palatable, breathtaking, order-of-magnitude type change. You want to go 20% faster? It’s barely discernible, and you will be back in your old mode before long.

In the world of software, we often sit around tables talking about what we need in the product, and when we could expect to see it. Development teams tend to come back with unacceptable time frames because they are doing things linearly, and are not thinking with enough urgency. But with pressure applied, somebody all of a sudden figures out how to do things differently, and get things dramatically sooner. Pressure changes things.

It’s like Peter Thiel’s thought exercise: “If you have a 10-year plan of how to get [somewhere], you should ask: Why can’t you do this in 6 months?”

Instead of bikeshedding small decisions on an experiment, ship it faster and iterate on the next one. Instead of assuming you can only publish two blog posts per month, ask, hypothetically, how you could double or quadruple your output? What would you have to sacrifice in order to do that?

In most cases, you’ll have to sacrifice less important initiatives, but you don’t want to sacrifice quality and standards.

Lesson for experimentation programs:

Increasing velocity could mean simply increasing the number of tests, but I mean this more broadly. How can you take what you wanted to accomplish in 6 months and get it done in 6 weeks?

It’s more or less a thought exercise.

For example, how can you shave time off of design and development approvals to get things up and running faster? Could you buy an experimentation platform instead of building one in order to better democratize running and analyzing experiments?

2. Raise standards

Slootman’s second principle is to raise standards of excellence.

Most people think they have high standards of excellence, but I can tell you from my various work experiences that only one company truly had quality expectations that made me stretch and grow (I won’t mention names).

According to Slootman:

Mediocrity is the silent killer. Organizations are not getting killed by their C players. Everybody knows who they are, and performance eventually is addressed. The people who kill organizations are your B players. It’s the scourge of the enterprise because there are many and they are generally accepted. Often, they are seen as not bad enough to fire, but not good enough to keep. They are the ultimate passengers.B players need to be pared: they either become A players, or they become C players and get flushed out. You can help by raising standards, by refusing mediocre outcomes.

He describes “drivers” versus “passengers” on the path to growth, and essentially, you want to identify who’s who and try to get as many drivers on board as possible.

Passengers aren’t just neutral; they weigh you down. It’s important to identify passengers quickly, and in economic downturns, part ways amicably.

Lesson for experimentation programs:

Take bigger swings and test meaningful differences in your marketing and product experiences. Moving the needle requires a sharp focus on qualitatively big ideas as well as weighting the biggest ideas based on impact, ease, and confidence.

This is best done from a leadership perspective. Leaders should stretch but not break the team. Push experimentation outside of your normal comfort zone. Stop simply copying competitors and running button color tests, and ask yourself, “what would a top 1% experimentation program look like?” Then hold yourself to those standards.

3. Narrow the focus

You might be asking yourself, “how can I maintain super high standards and ALSO increase velocity?”

Won’t that lead to burnout?

First off, increasing the pace and standards tends to energize, not demotivate people. People forget that burnout is also a function of apathy towards work. I’ve felt more burnout when the pace was slow and aimless than when it is intense and exciting. You want to be in the sweet spot:

The answer here is you need to narrow the focus.

Eliminate everything that doesn’t make the boat go faster. Knock it out of the park on the few initiatives you engage in, instead of trying to dilute your efforts across many channels, markets, and strategies.

As Slootman puts it:

When you narrow focus, you are increasing the resourcing on the remaining priority. It doesn’t have to time-slice and compete any more with a bunch of other stuff. And then things begin to move, stuff is getting done, and we move to the next thing. Many people and organizations are focused a mile wide and an inch deep. It can’t be a surprise when they progress at snail’s pace.

Lesson for experimentation programs:

Instead of “testing everything,” pick the targets most likely to drive results. Depending on your business, this is likely the highest visibility pages or experiences or those closest to the point of conversion.

It feels good to expand your efforts far and wide, but in all likelihood, your results will follow a power law. A few pages and a few tests will pay for the rest of your program. Narrow the focus of your program to the essentials, at least until you max out your current opportunities.

4. Get to default alive or default investable

How you approach a downturn depends on what your business health looks like going into the downturn.

It’s no surprise that Bain research found, “Greater discipline during boom times offered more flexibility during lean years.” Additionally, “getting it right during the lean years has a massive impact on companies’ growth rate after things improve.”

The common vernacular for startups is you’re either default alive or default dead. This was coined by Paul Graham in an essay written several years ago:

When I talk to a startup that’s been operating for more than 8 or 9 months, the first thing I want to know is almost always the same. Assuming their expenses remain constant and their revenue growth is what it has been over the last several months, do they make it to profitability on the money they have left? Or to put it more dramatically, by default do they live or die?

If you can become default alive and cash flow positive, you’re in a much more propitious position to take advantage of the silver linings of a downturn. This isn’t realistic for all startups, however. So David Sacks recommends instead focusing on getting to “default investable.” As he puts it:

Default alive’ is almost an impossible standard for early-stage startups since it means being cashflow positive.

On the other hand, ‘default investable’ means that your metrics are good enough to raise another round in the current environment.

He linked to the following image explaining great, good, and poor metrics for SaaS companies:

Maintaining a healthy cash balance and seeking profitability (or at least investability) allows you to “be greedy when others are fearful,” as Warren Buffett famously advised.

Or if you prefer a running metaphor, if you treat hills cautiously when running down and sprint when running up the hill, you avoid errors and conserve energy for when others are struggling. Ryan Holiday wrote about that:

As you race, it matters only how you behave on the hills. Downhill–did you resist brashness, exhibit more control and discipline here than on the straight-aways. Ignore your impulse and shorten your strides. Uphill–did you clench your jaw, look down once and then lift your gaze to stare directly into its eyes. Races are won here and not there–where it’s hardest to speed up, not where it’s easiest.

Lesson for experimentation programs:

Be a cash generator, not a cost center. Do whatever you can (cut costs, restructure the team, refactor your priorities) to make sure you’re not a line item to be cut by executive leadership.

5. Take advantage of the silver linings

If you’re poised for optionality, you can take advantage of the many silver linings of a recession:

• Cheaper and more available talent
• Cheaper CPMs, CPCs, and less competition on ad channels
• Competitors cutting back and restraining

This is really the secret: slow down when running down a hill (when others sprint out of control), and you’ll conserve energy to sprint up the hill when everyone else is winded.

Mark Ritson wrote his recession playbook and that was basically the synopsis of the winners:

If your category cuts half its ad spend and you maintain your ad spend, for example, you get a massive boost in your [share of voice] simply because everyone else is investing relatively less. And if multiple industries cut back their spending, as they usually do in a recession, the cost of media often drops ensuring your relatively big ad spend is now stretching to even more media value.

When others pull back their marketing budgets, you press onward. When others freeze hiring, you pick up solid talent.

Of course, these things depend on your business health and cash position to start with. But if you’re poised to invest now, it’s one of the greatest opportunities you have to pull ahead.

Lesson for experimentation programs:

Continue to invest in experimentation. When others are slowing down, that’s the best opportunity to put your head down and run faster. On the other side of the recession, you’ll be a few laps ahead of those who held back in fear.

Help Your Experimentation Program Yield Greater Returns During a Slowdown:

All of the above general operating principles apply to experimentation: increase the pace, up the standards, and narrow the focus.

But first and foremost – bear in mind that economic turmoil (or any turbulence) isn’t the time to cut back on experimentation. Uncertainty needs experimentation for risk mitigation.

You will likely contend with limited budgets. Experimentation gives you the flexibility to make informed bets on approaches, and roll out only those projects, features, changes that are likely to add revenue to your business.

Here are 7 concrete ways to adjust focus in your experimentation program, and optimize workflows for better results.

1. Lean out your program and costs

First things first: you have to audit your existing program and find wasteful areas.

Company-wide, this may unfortunately mean reducing headcount or limiting new hiring. If that’s what you’re working with, you’ll have constraints, but constraints can also breed innovation.

My first order of business would be to look at my experiment backlog and see how many areas of the website or product I’m currently focused on.

Then, ruthlessly prioritize. Shed the stuff that isn’t directly producing high impact results for the business. I’m speaking for the sake of the employee’s job safety as much as the general value of experimentation to the organization.

If what you’re working out isn’t visible and visibly producing ROI, it may be on the chopping block.

Ross Simmonds gave the following advice:

You need to think about the work you’ve been asked to do and consider how crucial it is to helping your team achieve their overarching goals. If you feel like it’s low value – start looking for and asking for additional responsibilities. If you’re in a smaller company – Go directly to the c-suite/founders and ask what you can take off their plate to help achieve [insert an organizational goal here]. If you can execute this well, it will give you their trust and make you irreplaceable.

This may require you putting down the passion project (like building a predictive targeting system that can automate personalization at scale and with perfect accuracy) and instead stretch your projects and skill sets to areas outside of your normal scope, but that your leadership cares about.

Generally, when in an economic pinch, companies face two options: fire good people or make more money.

If your company overhired in the first place, it’s important to admit this and see where you can cut headcount. This isn’t popular, but if it helps the company survive, it’s necessary. Startups in the early or scale-up stage struggle with this especially.

As Paul Graham puts it:

You should lean more toward firing people if the source of your trouble is overhiring. If you went out and hired 15 people before you even knew what you were building, you’ve created a broken company… So the solution may be to shrink and then figure out what direction to grow in. After all, you’re not doing those 15 people any favors if you fly the company into the ground with them aboard. They’ll all lose their jobs eventually, along with all the time they expended on this doomed company.

2. Lower the cost of experimentation

Expected value is a function of the cost of an action and the probability of its outcome. You want to increase the expected value of each experiment.

You can do that by increasing the probability of a win or the size of that win. Or you can lower the cost of running an experiment in the first place.

One way to do that is to outsource parts of the work to agencies or freelancers, particularly the design and development of experiments.

Another way is to automate the boring stuff. If you can build processes and frameworks for the parts of the experimentation process that take the longest (analysis, experiment design, permissions and review, etc.), then you can lower the resource cost of an experiment.

Investing in a great experimentation platform as well as educating other teams and individuals on how to use it also helps lower the cost of experiments.

However, don’t cut budget everywhere. Invest in areas that compound; strategic bets that will pay off in better economic conditions – brand, SEO, CRO, and product experience.

An HBR study actually found that “firms that cut costs faster and deeper than rivals don’t necessarily flourish. They have the lowest probability—21%—of pulling ahead of the competition when times get better.”

Many companies would benefit from looking at an affordable experimentation platform to lower the cost of experimentation. Unless you truly have crucial edge cases, building an experimentation platform is often an opportunity cost for your focus areas and your engineers’ time.

Additionally, you can leverage your vendor’s customer success team to help you with enablement, documentation, setup, and education. This helps you onboard more and more employees to begin running experiments, making it more affordable to run more tests.

3. Squeeze more juice out of your current efforts

Marketing and product leaders should look at what they’ve already created and see if there’s any way to get more value out of it.

In content, this looks like content optimization and historical updates. In marketing, generally, this is a good case for more investment in CRO.

If you’re running paid ads and can keep running paid ads through these times, then increasing the landing page conversion rate can allow you to continue to invest back into growth.

Unfortunately, these are exactly the efforts that tend to be cut during uncertain times. According to Ross Simmonds:

Most brands are going to make the mistake of pulling back on their SEO and CRO budget. This isn’t smart. It isn’t smart because if their competitors continue to invest in improving their content, building links and growing their SEO moat — the competitors are going to come out on top.

On a tactical level, CROs should shift the focus towards existing revenue generating assets, experiences, and pages. There’s certainly value in innovative experiments, but may be the time to focus on the checkout funnel, the demo form, the onboarding sequence, and the in-product PQL points.

4. Narrow the focus

“Optimize closest to the money” was one of the most important lessons I learned from Peep Laja.

There are always trade offs and opportunity costs. You can optimize literally anything. Doing so, however, dilutes your efforts (as resources are always a constraint, even at the biggest companies).

Instead of assisting every team at the company and optimizing and testing every part of your company, prioritize. Pick the places that truly matter and will move the needle.

While this will depend on your business, some of the biggest levers you have to pull are the experiences closest to the point of revenue:

• Pricing page
• Demo form
• Product onboarding
• PQLs

Pricing, in particular, is a great place to focus during shifting economic conditions and user behavior. Maybe you can’t change your pricing, but you can package things differently and learn how customers are responding to your pricing. Things like:

• Better terms for the customer (shortening contracts or offering discounts for longer deals)
• Usage or value based pricing packages
• Subscription pricing
• Unbundling so customers can pay separately for different elements
• Reducing the threshold required for incentives like discounts, free shipping, etc.
• Extending credit and payment plans to customers
• Offering introductory plans or tripwires

If possible, offering a freemium version of your product is a great way to build a user moat, deliver value, and have a ton of users learning your product once the economy starts revving up again.

While it’s important to optimize performance centers, contrary to what I would have thought, brand investment also tends to pay off during economic uncertainty. According to Mark Ritson:

Summarising a century of data, the reason brands should maintain their brand building budgets in a recession is not because of the recession itself, nor the behaviours of consumers. It’s because your competitors lose their nerve and are vulnerable because of it. If you can keep your head and your brand budget while those around you are reducing theirs, you will earn the post-recessionary benefits.

5. Customer research and identifying changing patterns

As mentioned above, consumer behavior patterns change during times of uncertainty. And despite your best predictions, you’re not clairvoyant. No one knew, for instance, that toilet paper would cause such a maelstrom during the pandemic.

There’s a reason experimentation teams are often wrapped up with “insights” teams. We learn stuff. Experiments should teach you stuff. The customer research you do to inform good experiments should teach you stuff.

Now’s the time to spread the knowledge. Build a knowledge base to store experiment and analytics learnings. Run weekly or monthly meetings or office hours. Get these insights into the hands of product and brand marketers, demand gen marketers, and product managers to inform their roadmaps.

6. Reframe “Losing Tests Are Learnings”

While I understand the sentiment, I’ve always hated the phrase, “there are no losing tests, only learnings.”

Learnings are great, but they don’t pay the bills. Experimentation programs should prioritize action and ROI.

This is narrowly true on a per experiment basis (though obviously they won’t all be winners), but especially true on a macro and strategic level.

Mark Ritson put it well in his recession playbook:

And let’s add failure to the newly unfashionable concepts we won’t entertain during the recession. I am tired of everyone in marketing pushing failure as an essential pre-cursor to success. It can be. But it does not have to be. And in recessionary times, it’s really rather important that it’s not. Because you aren’t getting the pre-recessionary opportunities to fail, learn, boast about it on the conference circuit and then rework things for another go. You fail and you lose. And the lesson is you have lost and it’s over.

Failure is largely a luxury of good times. Don’t let the fear of failure paralyze you, but stop celebrating merely “learnings” and go make some money for your company.

7. Nobody cares, work harder

At the end of the day, you control what you can control and you can’t control what you can’t.

After many conversations with my co-founders, we’ve come to the same conclusion on the myriad of recession fears: “so what? Work harder.”

If you’re smart, you’ve made your cuts, prepared for turbulence, and have some runways. What else can you do? Put your head down and get stuff done.

Conclusion

Experimentation should see increased investment during economic downturns for many reasons:

• It’s the house of customer insights, and since consumer behavior changes during economic shifts, keeping a pulse on this is crucial.
• Experimentation can drive further growth for existing assets, squeezing more ROI out of what you’ve already got.
• Experimentation is a swiss army knife that can drive further growth along many vectors, including product, organic, and paid channels.

However, it’s imperative you shift your mindset and become a revenue center, not a cost center. This includes amping it up (increase velocity, raise standards, narrow the focus). It may also include chopping off nice-to-haves in favor of the most impactful and critical activities.

At the end of the day, focus on what you can control and continue adding obvious and clear value. Do that, and we’ll get through this just fine.