Get Your Experimentation Team Up to Speed: Tips From 3 Experimentation Pros

Do you ever wonder why companies like Netflix, Google, Amazon, and Booking.com make a killing from experimentation but when you tried it at your company, it fell flat like an anvil in water?

Or maybe it didn’t turn out that bad. Instead, you often find yourself asking, “Where are all the double-digit lifts?’’ Especially when you’ve paid 6 figures for your testing tool. Sheesh!

A big tool should mean big results, right? Not really. Experimentation is more about your team than the A/B testing tool.

#teamovertools

The tool can only be as good as the brain using it. Not saying your people are bad, instead you can give them more to make the most of your investment in experimentation.

So, here’s how you can consistently give your team the boost they need to revolutionize your experimentation program — with tips from experts who have walked the talk.

Team Over Tools — The Zeitgeist of the New Experimentation Era

Once upon a time, companies would invest 100,000 USD in an experimentation platform and then hand it over to the marketing intern.

What do you think came off of this practice?

The sentiment that

1. A/B testing is all about fiddling around with site elements — like buttons and forms to (somehow) push more people down an abysmally selfish “funnel”.
2. Commoditized CRO is it! Experimentation who?
3. Optimization is too rigorous to get right, and we should all make do with the watered-down version of gut-driven decision-making—now credible—with the stamp of “statistical significance” on it.

In short, something akin to a $250 USD monthly retainer SEO where no one sees results, and the good name of search engine optimization is dragged through the proverbial “XYZ is dead” and “buzzword” mud.

Thankfully though, it is dawning on brands that getting a tool isn’t the end of the journey. It is merely the beginning.

One practitioner can’t get it right:

Heck, even several practitioners left to their own devices can struggle. Especially if they don’t have the qualities and skills we mention in a later section.

In fact, the ideal structure of an experimentation team is a layered and loaded subject. Manuel Da Costa has written at length about it:

He went further in the second part of the post:

Summarizing the excellent work of smart people:

• Optimization/Experimentation — whatever you want to call this division, the unit that supports and powers data-driven strategy needs C-suite buy-in. It needs to have the fanatical support of someone up top.
• And the “work” of experimentation is a lot more than testing. It involves coordination, communication, breaking down silos, and evangelizing testing.
• Of course, you do not have to come to the party with a 10-person experimentation team. But you have to understand that it is your team that will define A/B testing success. NOT the tool.

If you’re just starting out, Alex Birkett suggests a smooth way to glide into the groove:

When you’re starting your experimentation program, work with an agency or consultant. I know you think you can do it in-house, and you probably can. But there are technical, cultural, and process issues that are often best solved by an external expert.

I’d recommend having someone like your head of growth, or someone with a blend of technical know-how and strategic knowledge, manage the agency.

As you scale, you’ll want to eventually aim towards having a “center of excellence” that teaches, enables, and supports the rest of the organization in running their own experiments. Otherwise, the experimentation experts become bottlenecks by running everything through their own team. This center of excellence will often build or manage the experimentation tech stack, lend resources and time to helping teams get up and running, host office hours and training sessions, and answer questions and troubleshoot.

Alex Birkett, Omniscient Digital

Cross-functional squads are hard to beat no matter the size of the program or business. The tradeoff or red flag is if the team has responsibility but no authority, then there’s red tape regardless and it doesn’t work. But a group of ‘federated’ squads focused on journeys or growth levers (not channels) is my favorite setup. E.g., a squad focused around the goal tree around acquisition, vs one focused on monetization.

Ben Labay, Speero

All in all, each CRO program faces some key challenges:

And none of them will be solved by focusing on what tool you buy.

How to Train Your Experimentation Team?

Training your experimentation team starts right from hiring. While you can’t train someone who isn’t an employee, you have to make sure you’re getting someone with the right attitude.

But that’s just one step. This process stretches to the onboarding stage and beyond, where it should be a culture in your organization to treasure constant improvement. Let’s get deeper into experimentation team training.

Hire the Right Practitioners, FIRST

Pretty much like how you train a dragon. 😉

Start with the right temperament!

Here’s Jeremy Epperson’s take on the 6 traits of a high-functioning practitioner:

A dragon can burn you. So can a recalcitrant CRO practitioner.

❌ They may not follow established processes, derailing communication.

❌ They may not be diligent about documenting (new) processes, experiment results, and learning. Insights are the name of the game. If you’re not learning, why are you even testing? Some of the most valuable hypotheses are ones that are worthy of iterations.

❌ They may not bring critical thinking to data analysis. They may not know how to do that. Or they may simply prefer to let their brain come up with patterns and connections based on preconceived notions since going against the grain of subjective storytelling is tough!

We asked 21 experts, and this is a breakdown of the skills (and mindsets) they value in other practitioners:

• Grounded in experimentation and statistics basics: Knows and understands how to run experiments and apply basic statistics in interpreting results and extracting insights from every test.
• Leadership and communication skills: Excellent at working with stakeholders; believes in experimentation, evangelizes it, and sets the right expectations; level-headed with tests and results; and can tell the stories behind numbers.
• Research skills: Can handle quantitative and qualitative research like a champ, asks the right questions, and uses the right research methodology for the right problems.
• Data-driven: Has gut instincts and creative ideas but can always validate them with data.
• Empathetic: Puts in the effort to understand customers and users and step into their shoes frequently. Can also direct this effort to stakeholders, for example, to communicate data in the most effective, business-influencing way.
• Resilient: Doesn’t lose faith in experimentation if tests failed. That isn’t a failure in itself. By conducting the test correctly, they can validate that something doesn’t work.
• Process-oriented: Systems, processes, workflows — A good practitioner has to be obsessed with them. For long-term success, you need solid systems.
• Curious: Keeps an open mind and is eager to find out new things and learn. Asking questions continuously is the foundation of testing ideas and hypotheses.
• Can generate hypotheses and ideas: Beyond just asking questions, can they define the target audience, state the problem, suggest a solution, state an outcome, and define the goal of the experiment?
• Attentive to detail: You’ll find this on almost every resume, but very particular in CRO, a lot could slip through the cracks if they aren’t paying attention and lose opportunities to optimize.
• Developer skills: They don’t need to have professional-level skills. Some basic knowledge is important so they can test deeper than what visual editors allow. And there won’t be too many hold-ups waiting for the dev team to assist.
• Creative & Entrepreneurial: With so many best practices and “tried and tested” methods out there, you need someone who’s attracted to doing something new and trying radically different ideas with sound judgment.
• Interested in experimenting: If they’re not excited about experimenting (or at least care about it), not much will keep them going for long. And you need to be in this for the long ride.
• Critical thinker/analytic: Thinking critically about data… “The data says this. Why? Let’s find out” “It works that way, but why? Let’s find out”. That makes a natural tester.

Hire folks who tick these checkboxes.

(Note: All of these might not be in one person, but they should be in your team.)

Then empower your CRO team to challenge you. Because the very DNA of experimentation is all about challenging assumptions and taming rampaging HiPPos.

That level of influence in the right individuals helps propagate the culture of experimentation in your organization.

Charity and data-guided robust thinking both start at home. Jeremy Epperson understands this — how never questioning the status quo can lead to stagnation.

Take on the Mindset of Investing in Them

While you’re at it, invest in processes that will help said people be more:

• Focused
• Productive
• Free to ruminate, think, and make a mess within the confines of your well-defined experimentation framework.

This is so cardinal to experimentation that three out of four pillars of experimentation maturity have to do with people and processes:

• Process and accountability
• Culture
• Expertise

Sina Fak broke down people and processes into 10 characteristics that contribute most to a successful experimentation program. Check them out:

These 10 characteristics can very well be a guide to running an experimentation program that generates the results Netflix, Google, Microsoft and others are seeing (outside of using the right tool and tech stack, of course).

Pay close attention to 1 to 5 and you’ll see the same qualities in the pie charts above.

When you have that plus the mindset of helping them grow, the only way to go is up. After all, in the words of Shiv Narayan of How to SaaS, the building blocks of a stellar team boil down to

1. Finding, convincing, and hiring talented people to join the team
2. Aligning the team around an inspiring vision to collaborate on
3. Giving the team all the support and resources they need to succeed
4. Helping level up team members’ skills, mindset, and capabilities
5. Coaching team members through challenges and growth plateaus
6. Building the right environment and culture to help team members thrive
7. Removing people who put 1-6 at risk and cannot be coached out of them

Onboarding Is More Than a Checklist

How you onboard leaves a lasting impression. In fact, most team member churn starts at the onboarding.

While there are different flavors of onboarding—SOPs vs a more hands-on learning approach—all three experts we spoke to stressed the importance of the first 90 days. Getting granular here… here are the specific buckets you should focus on

• Summarizing key context
• Granting access to resources (SoPs, videos, trainings)
• Pulling fresh ideas from the team member
• Empowering to explore (Knowing the rest of the team has their back)

Here is Alex Birkett’s advice on leveraging new team members’ fresh perspectives:

Experimentation is different at every company, so the main thing to help them feel comfortable is to give them all the necessary context in the most digestible form. What are the key strategic initiatives? What have the challenges been, and what is expected of this individual to help work through those challenges? Who are the key team members this person should meet and get to know? What are the strict guardrails that you must operate within? And what [are] the 30-60-90 day plan and expectations of this individual? Should be done transparently and upfront.

Then, of course, give them access to everything they need — testing tools, qualitative research tools, analytics, Slack channels, internal email newsletters, meetings, etc. It’s always going to be drinking from the firehose at the beginning, but better to get them involved as early as possible to facilitate organizational understanding.

Finally, my underrated tip: get as many fresh ideas from this new hire as you can. They’re coming in with fresh eyes, so they can better diagnose problems with your process and program than you can. Get this feedback ASAP! They’ll eventually drink your company’s kool-aid. Pull in experiment ideas, but also ideas as to how to better operate and scale.

Alex Birkett

Getting new team members on the flywheel right away is Ben Labay’s suggestion:

SOPs are boring. The best is to start them on their own hero’s journey. Meaning to get them on the flywheel: give them a research task or a set of existing research to help ask questions on customer problems and opportunities. This helps uncover the first part of the journey, the problem/agitation.

Ok, now we have a good quality problem, so get them to come up with and rank out solutions.

Boom, now spec it out. Tell the story of the intervention, using all the fun test documentation/SOPs you might have.

Then the test and learn part, they see it running, analyze the results, etc. Then the decision and communication of it, have them own the final stage there too.

So in the end, they ‘own’ their own flywheel as part of the onboarding training. Learn by doing.

Ben Labay

What about the typical 90-day onboarding at the agency ConversionAdvocates?

The best person to ask is Jeremy Epperson. He has spent 70 hours of personal time creating, refining, and standardizing team training. Why?

Because people are key!

A clear and structured board sets the right expectations, has links to documentation, and provides access to tools. This is the backbone of our onboarding.

1. First day is listed out by hours. But we don’t stop there:
    
   1. Objectives are set for the first week, first 30 days, 60 days, and 90 days.
   2. All the meetings are listed out.
   3. Training material is parsed, week over week.
   4. Client work slowly ramps up with active support and feedback.
2. 1 hour per week is dedicated to productive sessions between the direct manager and the newly onboarded team member. This typically covers questions like:
   
   
   1. Where are you stuck?
   2. What do you need help with?
   3. What training do you need help with?
3. Each week we run trainings on various aspects of CRO — getting into the technical weeds. For example, if a test hypothesis proves to be a challenge during implementation, then how can the developer communicate this challenge to their direct manager to set the right expectations with clients?
4. SoPs are there to guide, inform, empower newly onboarded team members.

Jeremy Epperson

Extend Training Beyond Onboarding. Duh!

It’s common practice to send the new hire off to their first project after a rigorous onboarding process. You might even be nice and induct them into your L&D.

But with your top-performing experimentation team, you have the choice between doing the bare minimum (“Hey! We got you access to a CXL mini degree that you may or may not use”) and setting the gold standard for team training. This would look something like this:

Yes, Jeremy’s process is spectacular for onboarding and providing ongoing support and training. For your company—most companies, in this case—ongoing education will lie somewhere in the middle.

The middle for you could include:

• One-on-one meetings for 6 months after onboarding
• A personal development plan customized to their unique needs, skills, and proficiency level
• SOPs for experimentation tasks
• Direct access to anyone on the CRO team who can provide guidance and unblocking, and
• Reimbursements for tools, books, and training that can help them improve

We’ve spoken to 3 (other) experts—aside from Alex, Ben, and Jeremy—and compiled their takes on team education. Their insights form the foundation of the 6 steps you can take to extend your experimentation team training past the usual onboarding process.

1. Add “Continuous Learning” to Org Values

Sounds like a “nice to have”.

But both Ben and Jeremy touched on this core tenet in their answers. A culture of cruising can’t coexist with a culture of excellence.

When a team only does what needs to be done as it has always been done with zero curiosity and improvement, they’re not going to produce anything fantastic. And it’s usually a reflection of the way things are done in the organization. Culture matters more than you think!

At Speero, we have mechanisms like

• Book clubs — focusing on great reads on or adjacent to the CRO industry
• ‘Learning Fridays’ once-a-month workshop sessions where we all watch a course on CXL and take interactive notes on a miro board
• ‘Show and tell’ sessions every week by different people meant to teach the team on tests, SOPs, new frameworks, etc.
• A specific value regarding ‘elevate and educate’ and we reinforce this by having dedicated slack channels for each value, and whenever anyone sees evidence ANYWHERE in slack on each value, there’s a value emoji that’s added that auto-sends the message to the respective value channel.
• This gets the team to reinforce the value by organically recognizing when we’re living our values.We also have a ‘Living Our Values Everyday (L.O.V.E)’ award where we nominate a teammate monthly and they get to pick a charity that we give a nominal $50 donation to. It’s a token gesture, but this part of the mechanism ‘system’ that helps push our values, “educate” being one of them.

Ben Labay

And it wasn’t too different for Jeremy:

• We have an experimentation learning repository on ClickUp that our team members have access to.
• Every week, we set aside 2 hours for personal learning and development. This is built into the contract of our practitioners.
• Also, we’ve documented every aspect of the business and turned them into SoPs — with video and written documentation in a checklist format.
• We have different channels for different topics where people can ask questions and get answers. Then, the messages in these channels are grabbed and put into the repository on ClickUp.
• We invite a business coach to audit our processes and culture for gaps.
• Our core company values and 12 team heuristics are embedded into our daily ops.
• ‘See something, say something’—be it QA-ing issues/client frustrations—if anyone on the team senses something is off, they have to communicate it.

Jeremy Epperson, Chief Growth Officer, ConversionAdvocates

The pattern here is the togetherness in growth, the constant sharing, and reinforcing the culture of excellence frequently.

2. Show & Tell

Remember, one of the qualities of valuable CROs we listed earlier was “interested in experimentation”. Sometimes this is ignited and cultivated right there on the team.

How do you get someone interested in experimenting so they have their own inner fuel to power their training? Show and tell, like Alex does:

You can’t really force people to learn something if they aren’t interested in it, so my first job is to get people interested in experimentation. The easiest way to do that is to get them involved in experimentation. If you have the time and availability, literally helping someone run their own experiment is the best way to do this. Remember that hit of dopamine you got when you first launched a test? When you first got a winner? Give that same feeling to someone and they’ll be taking CXL Institute courses on their own in no time.

If you can’t help someone run their own test, post a call for experiment ideas. Manually ask people in Slack what ideas they have. Then put those in the roadmap and give all the credit to the person who gave the idea.

After people are interested, you can hook them into your education system, which could include: weekly newsletter readouts and weekly experiment meeting readouts, office hours sessions, courses and training materials, conferences and events, and documentation.

Alex Birkett 

Then, you can also promote the next one…

3. Teach & Learn

Remember that “Show & Tell” aspect of ongoing education?

Well, for those who are “showing”, an incentive should be in place. A nudge. Rishi Rawat calls it “Teach & Learn”:

We have a fairly simple program. It’s called learn and teach. As work is being done, we are constantly learning new things. Some of those insights need to be shared with the team. Once every 2 weeks we have a team meeting where each member shares key learnings from their own work lives.

Rishi Rawat, Frictionless Commerce (Taken from the Testing Mind Map Series)

Back in school, this was a trick for solidifying something you studied before an exam. If you can explain a concept to a colleague to their understanding, you will understand it even better and it’ll be hard to forget.

4. Put the Option of 3rd Party Learning on the Table

While most of the recommendations so far have relied on organization processes and learning repositories to deepen knowledge, for companies that may not possess exhaustive in-house resources (yet), giving practitioners access to something like Reforge is a safety net and a motivator they appreciate.

Our experimentation chapter is a group of 10 growth-minded individuals, lifelong students who love to be proven wrong by data.

Seventy years of optimization experience combined, all from multiple backgrounds.

Apart from what you can expect from chapter ceremonies and peer review, we have Reforge for our team.

We ask all to go through both the Experimentation Deep Dive and Growth Series.

Having a common language is an underestimated superpower for high-performing teams.

John Ostrowski, Principal Growth Experiments, iTech Media

5. Actively Manage the Learning Journey

So under-rated.

That’s why we celebrate Manuel Da Costa’s inclusion of an “Orchestrator” in his experimentation team structure. We understand that the Orchestrator isn’t supposed to “manage” individual growth trajectories, but it does acknowledge the fact that a practitioner needs to get organized and keep track of the mundane as they chase the next big idea.

Tools help!

We have SOPs around the fundamental areas of testing. Ultimately, we’ll probably have some sort of our own “testing bible” as the years go on. I currently use tools like Roam Research and Obsidian to connect thoughts together on articles I read in the space, then share them with the team after I’ve developed my own thoughts and arguments around a certain topic.

We use Airtable for all of our clients, but also internally in building out different swipe file types.

Learning has always been one of the most important things for me and in turn, my small team. Naturally, I want team members who have the right character qualities, like the hunger for knowledge and to be better. That is the same attitude one should have for testing—a “continuous improvement” type mindset.

Ryan Levander, Rednavel Consulting

6. Solicit Expert Help

Last but not least, CRO agencies live and breathe experimentation. Leverage their know-how.

This can be a collaboration where they jumpstart and then grease your flywheel, helping you structure and maintain a world-class experimentation program. Or it is a more case-by-case basis engagement where they focus on areas that need to be improved and team members who have to be upskilled. ASAP.

I’m an agency founder myself, and one thing I always tell clients is that they should use us for our tools, templates, and training wheels. We’re an open book. Ask us how we prioritize keywords, how we decide on the angle of a piece of content, how many 301 redirects it takes before you incur performance issues, why this CTA was chosen for this blog post… whatever.

Agencies do this stuff day in and day out, and they should have amazing answers for this stuff. If they don’t, push harder or get a different agency. Many agencies, like mine, do this stuff proactively. But even if that’s the case, you on the brand side should push for even more knowledge and frameworks. That’s the way to get the most value out of an agency: try to steal all their secrets.

Alex Birkett

Ben from Speero chimes in too.

We train a ton of teams. We run workshops, we have CXL, we do consulting, etc. We implement what we learn on ourselves into our clients’ organizations. It’s a feedback system.

Ben Labay

We’ve written a guide on how you can make the most of CRO & A/B testing agencies in your in-house program.

Conclusion

The output of your experimentation tool is limited by the input you provide. And who does the input? Your people. Investing in training and educating your team does more for your experimentation program and journey than any other aspect of the program.

They’ll be in a stronger position to do great things with your tool short of drawing water from a rock (choosing the right A/B testing tool is still important).

You can kickstart growth by shifting towards a culture of excellence and constant improvement. Follow the expert tips we’ve put together for you in this article.