How to Create a Communication System for Your A/B Testing Team

We can run the best tests in the world and have the most accurate data, but unless it’s acted on it becomes useless.

Vital test results that could change the face of a company can sit ignored and lost in the noise or worse: debated against opinions, even when the proof is right in front of you.

It can be incredibly frustrating and the bane of many an A/B testing team. One thing is clear though. If we want to get past this, then we need to learn to communicate better:

• To streamline communication with stakeholders,
• To get actions taken from results,
• To collect new test ideas from across the team,
• And much more.

Fortunately, that’s exactly what we’re going to cover in today’s guide. Why communication is important for A/B testing, how to improve it, and how to structure the delivery of that information depending on who you’re talking to in the company. (Note: This guide is geared towards companies that run their testing internally. If you work with an agency, you may benefit from reading this comprehensive guide.)

So let’s dive in…

What Is a Culture of Experimentation & Why Does It Hinge on Communication?

As an A/B testing program ‘matures’, it moves away from simply testing for improvements and starts to evolve. The end goal ideally is to become a data-driven company. A business that makes its decisions based on data and information and away from opinion.

This can help the company to grow and make offers based on what their audience and market want, and it’s a key methodology for many market leaders.

A large part of that evolution is moving away from simply testing for more conversions and even thinking of experimentation in one department, and instead, moving towards a culture of experimentation. This is where the entire company embraces the mindset and practices of experimentation. (Where testing is democratized and used in every team and department.)

Not only does this help to get buy-in (because everyone is applying testing principles), but you’ll be surprised how many improvements you can find across your entire company when you adopt these new ways of thinking. Lift on a sales page is great but imagine if your return department noticed a new way to lower churn, simply by A/B testing their communication processes?

The reality though is tough. Wanting this culture and building it isn’t as simple as everyone agreeing to the idea.

Before we explore some common mistakes hindering the development of a culture of experimentation, let’s consider some practical steps to foster this mindset within your organization. Speero has a comprehensive post on this topic:

Common Mistakes That Cause Companies To Fail At Building An Experimentation Culture

Testing programs struggle when communication fails. In fact, all projects can fail when communication breaks down.

In a study by the Project Management Institute, they found that 56% of failed projects stemmed from poor communication.

While 86% of Executives and employees in a study by Pumble stated that they felt poor communication was the cause of major failures. Interestingly, in the same study, they found that teams with good communication saw a 25% rise in efficiency. (They also found that 80% of completed projects usually had a communication plan in place).

As you can see, learning to communicate better is vital. But when we look at communication problems in CRO programs, we usually find the following major issues…

Issue #1: Information being stuck in silos and not being shared across all teams that can use this to help their departments

In a study by Speero, companies that have been experimenting for longer periods of time share their findings across departments and hierarchies (67%) while newcomers don’t do this often or at all in some cases (94%).

Issue #2: Put the band-aid of having one person in charge of driving the mindset shift to a culture of experimentation

Don’t have one person shoulder the change. Have them in charge of the communication, but don’t expect them to procure resources for the program or be the key driver of its success. This can come only as a result of org-wide awareness.

That being said, the right single person CAN make a huge impact, but it’s never usually the tester but instead an executive…

Issue #3: Executive stakeholders only being as committed as raising budget and not creating top-down examples for others to follow, causing a lack of change and adoption across the company

If you want to build a culture then it needs to come from the top down.

Issue #4: Reporting only on revenue, not on insights

Being data-driven means understanding all areas of your business and its impact. Changing to an experimentation culture starts with HOW you present data. It can’t always be from an ROI-only focus. Take the opportunity to set the narrative of taking risks & learning from day 1.

Issue #5: Not communicating effectively

Or it can be as simple as struggling to get and deliver the information needed. Either it’s hard to access or people are not consuming it.

The fact of the matter is this: Until you can set up clear lines of communication, then you will struggle to scale your testing program and build the testing culture you want.

So let’s show you how to improve.

Check out this 4-minute clip from Convert’s A/B testing course to learn more about developing a strong communication system, creating goals, and structuring communication.

How to Structure Communication in an Experimentation Program?

To understand this best, we’re going to recommend the RACI assignment matrix. This is a responsibility assignment chart that will help you to plan out each of the key objectives, milestones, tasks, and responsibilities in your project, for each of the people or ‘stakeholders’ involved.

Identify The Stakeholders in Your Experimentation Program

The RACI matrix’s goal is to help you create a flow of communication, while also ensuring that tasks are being completed.

You can use its acronym to help you plan out who your stakeholders are and where everyone fits in your project:

• Responsible. Who is responsible for completing the tasks or making decisions? This is usually a group of people.
• Accountable. Who will sign off on tasks and make sure milestones are completed? Who will be assigning tasks to those responsible for completing them? This is usually the project leader and is a singular role so that they can give the final decision.
• Consulted. Who needs to be consulted on the project’s tasks? Who needs to give input on a task to help it be completed?
• Informed. Who needs to be informed on the project’s status? Who needs updates on progress but doesn’t need day-to-day input?

Sometimes those people can fill multiple roles and may need to attend different meetings or communication channels, but it’s best to be clear on who does what and where to include them. This way you can decide on how to communicate and transfer information between those positions.

When it comes to an A/B testing program, your usual list of stakeholders includes:

• C-suite overview (Informed)
• Team leader/Project manager (Accountable)
• Testers (Consulted, Responsible)
• Developers (Responsible)
• Designers (Responsible)
• QA researchers (Consulted, Responsible)

Now obviously, not every testing team is the same, so here’s the best way to identify your stakeholders and create your own RACI matrix.

• Start by mapping out each of the tasks required to complete this project. To simplify you can just add major milestones at first. Have these as a descending column on the left-hand side.
• Then, go ahead and identify each of the stakeholders in this project. Have them as a row across the top of the matrix.
• Then, start to fill out each of the cells with either a Responsible, Accountable, Consulted, or Informed. You don’t need to fill out their exact tasks just yet, this is just a guideline for now.
• Make sure that every task has at least one stakeholder defined as responsible for it!
• At the same time, make sure that only ONE stakeholder is responsible for each task. This way you won’t get conflicting or missed completion of tasks.
• Once you have this rough outline, you can use this to chat with your stakeholders and fill out the specific tasks for each milestone, and confirm that everyone agrees with this assessment.

Note:

Not every cell needs to be filled out. The key thing is to make sure someone is accountable, someone is responsible, only those who need to be consulted are, and people are not being informed on everything they don’t need to be.

Decide On Your Experimentation Program Communication Tools In Advance

The two main decisions are deciding which tools to use and how your team will operate. Will your communication be in person, synchronous, or asynchronous?

With the rise in remote work and international teams, forming async habits early on can benefit even onsite teams. (We’ll cover some of these as we go, but Zapier has some fantastic guides to async team management here.)

Then you need to decide on your tools and HOW you will use them.

Making sure you all use the same tools in advance and plan how to integrate them will help to stop missed messages or poor communication issues.

In fact, not having clarity on this is a major cause of why communication plans fail. In a study by Citrix and the Economic Times, they found that employees spend almost 25% of their time searching for information to do their job when no clear communication or information infrastructure is in place.

With that being said, here’s a list of my top 5 tools for testing teams:

• Slack for day-to-day communication
• Asana for project management
• Zoom, Skype, or Meet for calls
• Email for booking in calendar meetings, sending messages, and interacting with other teams who may not be part of your Slack or Asana projects.
• A testing database/learning repository to keep track of what you have tried so far, the A/B testing results, what worked and what didn’t, learnings, and insights. Not only will it create a backup of all your work, but it can help onboard new testers AND it’s a fantastic way to save tests and use them for reporting to C-suite, teams, and even company-wide (more on this below).

The 4 Different Communication Types For These Stakeholders

So now that you’ve identified who your stakeholders are and where they fit in your RACI chart, it’s time to decide on just HOW you’re going to communicate with them, what you need to cover, and how often.

I’ve put together a quick RACI chart below based on my recommendations, but you can alter this to suit your needs.

There are 4 major communication goals and they each align with the RACI model:

• Vision & Mission (Informed)
• Roadmap (Accountable)
• Process (Consult, Responsible)
• Results, Learnings & Wins (Informed)

1. Vision & Mission

This type of communication aims to get everyone on board with the end goal. Usually only done once at the start of the project, but feel free to recap each quarter and realign effort and action.

In theory, this could be limited to the executive team, the project lead and the testing team but it’s my belief that if you want to build a culture of experimentation, you should set your vision for that project and get company-wide buy-in by including them in your experimentation project vision.

In a study by TradePress, they found that 85% of employees claim to be most motivated when updated about company information, but 74% felt like they were missing out on this information.

So share your vision for experimentation with the whole organization. Not only does this help to create that culture and get people aligned, but you can also find other teams that can help with your goal that you might not think of.

Example

I recently purchased base-level car insurance from AA.

I didn’t think of the cover too much, I just wanted to get a new car covered asap. 24 hours later I decided I wanted to upgrade to their most comprehensive package, which worked out at almost 55% more per month, so a big difference in ROI.

The issue is that it was quite hard to start paying them more money. Let’s look at the friction points and where a member of the team outside of testing might help us (especially if they have an experimentation mindset).

Friction Point #1. There was no one-click upgrade (although I could get a 2nd quote).

Friction Point #2. Online chat told me that the only way to upgrade was to call directly.

Friction Point #3. There was no option for this during the call. I was sent to the generic waitlist for 25 minutes.

If AA’s goal was more members on their comprehensive plans, a webmaster could easily spot the one-click issues. (I have to imagine this happens often as there was no clear data on what each level covered and so I would expect a lot of people may get the base plan asap so they can drive the vehicle immediately before looking in more detail later.)

A cool idea could be that if someone fills out another quote for a vehicle that’s already under their cover at a different level, then this could also be a trigger to contact them directly perhaps?

The call center seemed to be dealing with all calls,including cancellations, claims, and upgrades. It means they are widely trained operators, but there’s no fast path to upgrade and spend more money! (No press X option to chat about upgrading.)

A tester might not think to look at this because when was the last time you ran QA on your phone lines and live chat, right? But someone in that particular role could pick up on common issues, language, and complaints and feed that back to the testing team — which is why company-wide culture is so important and why I think you should include them in your Vision and Mission.

2. Roadmap Communication

This is your plan of attack. It should set the end goal, the major milestones on that path, and then the tasks associated with those milestones.

Executive teams will want to know of major milestones hit and monthly and quarterly progress.

Team leads will want the same, but also weekly reports. This way they can make sure they are on target to hit tasks and communicate with team members accordingly.

Optimization teams need to be aware of the major milestones and their OKRs to hit them. They should be providing weekly reports, then meet up for monthly + quarterly huddles with the team lead to make sure everything is on track.

3. A/B Testing Process Communication

The aim here is to make sure that everything is on track to hit the roadmap milestones.

Here, you can break down the experiments that are running and how they tie in, request resources and allocate budget, etc. If the roadmap was the recipe list, then this is the ‘how to’ of how to make the cake.

Tracking this ensures projects get completed on time, helps focus effort, removes duplicate tasks or missed task issues, and keeps C-level buy-in if presented at a later date. Again, this can be covered in monthly and quarterly calls.

4. A/B Testing Results, Learnings & Wins

This is where you see and share the impact you’re making. Depending on the stakeholder, they should all see this to some extent. The project lead and testing team are the ones putting this information together, who then share it with the C-Suite and company-wide.

There are 3 main areas where you might share this:

• A newsletter,
• Your learning repository,
• And if you’re really trying to build a culture of experimentation, a learning Slack channel.

So let’s look at how you might use these.

Your Wins and What Ifs Newsletter

The goal of this internal newsletter is to get buy-in and to build an experimentation culture.

It should feature current tests and their results, along with insights and learnings. This helps keep everyone involved and see the power of experimentation in action. Not only that, but specific test results may also help add further insight that affects multiple roles.

You can also get further buy-in by announcing the upcoming tests for the next month and running a ‘guess the test’ quiz.

This method can help you gamify and build a testing culture where people can have fun trying to guess which version will win and why. It helps all stakeholders be involved but can also remove opinions about tests when they see the data behind the results, helping to cultivate a data-first approach vs an ‘it’s always been done that way’.

It’s a simple idea but it works great.

Your Learning Repository

Your repository is more a database of all of your tests won and failed, your insights from each test, what you tried, and images of each test.

In reality, its main purpose is to help not re-run test ideas by accident, but to also help train up new experimenters.

That being said, having this accessible company-wide can also help get more of your company interested and involved, especially if you link to the test results held in the repository from your newsletter!

Take it a step further and add some fun and flair to how you present this information:

You don’t need fancy thumbnails, but you could simply hint at the test result from last month, guess the test in the newsletter and then send them to the repository for all the information.

Those who want the answer can find the result, and those who get hooked on experimentation can look further at what you have been testing!

Socialize and Incentivize Your Experimentation

Finally, you can get even more engagement by sharing your wins in an all-hands Slack channel, dedicated to wins and learnings.

That’s exactly what Specsavers does:

Specsavers not only have weekly and monthly huddles, but they have a ‘learnings’ channel for anyone involved in optimization to share what they have found out. This helps create dialogue while fast-tracking new ideas.

Want to take it a step further still?

Use that same channel to incentivize your communication and testing. You’ll be surprised at the innovation and ideas that come through when your team can get direct personal benefits from improving on new test ideas!

This can often lead to more sharing of ideas and insights and have a knock-on effect on other teams and members.

How Often Should You Communicate With Your Stakeholders?

As a rule of thumb, frequency of communication depends on the stakeholder and their roles/goals in the different communication methods and also which communication goal you’re trying to fulfill.

Not everyone needs to be communicated with all the time, so let’s break down each stakeholder group from an order of most frequent communication to least and what you should cover.

How to Get Each Type of Stakeholder Communication Right?

The key thing is to look at each stakeholder through the lens of their RACI designation.

Experimentation Team + Lead (Accountable, Responsible, Consulted)

• Initial vision and roadmap meeting and process plan.
• Daily communication available if needed. (Slack is great for this.)
• Weekly progress of what they have been working on. Not to micromanage but to get an idea of what’s working and the process towards roadmap goals. Asana works great for this. Can also check in with tests and projects that ended that week and any milestones hit.
• Monthly analysis of tests and learnings to be discussed, along with new test planning/iterative tests. Monthly roadmap meeting to make sure all teams are on track.
• Quarterly planning and roadmap analysis. Use this to decide to stick to the current process or edit and adjust.

Product Team (Consulted)

• Initial vision and roadmap meeting and process plan (especially if trying to build a company-wide culture of experimentation).
• Weekly progress reports if working on projects together.
• Otherwise monthly and quarterly analysis and learnings reports to be shared across the teams so that they can take action on findings also.

C-Suite (Informed)

• Initial vision and roadmap meeting and process plan.
• Monthly and quarterly roadmap and process progress reports, along with tests and learnings. Be sure to deliver information based on their goals (more on this later.)

Rest of Company (Informed)

• Initial vision and roadmap meeting and goals. They don’t need all the details but the end goal and major milestones will help them understand the plan. “We aim to increase ROI by improving our sales process and also reducing churn in Q2, before looking at repeat sales in Q3, etc.”
• Monthly newsletter reports on tests, learnings, and wins to help keep them engaged and build culture. Cover tests run, findings and insights, what worked and what failed. As a bonus, you can discuss new tests coming up and quiz company-wide on what they think will do best.

One key thing to understand is that you should set out and decide on these communication cadences and frequency in advance before starting your project so that each stakeholder knows their responsibilities and when to expect or deliver information.

How To Communicate A/B Test Results To The C-Suite?

Because communicating with C-suite is a common issue that testing programs have, and because having their buy-in and support is an important part of keeping your testing program running smoothly, l wanted to just recap the best practices here.

We wrote about it in detail in another guide here, but here’s a quick overview:

• Be aware of the cadence and frequency that you have agreed upon.
• Be sure to focus the communication on the specific goal. Is it a roadmap meeting, a process check-in or is it an update on learnings?
• Get them on board with the realities of the project from day one.
• Try to get them involved by sharing your learnings and ‘guess the test’ campaigns. Remember that top-down adoption works best for experimentation cultures.
• Present the information as simply as you can. What you learned beforehand and how you came to the test idea. What you tested. Show the control, the variations, and the winner. How it worked out on a micro and macro level. Include the results.
• Make them look good. Don’t just focus on sharing revenue alone, but do present results with what they care about first. Then share further insights and tie that back to how it will help them hit their goals.
• Don’t place blame on opinions. If the tests failed, figure out why and how it helps you and them. Maybe you saved a small fortune by not pushing out a design change live? The key is that you don’t want them to feel threatened or accused. Experimentation is about learning from failure to find what works.
• If the test won, give the ROI and MRR effects, along with the actual focused metric. It could be your test wasn’t on a page that directly affects revenue, but try to give insight into models of the specific lift and how it affects the business on a whole. (More leads, lower churn, more time on page, which leads to a higher ROI.)
• You may get pushback on results, especially from people whose ego or role is directly tied to a test. (Perhaps they are an expert on X but for your audience, their advice performs poorly). Acknowledge the results may be wrong, but frame it as “Yes it might be wrong but what if it’s correct? How could that affect us?”. That way you move the importance off them and onto the situation.

Conclusion

So there you have it. Our in-depth guide to improving communication in A/B testing teams, so that you can build a culture of experimentation.

You will hit some learning curves but stick at it. The more you communicate, the better your chances of pushing through it and seeing the benefits!