Where Should You Invest Most Of Your Testing Budget?

So you just got a budget for your A/B testing program and you’re not sure what to invest in first.

Is it a tool? An agency? Or perhaps an entire testing team?

We recommend you invest in your team first for long-term benefits. Because without talented drivers behind the wheels, no tool will yield profitable results. And agencies will just cost more.

However, if you want to explore the full range of your options, you’ll have to weigh your strategy and goals and see which one aligns best with them.

Let’s explore…

Should You Hire An Agency Or Run CRO In-house?

Honestly, it depends.

Agencies are great because they have experience and you can offload almost the entire testing process to them.

If you have around 10,000+ visitors a month and simply want to start seeing lift as soon as possible for your current campaigns, then working with agencies is the best choice, if you have the budget for them.

How Much to Invest in an A/B Testing Team?

You could probably find people on Fiverr who claim to have CRO knowledge but I would steer clear.

The average costs are anywhere from $2k a month for a freelancer to 8k a month for a small agency or as high as 25k a month for a top-tier agency with a dedicated team.

We actually have an entire list of recommended agencies here.

Sure, agencies are not cheap but remember: CRO is a high-demand, highly skilled role with almost guaranteed ROI over time.

In fact, our data shows that agencies tend to get more winning tests on average while running the same volume of campaigns when compared to in-house teams. This is because they follow a ‘biggest wins first’ process and have more experience of where they can get the most impact for you.

That experience adds up and helps you get results fast.

Contrast that to beginner testers who might struggle because they are focusing on small actions and improvements.

That’s not to say agencies are always the best choice though. In-house results can really pull ahead once the team ‘matures’. Then, you’re more likely to see a higher ROI from in-house testing, simply because you’re acting on more data and becoming a data-driven company, that knows its audience and product.

You only have to look at companies like Booking.com who have a business-wide focus on CRO to see how much it can affect your growth.

Harvard Business Review wrote a case study on how Booking.com became the largest travel accommodation platform in the world, all from testing.

What if you don’t have 2k-25k to hire an agency? What if you don’t want just lift today, but also to become ‘data driven’, a company that takes all its actions based on information from its audience?

Then, you’re going to want to start running CRO in-house.

In-house testing is a great place to start if:

• You want more lift on current tests for less cost than an agency.
• Are an SMB who might not be at the point that qualifies for agencies criteria. (Traffic/budget etc.)
• If you want the ability to scale and grow over time and build a company-wide testing culture.

It can be a little confusing if you’re just starting out, so in this guide, I’m going to walk you through where you should spend your A/B testing budget.

What If You Wanted To Run Your Own CRO? Where Should You Invest Your A/B Testing Budget?

Bear in mind that this is all subjective, but this is the exact process I would follow if I had an A/B testing budget and wanted to start CRO for a company, and then use the ROI and insights gained from it to invest further.

 I’ve based this list on a few criteria:

• What is the main need or priority to perform tests?
• What do you need in place before you can take the next step?
• What habits, tools, and programs should you invest in to mature your testing team, based on the most ‘mature’ companies out there? (Those who are leading the game and running $500 Million per year, largely due to being data-driven).

If you don’t know what maturity means in terms of CRO, we’re talking about how intricate or invested your company is in CRO. From someone starting out and running basic tests to a company that makes every choice based on their results.

I’ll walk you through each part, why it’s important and when to invest in it.

So let’s break it all down…

1. Get Your A/B Testing Tools First

The most important thing you can ever invest in is your people. However, you can’t run a test without tools, right?

Ideally, you’re looking at 3 types of tools:

• Analytics on your site to track your current data. This can be Google Analytics or any other. (Note: You should also be tracking your current conversion rates and be able to see where sales came from before you start testing.)
• A Quantitative testing tool that can measure the exact actions people are taking, such as our Convert app. It will give you the numbers on who took what actions, and how it has improved or decreased, or simply made no change. Without this, you can’t test anything. With it, you can start making smart choices.
• A Qualitative tool so you can then try to understand that data, such as Hotjar or others. It will allow you to survey your audience, see where they move their mouse, see if they have issues clicking things, etc. The best-case scenario is to get a QA tool that integrates its information directly with your quantitative tool.

CRO is all about measuring results, figuring out issues, and then attempting to fix and improve them. You can’t do that if you can’t track and measure the data.

Now, you still need to pay for tools and they come in all different costs and range of added functions. We break down how to buy an A/B testing tool here, and the cost of A/B testing tools here.

(We even show you how to analyze if you’re paying too much for your current tool and how to switch to a new provider mid-contract.)

So, tools first, but remember, any tool is only as good as the person using it…

2. Hire And Invest In A Dedicated Tester

The majority of tests that you run will fail.

It’s not a bad thing because you’re proving what works and what doesn’t and getting incremental wins while dodging losses.

Having a dedicated employee who runs your tests gives you that opportunity to run more, learn more and grow faster. Heck, you can even use that knowledge to help improve products and experiences.

The thing is though, a lot of companies who are just getting into testing try to add it to a current employee’s role. Don’t take your marketer off projects and turn them into a 1% tester.

This is why most people think that CRO doesn’t work for them. They ran a few tests and focused on the wrong things, or even read the data wrong.

Don’t scrimp here. If you can get a tester in-house, it will be one of the best long-term investments that you can make and you can’t really go much further without them.

3. And Then Train Them!

#teamovertools

Once you have a tester, take the time to train them! The tests you can run and the insight you can get are all down to the experience of the tester.

Without it, they might:

• Set things up incorrectly,
• Struggle to find test ideas,
• Come up with an untestable hypothesis,
• Not run a test for long enough,
• Read the data wrong and miss a win,
• Read it wrong and implement losers,
• Test the wrong thing, or worse,
• Not test something that they should be…

Not everything works as we think it should. Best practices with website design can work but not always…

A beginner tester might not have tested this or not even thought to have tested it. They might be just running campaigns on specific pages and not checking this data. This is why it’s so important to train them up to learn ALL the ins and outs of testing.

So where can you train them?

There’s a fantastic program by CXL Institute, focused purely on CRO and A/B testing.

Sidenote: When you become a Convert customer, we give you access to the CXL training.

It’s incredibly in-depth and made by some of the smartest testers out there. I highly recommend it.

It will not only give your tester an understanding of all elements of testing, but it will help them prioritize those large impact areas to work on first, and the types of tests that will make a difference.

Once your tester understands all this, it will help them set up a solid A/B testing strategy.

4. Create A Learning Depository And A Testing Culture

This is important if you want to scale, become organized, create buy-in from managers, solve mistakes fast, and build a testing culture within your company.

We talked about this in another post, but allow me to recap quickly here. A learning repository is a database of all your past tests.

It shows:

• Each test you’ve ever run,
• The page you were testing,
• What your hypothesis was,
• The test and what you changed (including the control and the variation)
• The resultant change in lift, drop or if it stays the same,
• The change in average order value. (Either positive gained and if negative, showing the money saved by not pushing it live.)
• Your thoughts on those results. (Did it lead to new insights, etc.)

Something broken on your site?

Having a repository that includes past issues will help your team to find and fix issues much faster.

Want to train new people?

Having all learnings in one place allows your tester to learn from past tests.

Get better buy-in!

Presenting information like this makes it easy to take your tests to your boss and prove the value of CRO.

Build a testing culture

Speaking of buy-in, you can even use these tests to build a company culture of testing. Share the side-by-side shots and the hypothesis with your team and see if they can guess which one performed best. Share the results afterward.

This gets people excited about CRO. Especially if the majority of people believed that a test failed, only for them to see it won. This helps them to see that not everything is about opinions.

You could also help them improve in areas that they might never have thought of! (Imagine if you found out that certain words and angles connect better with your audience, and then your customer support, copywriting and ad teams all found out about this, and the impact it could have.)

But excitement is not sufficient to totally remove biases and create a long-lasting testing culture in your company.

It is not as effective as when you get C-level executives—the final decision-makers—involved in rolling out an experimentation culture.

You see, de-siloing testing doesn’t work out so well because it requires a much different skill set. But if you work closely with management to dish out solid reasons for testing, gradually they’ll adopt persistent testing and data-driven decision-making practices.

How can you present these tests and win over management?

There’s a fantastic site that you can model called GuessTheTest.

It’s designed to help testers get ideas, while also helping them improve their hypothesis skills. The site works by showing multiple A/B tests and the reader can guess which variant wins.

They regularly feature different A/B tests you can click on and then see the results of how many people guessed it right or wrong, but also the data behind the test, what was run, and how it converted with the winner.

It’s super helpful for testers as it also helps us challenge our own opinions.

This is the same thing you can do to build up that testing culture in your own company, and it costs you nothing. You don’t need to build a website like GuessTheTest has. Instead, you can simply share what you are already doing with the images and information you’re putting into your repository.

Send a company email out with the control and the variant, and ask people to vote on which they think won. Then send a follow-up email with the results of the test, the hypothesis, and any insights.

Ensure you engage your management team in this exercise. Or at least tell them the story, from the beginning to the outcome. Let them see how wrong opinions can be. You’ll be blown away at how it starts to help build that testing culture, and how much it helps teams in totally different areas.

So how to build a repository?

It can be as simple as a folder on your computer, or you can look at platforms like ‘Experiments’ by GrowthHackers. It’s a learning repository and project management tool in one.

5. Streamline Your Processes

One of the most valuable things you can do to:

• Improve your tests,
• Increase their frequency and quantity,
• Remove errors,
• And lower the cost per test…

… is to set up systems and processes to streamline your entire testing system.

From the hypothesis and organization of what you want to test, to pre-test set up, how to push it live, what to do during testing, and what to do after.

If you build systems and SOPs (standard operating procedures) for each element that you repeat, then your entire testing program will not only improve but will increase your chances of success.

Why?

Each test will run smoother with less chance for human error, while also being implemented faster. It’s a win-win situation.

The problem, of course, is that hardly any company does this. In fact, it comes up again and again in CXL’s state of the industry report.

But the agencies that run them all day?

Or the internal teams that scale their testing culture?

These are the ones who ALWAYS focus on building systems asap…

It’s easy enough to make these also.

Map out what you do for each test, then create documents to follow for each new one. You can simply tick the boxes to make sure that everything has been done.

For instance, here’s a straightforward experimentation testing process (based on Ben Labay’s Phase-Gate Testing Framework) you can start with…

Phase 1: Collect preliminary insights (to discover problems or opportunities)

Start with understanding your customers and your position in the market. To collect these insights, research your customers and market by running surveys and polls, stakeholder interviews, user tests, etc.

The major questions you’ll want to get answers to are:

• Are we meeting the needs of our customers/users?
• How does our target audience perceive us?
• What is our brand’s position in the market?
• What data have we already collected about these problems/opportunities?
• What would we need to change in our product, marketing, or service?

These will lead you to come up with a concise problem statement. You’ll be clear about what metrics to measure and how to achieve them. Also, you’ll be able to assess if the problem or opportunity in question is worth pursuing or can be put off till a later time.

The insights you draw from this initial assessment will help you prepare a research summary with your proposed hypothesis.

Use this to get buy-in from executive stakeholders because they’ll have to be on board with the test’s alignment with business goals.

This brings us to the next phase…

Phase 2: Prepare a test plan and prove feasibility

In this phase, you’re making a case for the feasibility of the test to garner support from the execs.

They’ll want to see how the value of the potential outcome justifies the resources required to make it happen.

These are the questions you want to answer in this stage:

• What resources are required to run the test?
• What key changes will need to be made?
• Does this have any impact on other projects/tests?
• What are the KPIs and how do they indicate success?
• What are the possible outcomes?

Summarize the answers to these questions in a document describing the bare-bones scope of the tests. Ensure it’s easy to understand, maybe on a spreadsheet, so you can use it to get approval from management.

Once you’ve gotten approval, you can move on to planning…

Phase 3: Prepare a more detailed plan of the project

Here’s where you get everyone involved on board with the project. Execs should agree with the balance between cost implications and expected outcomes. And cross-functional stakeholders should be ready to participate.

To achieve this, prepare a more detailed plan for the project that answers these questions:

• What data/info do we need to complete the test?
• Where and how are we running the test?
• What metrics indicate success and how are we measuring them?
• Have we prepared a decision tree outlining possible outcomes?

Tie these all together in a plan to get full alignment from everyone involved in the project. This will also be a valuable addition to your learning depository.

You can launch the test when all stakeholders are on board with the entire plan.

Phase 4: Launch and run the test

You’ll launch your test in this phase while performing checks to make sure everything is on track. Perform single or multiple QA and QC checks and weekly performance checks.

The clarity you want in this phase should answer:

• Is the test built and running according to the plan in prior phases?
• Are we collecting data and reporting the way we intended?
• Are we on track with the test project timeline?
• Have we resolved the impact this test may have on other projects/tests?
• Is everyone involved still on board with the progress?

If anything sticks out as an issue or there are any necessary changes to be made, have them documented in an incident log.

Report back to executive stakeholders and let them know how the test is running. Are things going according to plan? And are you accurately collecting data and preparing reports?

This will usher in the next phase…

Phase 5: Prepare a test report

In this final phase, you’ll want to prepare full documentation of the test detailing the analysis, results, insights, and decisions made as a result of the test. This will be part of your tests archive and learning depository.

What you’ll want to include in this test report are:

• Whether the test met the goals
• What immediate fixes were made because of the insights you got
• If you found out anything new outside the scope of the test, and
• The key decisions that were inspired by the test results

Besides the detailed documentation of the test, it’ll be beneficial to create a scorecard of the test that provides vital info at a glance.

Now, all you have to do is repeat the 5 phases for the next test.

6. Improve Your Data Quality

Once you start to get a lot more traffic and results, it might be a good idea to switch up your analytics platform for more detailed reporting.

You can upgrade to Google Analytics 360 once you’re at 10 million visitors a month (I know, right?… wow)

Or you can start looking at real-time, cross-channel tools like Adobe Analytics and Heap. (Both of which integrate directly with the Convert app.)

These tools can give you more accurate data, help you to understand user behavior, and also test and track things that standard GA just can’t.

Not only that, but they also plug into your CRM and other tools.

7. Add Additional Support To The Testing Team

So far you’ve probably just been using a WYSIWYG editor that’s part of your testing tool.

However, if you want to start running more intricate tests, with faster and cleaner code, it’s worth hiring a Javascript developer.

This way you can have the developer make the test using their own preferred tools and workflows, and then upload the code directly into your testing tool without any additional code bloat.

And a bonus?

Because the test is still implemented via the tool, this will allow you to use all the segmentation and measuring functions, and it can even be shut off automatically once a winner is found.

(One sticking point we see with people using developers is them bypassing tools or building their own testing platform. This can work, but if you need a developer to design the test, turn it off when finished, and implement the winner, it can become an issue.)

8. Start To Build Testing Into Other Teams

Now that you’re running more tests, tracking data, and experience, it’s a great time to start trying to implement testing across different teams.

No longer just sales but product or traffic etc. Any team where improvements can be made.

Remember: It’s not always about huge wins, but the aggregation of small ones. A 1% lift in 100’s of areas can lead to a huge difference in how your company operates.

You can either start to train someone on each team or bring people in and train up to manage testing in those areas. It’s usually best to start slow and use the core testing team as a role model to help inspire and provide support to the new team.

Just don’t try to build every team all at once:

The key is to help build one more team and make them successful then move onto the next. You want each team to be able to run and be efficient on their own if you want to build a testing culture!

9. Add In Customer Experience Measurement Tools

Speero (CXL’s agency) recently wrote a report on CRO maturity, based on feedback from 210 different companies all running CRO in 2020-2021.

Their goal was to get a more detailed view of the different experimentation programs out there, and how the maturity of the program affects the results.

A few things really stood out to me:

• The most successful companies are data-driven. They make all choices based on test data and not opinions.
• They share their results company-wide, to help build a testing culture.
• And they are just a few of the companies implementing Customer Experience Measurement.

There’s a reason that not everyone measures customer experience. It’s not what you would think of when you talk about CRO, in that we’re not improving conversion rate. We are, however, tracking, measuring, and improving all connected aspects that add to higher sales, repeat sales, and retention, using the same methodology as we might use for QA and Quant type testing, just in more detail. (All the things that might have a direct impact on conversion rate).

Customer Experience measurement often requires more specific tracking, but when you do this, you can start to equate a monetary value to improvements in the customer’s experience.

Not just that, but you can also track retention over time. (It’s far more profitable to make more sales to a current customer than to bring in a new one.)

So what can you use for this?

The process is actually quite large which is another reason it’s not always adopted.

Ideally, you’re creating interaction and feedback systems along the entire customer journey, even after they leave your site and have received their product. This requires a human element and a few added tools to survey etc.

Some agencies specialize in CX and so you can get started by using them (or stick with them).

Or you can start looking at tools like Qualtrics and others and build out your program internally.

CX is a new topic with a learning curve, but the benefits can be substantial, especially when you feed that back into your CRO testing.

10. Avoid the Trap of Linking Experimentation to Revenue

It is tempting to say “Because we ran this test and used the insights to make that decision, we will make X amount of money.” That conclusion disregards reality.

The world isn’t as simple as we’d like it to be. So many factors affect the outcome of things that it isn’t realistic to think a certain result, when rolled out, will repeat itself consistently.

Then why do some A/B testing tools, agencies, and CRO practitioners still tie A/B tests to revenue projections?

The answer is simple: It’s exciting and often attracts buy-in.

But this is an easy way to set people up for disappointment and make them lose faith in A/B testing. And it is why people are erroneously attracted to pumping money into tools that report revenue.

Here’s why spending $100,000 a year on such tools is a huge mistake:

There’s a great deal of uncertainty in the real world. 

There are many reasons why a test result won’t repeat itself exactly and you shouldn’t ignore them. Markets change, customer behavior changes, competition changes, budgets and prices fluctuate. The future will not necessarily be the same as the time you ran your test. 

When you extrapolate your result to predict revenue gains for the next year or so, you’ll be oversimplifying the ROI calculation and making wrong projections more likely than not.

The real purpose of testing should be to make decisions for moving forward.

Testing has more to do with goals and insights than revenue. Informing better decisions should be the focus of A/B tests. This positions decision-makers on a more confident path towards growth and profit than having attractive revenue numbers to look up to and risk being let down by.

But often it’s difficult to get people interested if a tangible revenue goal isn’t at the end of your experimentation rainbow.

What to do instead?

Pull the conversation around experimentation from projected revenue towards the cost of not experimenting. What would be the loss if no action is taken? 

Also, frame the success of testing projects as the level of confidence you get in making certain decisions regarding a specific objective. Not only will this shift the focus away from expected revenue gains, but it will help get stakeholders aligned with the right focus of an optimization program.

If you are still pressed for numbers, give a range. Since experiments only show a percentage certainty of obtaining an outcome, this is a smart way to appease those asking for a number. For instance, if the results showed a 4.55% lift, tell the stakeholders to expect between a 3.2 and 5.9% lift.

Conclusion

So there you have it. The exact way that I would go about spending and investing my A/B testing budget when just starting out and how I would scale and further invest it over time.

You can use this guide to help you figure out where you are on your testing maturity journey and implement any missing steps to help you grow further, or simply use it as a guideline to get started today.