Blog/A/B Testing

What is Post-Segmentation in A/B Testing? And 5 Tips for Using Advanced Post-Segmentation to Get More Out of Your Experiment Reports

By Dionysia Kontotasiou

June 2, 2022 ·

Segmentation goes beyond the scope of what traditional A/B testing can do, allowing you to better understand your audience, by segmenting them into specific target groups.

Post-segmentation, in particular, allows you to define audience segments once a test is completed, leaving room for you to make useful discoveries about your audience during the testing process.

Convert provides some great tools for both pre- and post-segmentation, so you can optimize your prices, better understand your audience, receive a greater ROI, identify even the most subtle issues that may be causing problems for your site, and draw more informed conclusions overall.

Recently, I had a troubleshooting call with a Convert customer for a large multinational retail chain. I asked:

“How can you tell if a product’s promotion had a positive effect on future sales of the product or other similar products?”

They responded:

“Simply by running an A/B experience to all website visitors and comparing pre- and post-promotional sales.”

Although this answer may receive a high score from a number of organizations, it would be regarded as inefficient or incorrect by any analytically evolved organization.

This kind of simplified analysis works in a steady state, where there’s no impact from changes in external factors or factors that cannot be controlled.

However, this is not the case in most practical applications.

In today’s world, external factors can change overnight, and the cost of drawing incorrect conclusions can be high.

So, how do you determine the efficacy of a sales promotion?

A simple solution is to conduct A/B tests or experiences for targeted audiences.

The goal of A/B testing or split-testing is to increase your conversion rate, or the number of actions your customers take, such as clicking a call-to-action button, text link, or navigation link.

To be effective, you must first consider your audience as a whole. This point of view begins with the following questions:

Who are my potential customers?
What brings them here?
What do they want from our brand?

Once you’ve answered these three critical questions, you can start testing audience segments or simply use segmentation.

Let’s Start with Some Definitions: What is Segmentation?

Visitors to your website are not all the same. They may come from different countries, speak different languages, use different browsers, access the site through different websites, use different devices, and have a variety of other subtle and significant differences.

So, what’s the point of A/B testing all visitors at once? Non-segmented A/B testing results can be illusory, leading to biased actions.

However, you can refine the audience you want to understand by segmenting it.

This is especially useful when you only want a subset of your visitors to participate in an A/B test so you can see how the changes affect that specific target. You might be interested in running an A/B test on your website with only visitors coming from a specific region, using a specific browser, or accessing your site during weekends.

With segmentation you will be able to answer these questions:

Is there a difference between how this experience works for new customers and recurring visitors?
Is there a variation that works particularly well for a certain type of traffic?
Is there a variation that performs particularly poorly on a certain browser or operating system? Is it possible that there is a flaw?
Is there a trend between users who convert and those who do not convert within each variation?

Pre- or Post-Segmentation?

Another consideration when segmenting is whether to design an experience with segmentation in mind or define the segments after the experience is activated/completed.

If the experience was designed with a specific segment in mind, the sample size and thus statistical significance are usually sufficient to allow a decision.

On the other hand, if you discover some patterns through slicing and dicing random segments, it is critical to ensure that a sufficient sample size is available. If that’s not the case, you must either extend the experience duration or re-test.

Segmenting experiences from the start does not always aid in the research/hypothesis process. The goal of an experience is to determine which segments respond to which variations, and this can be difficult to do if they are divided before testing begins. If you don’t know ahead of time that you’ll be segmenting your results, run follow-up tests for specific, high-performing segments until you have a sufficient sample size to use.

Post-test segmentation can aid in the identification of these patterns and signals, which can then be used for pre-test planning, as well as potentially testing specific areas with appropriate context and sample planning. You may then create and perform completely separate tests using device-specific hypotheses.

You can also generate reporting if you run the experiment with segments in mind from the start (as is frequently the case in follow-up experiments).

Why is it Important to Segment Your A/B Test Results?

Traditional A/B testing tools allow you to run experiences to the widest possible audience, testing an original or baseline version of your website against new variations.

Segmentation, on the other hand, allows you to zero in on each customer’s expectations, during the testing process.

Better Knowledge of Your Customers

Segmenting your visitors entails a thorough examination of who makes up that audience and what traits you notice among them. It also allows you to determine factors such as which groups are the most loyal to the company or which groups spend the most.

That information will allow you to better match your future marketing efforts, as well as provide a higher quality, more engaging customer experience for your visitors.

Better Price Optimization

It is necessary to conduct research in order to understand both your customers and your organization. And, while it isn’t always easy, determining the ideal price for your product or service isn’t impossible– especially if you have the necessary tools and a solid grasp of basic pricing ideas. While testing new prices, you can identify the groups of people willing to spend a little extra for a certain enhancement to your products or services, by using a segmentation strategy on your A/B experiences.

Greater Value Creation

You’ll notice a more significant ROI on your marketing campaigns and initiatives when you use a segmentation strategy of your visitors, rather than a general approach.

The average success rate of an untargeted experience that goes out to an entire group of visitors will be lower than that of a properly targeted appropriate and differentiated experience at various segments.

Essentially, optimizing segment by segment is more effective than optimizing for the total visitors.

Inconclusive Experiences

Let’s say you’ve been running a test or experience for a long time and the findings reveal a small lift, but no conclusion can be drawn. What happens now?

No need to worry. Although it may feel as though the test was a waste of time and resources, the information you gathered is actually still incredibly useful!

Segmenting the data that has been collected is the first step in dissecting it.

Examining your data based on distinct segments will provide more information to help generate iterations.

If you performed a test that changed how a product detail page was displayed, try segmenting the data by device. By doing this, you might just find a clear winner with statistically significant results.

Identification of Bugs and Broken Elements

Even if you’ve tested your code as fully as possible, you may have overlooked crucial responsiveness issues on certain devices (either due to the operating system version, internet connectivity in specific locations, or the device itself).

The current state of fragmentation in the smartphone business can be a major headache for developers. Customers often give negative feedback on designs and features they would otherwise love, due to load time, responsiveness, and crashes (rather than the usefulness of the original concept).

Let’s say you’re trying out a new banner for your homepage or landing page and you’re finding that it’s underperforming compared to the original version. This could be because the original version is more user-friendly. But, perhaps you still believe the new version to be better, based on your intuition and customer input. This would be a good moment to use segmented data, to see if there’s anything else going on.

The segmented data demonstrates that the unfavorable ratings are related to performance issues on a subset of Android mobile app versions. The new flow’s design does not need to be rejected; nevertheless, the code has to be updated for further testing.

Common Segments You Can Use in Your A/B Experiences

Here are a few examples of common segments to consider, when running experiments:

Geographical Segmentation

Geographic location is an important component in defining market positioning and product sales because people will have differing wants and demands and exhibit distinct features, based on where they live.

This is something to think about when running your A/B experiences. It’s why you should include geographic segmentation in your advertising and marketing campaigns.

Weather Segmentation

The weather may also be used to segment audiences, helping eCommerce companies with seasonal product catalogs sell relevant clothing and other goods.

Weather has a huge impact on customer behavior, and it provides a valuable opportunity for businesses to influence decision-making and enhance conversions.

Weather targeting gives websites a greater understanding of their customers’ needs, as well as a track record of meeting those demands.

Weather marketing allows you to be as creative as you want, and businesses of all sizes should use it as part of their overall marketing strategy.

Once you recognize how temperature variations affect your company, you can then adapt and optimize your company procedures.

Ad Targeting: Use weather data as targeting signals in your ad logic (e.g. show one ad variation if raining, show a different one if conditions are clear).
Ad Content: Use weather data as content within the ads (e.g. show current conditions / temperature values).
Both Ad Targeting & Content: Combine both weather targeting and content for a powerful user experience.

The weather segmentation should be combined with the customer’s geolocation to make sense. For example, a temperature of 20°C in northern Italy would prompt people to buy new T-shirts, whereas the same temperature in the south might prompt them to buy a light sweater or a long-sleeved shirt.

Behavioral Segmentation

Marketers need behavioral segmentation to understand and respond to on-site behaviors, such as an audience’s interest in your content or the frequency at which they interact with certain aspects of your website.

Behavioral segmentation is also important for identifying and engaging the users who generate the most income per user, which is usually a small percentage of a site’s total visits. This is particularly crucial in A/B testing, because segmented tests allow you to learn how high-value clients react to the experiences you’ve created across your website.

The segments listed below will help you understand how to handle a set of customers’ specific requirements or aspirations, find ways to improve their customer journeys and evaluate their potential worth to your company.

Recent purchases
New vs. returning visitors
New vs. repeat purchases
Logged-in vs. logged-out visitors
Plan types or loyalty program levels

Technographic Segmentation

Customers are largely classified, in technographic segmentation, based on the applications and devices they use for online purchases.

These can include:

Internet browsers that consumers use to access company websites
Email applications used to open marketing communications
Devices used to complete purchases
Operating systems
User-agents

Technographic segmentation should be used to better understand and optimize your customers’ experiences across all platforms and devices. This type of segmentation will allow you to answer questions like:

Are mobile app users engaged with my content for longer or shorter periods of time?

Even when using a responsive design, the mobile and desktop experiences of a website can often be significantly different. This can affect the appearance of your content, as well as the engagement it receives.

Do my mobile app visitors convert at the same rate as my desktop visitors (sales or leads)?

While your business objectives may stay the same, you might find that some goals are more easily converted by mobile visitors than others. This could have an influence on your cross-platform strategy and funnel design.

Do all browsers offer the same UX experience to my visitors?

If you see that conversions are unusually low in one browser– let’s say Safari– you can try to investigate further. You might discover that the checkout button on your shopping cart page doesn’t appear correctly when viewed on Safari and that a few easy tweaks could significantly boost conversions on that platform.

How Does Convert Handle Pre- and Post-Segmentation?

Segmentation in Convert can be done either before or after the experiences are activated:

Before can be done by defining audiences,
After can be done by segmenting the report stats depending on the metrics obtained during the experience.

Applying segments pre- or post-test, in the Convert Experiences tool, is as simple as using filters or including/omitting specific audiences in a report.

The power of segmentation can be especially useful in the development of A/B testing experiences.

In the most basic circumstances, it provides companies with a greater understanding of their customers’ various tastes and behaviors.

In more complex scenarios, companies can create entirely new online experiences in response to triggers like client login, region, or time of day. These modifications range in complexity, and can be as simple as varying calls-to-action, or as complicated as updating components of the site.

Pre-Segmentation with Audiences

A group of people that you want as part of your experience is referred to as an audience. Before the experience can be activated, the audience must be added so that the data collected by Convert only affects those individuals.

Only those visitors who meet those qualifications are eligible to enter. You can establish an audience that captures that information about your visitors and apply it to an experience. You can also mix different audiences using AND/OR conditions for more precise targeting parameters in a specific experience.

Visitors who do not meet your audience conditions are not included in the experience. They are excluded from your experience and just see the original version of your site. They are also not included in the experience’s count of unique visitors on the “Report” page.

You can create an audience for your experience by going to your Experience Summary and locating the Audience section. Clicking on the New Audience dropdown will give you the following options:

Presets that will allow you to choose from 111 pre-built audiences and will help speed up the configuration of your experience audiences.
Saved Audiences iswhere you’ll find audiences you created previously, which are not in use for the current experience.
New Audiences is where you can create your own audiences if none of the Presets fit your requirements.

When creating a new audience, you have to choose from three different types of audiences: Permanent Audience, Segmentation and Transient.

Permanent Audiences

This type of audience is checked only at visitor bucketing time (first page view) and not checked again.

If the audience conditions are met by the visitor, the experience will run.

If the audience conditions are not met by the visitor on subsequent visits, the experience will continue running for that visitor.

Segmentation

Segments are a group of visitors that qualify for certain audience conditions (such as location, the device used to access the site, time of day, and any other user behavior) and can be used at a later time for targeting.

Visitors in a specific segment are likely to behave or purchase in the same way.

You can create segments by defining audience conditions that allow Convert Experiences to determine which segment a visitor qualifies for and run the appropriate variation.

When this option is selected, content-related conditions become available in the conditions builder, allowing you to create segments based on URL (e.g. a segment of users that reach the pricing page).

Transient Audiences

With a Transient Audience, the experience will run only if a visitor meets the audience conditions. However, if a visitor does not meet the audience conditions, the visitor will no longer see the experience. Audience conditions are checked on every page view, rather than just the first one.

Once you’ve selected the audience type, you’ll be presented with the audience conditions, which can vary depending on your needs. We have shared in another article how you can target audiences based on details like visitor source, goals, and geolocation, to name a few.

Post-Segmentation with Basic Filters

If you did not add an audience to your experience upon activation, you still have the option to segment your data while the experience is running or once it’s completed.

Simply go to your experience report and select the Segment option from the top menu:

By default, Convert will show results for all visitors who entered your experience. Convert lets you then filter your results to see if certain groups of visitors behave differently from your visitors overall. This is called post-test segmentation.

In Convert, you can always filter results with default segments, which are captured automatically and included in all pricing plans:

Segment by browser
Segment by device type
Segment by user type
Segment by country

Let’s see an example of what it looks like on your report. Here is a report without post-test segmentation enabled:

Here is the same report with post-test segmentation enabled:

Post-Segmentation with Advanced Filters

You can also add advanced custom segments to your experiences. The below attributes are included only in specific pricing plans:

Segment by traffic source
Segment by continents
Segment by 10 custom segments (audiences)

For the last option, you must first create an audience with a “Segmentation” audience type (as previously mentioned).

Later in the report, you can use that audience to segment your data like this:

You can use these attributes for custom segmentation, which will give you a more granular view of visitors who are important to your business.

For example, imagine you run an experience that tests a pop-up promotional offer, generating an overall positive lift. However, when you segment for visitors on mobile devices, you see a statistically significant loss. It could be that the pop-up is disruptive on mobile devices, or that it’s difficult to close. Based on what you’ve learned, you might consider excluding mobile visitors when you execute the change or perform a similar experience in the future.

Segmenting your results is one of the most effective ways to acquire deep knowledge (beyond the ordinary visitor’s activity), as well as boost your experimenting efforts.

Here’s what you can do with these advanced segments:

Tip 1: Create Advanced Segments Based on Hubspot Lists

You can target and post-segment visitors that are members of Hubspot lists or Hubspot Campaigns, using Convert experiences on Hubspot web pages.

This allows you to leverage an organization’s investment in a given Hubspot list.

To achieve this, modify the Hubspot templates by assigning the Hubspot Template variable list to a Custom Convert Page Tag. This will allow you to use the Custom Page Tag to target experiences or create post-segmentation segments.

Start by modifying the Hubspot template and insert the following code at the beginning and inside of the HEAD tag:

<!--begin Convert Experiences code-->
<script type="text/javascript">
// The following line assign the Convert page tag variable with the Hubspot template variable
let _conv_custom_v1 = {request_contact.list_memberships};
let _conv_custom_v2 = {content.campaign_name};
</script>
<script type="text/javascript" src="//cdn-3.convertexperiments.com/js/123456789-123456789.js"></script>
<!-- end Convert Experiences code →

Create a segment within the Advanced Audience Editor. Select a Custom Page Tag condition and compare it to the Hubspot List or Hubspot Campaign that you want to target.

Tip 2: Create Advanced Segments Based on Goal Completions (“Triggered Goal”)

Did you know that you can use goal completions to set up your advanced custom segments?

This powerful feature provides new methods for filtering your visitors. Creating experiences based on the completion of the “Confirmed Orders” goal can boost conversions for recurring buyers (by creating a new experience on their future purchases).

Go to Audiences and drag and drop the “Triggered Goal” condition from the left list to the right side. Build the audience to identify users who have completed goals:

Later on, you can use this custom segment to filter your visitors on the experience report.

Tip 3: Create Advanced Segments Based on Visitor Cookies

You can segment your experience only for visitors who have been to a specific page of your site, by writing a cookie when they visit that page and setting up an audience to target the visitors with that value.

Tip 4: Create Advanced Segments Based on Your Data Layer

This is a very common example of web analytics tracking. You might have a Data Layer which feeds data to your analytics tool about each visitor, such as login status, user ID, geolocation, or metadata about the page (optimal resolution, image copyrights). Often, this data isn’t available in the presentational layer or markup.

You can use this Data Layer to store session visitor data and later present a specific experience to a certain visitor group.

To do this, you’ll need to create the experience that you want to show this audience group and configure the segment via the Advanced Audience Editor.

Enter the following JS code:

"medical" == dataLayer[0].industry;

Now, you can segment your experience results only for those visitors whose industry value of the Data Layer is “medical”.

Tip 5: Create Advanced Segments with 6Sense API

Predictive prospect analysis and lead scoring platform, 6sense, can identify and analyze the requirements of prospective customers and predict their behavior in various stages of the buyer journey.

With Convert Experiences, you can query the data aggregated by 6sense and segment based on this.

For example, if you want to segment your experience results only for visitors in the “Awareness” buying stage, you will need to create an audience called “Awareness Stage”.

You’ll do this by adding a JS Condition with the following code:

(function() {
var visitorData = JSON.parse(localStorage._6senseCompanyDetails);
if(typeof(visitorData)=="undefined") {
convert_recheck_experiment(); return false;}
else return (visitorData.scores[0].buying_stage == "Awareness");
})()

The Audience should be configured like this:

External Segmentation with GA Custom Segments

Apart from built-in pre- and post-segmentation, Convert also offers a way to send data to Google Analytics and segment your visitors there.

Google Analytics allows you to segment your website visitors based on their shared characteristics and demands. The Google search engine collects a wide range of user details by default, including screen sizes, Internet browsers, referring sites, and page types visited. These attributes can then be used to categorize your visitors.

To get started with segmentation in Google Analytics, go to the “Customisation” tab and select the custom report you created to send your Convert data to. You’ll now be able to see the percentage of users and sessions who fit the segment filters.

When you click “New Segment,” you’ll be able to design your own segment with the filters you choose. Google Analytics now allows you to have up to 1,000 segments per account, but only 100 segments per view.

The image below displays the various possibilities for configuring the filters for each segment.

After you’ve finished creating a new segment, click “Preview” to see how it will alter the existing report.

Conclusion

Segmentation takes A/B testing to a whole new level, allowing for greater conversion rate optimization (CRO) across your audience. In a world of haphazard A/B testing, segmenting your testing efforts adds a layer of accuracy and completeness that is simply not otherwise achievable.

When testing new features, segmentation ensures that your experiences will have a better probability of declaring a winner.

You’ll have a good knowledge of where the visitor comes from, what their intentions are, and how to test their behavior.

Your A/B testing will become exponentially more beneficial, thanks to the wide range of secondary insights created by every experience, which can be used for follow-up tests, identifying pain points, and gaining a deeper understanding of how customers interact with your products.

Mobile reading? Scan this QR code and take this blog with you, wherever you go.

Originally published June 02, 2022 - Updated November 11, 2022

Written By

Dionysia Kontotasiou

Convert's Head of Integration and Privacy, helping customers with technical queries.

Edited By

Carmen Apostu

Head of Content at Convert