Tips to Set Up Winning Split URL Tests (and Some Common Mistakes to Avoid)

Split URL Tests offer a variety of benefits not included in A/B testing, to optimize the user experience of your website and improve your performance metrics.

In this article, we will go through how to set up Split URL Tests, using Convert Experiences, as well as help you avoid some common mistakes.

Split URL Testing provides so many benefits, including the ability to make significant modifications to each website version that you test.

So, if you’re looking for a testing solution that will help take your site to the next level, you’ve come to the right place!

What is Split URL Testing?

The term “Split URL Testing” is just another name for a redirect experience.

It’s called Split URL Testing because it involves comparing an Original (which has its own URL) to another Variation on a page with its own URL.

Unlike A/B tests, where the Original and Variation are hosted on the same URL and traffic is routed to one of them, Split URL Tests have both Original and Variation on their distinct URLs.

With Split URL Testing, people enter the experiment when they arrive at the Original page URL (and are assigned one of the Split URL Variations). Next, they either stay on the current page (if the Original was chosen) or are immediately routed to the Variation URL they received.

Convert’s Split URL Experiences save time, allowing users to test their pages without having to edit in the Convert Visual Editor.

The two most common examples of a Convert Split URL Experience are:

1. Different landing pages with different URLs:

• Original – www.convert.com/landingpage1
• Variation – www.convert.com.com/landingpage2

2. Subdomain or a separate domain to test a redesigned page:

• Original – www.convert.com
• Variation – www.beta-convert.com

Split URL Testing Visual Flow

Here is an example of how Convert serves URLs to visitors bucketed in a Split URL Experience:

Benefits of Split URL Testing

A/B and Multivariate Testing are great for quickly evaluating minor changes to UI elements. Most A/B testing software offers A/B and Multivariate Testing capabilities with a user-friendly interface, allowing testing variations to be developed with just a few clicks.

However, when it comes to making significant changes to your website, the Visual Editor provided by these testing solutions falls short. A new web page almost always requires the collaboration of the design, UX, and development teams, making it a much larger undertaking.

Split URL Testing provides the flexibility to make these changes without restriction, while still getting the benefits of A/B tests. Convert’s Split URL Experiences are helpful in a number of ways, allowing you to:

• Try out a completely new design, while keeping the old one in mind. Users are able to compare the reports from two variations to see which features function best in each.
• Run experiences with changes that aren’t UI-related. Trying to improve page load speeds by switching to a different database? This is just an example of the backend or development changes that affect websites. Split URL Experiences ensure that the modifications that aren’t visible have no negative impact on user experience.
• Evaluate the functionality of multiple web pages as a unit. Workflows have a significant impact on conversions, and testing new paths before implementing them is an excellent method to see if any issues have been neglected.
• Improve user experience, with faster connection and better browser performance. While A/B tests can suffer from the flicker effect (which occurs when a Variant loads slowly and clearly in front of the user), Split URL Experiences are capable of loading many sites at once without an issue. (Slow page loading tells users that they are participating in a test, thus skewing the test results.)
• Test different page designs or content without jeopardizing your conversion rate. By randomly splitting website traffic between the Original version and Variations, you can be sure that any statistically significant variance in performance is related to the experience, rather than factors such as traffic source or time of day.

How to Set up a Split URL Test in Convert Experiences

Create a Hypothesis

Once you’ve determined why you want to test a specific area of your sales funnel, it’s time to establish a hypothesis.

With your hypothesis, you’re attempting to solve a problem by speculating about what might perform better than the existing version.

Perhaps, you believe that including a promotion banner will boost conversion rates, or maybe you believe that presenting shipping fees to all of your visitors will increase engagement.

If you’re having trouble coming up with viable solutions, have a look at what your competitors are doing. Ask yourself:

• What are they doing that you aren’t?
• What stands out the most?
• Why do they have a higher conversion rate?

Consult your team and review the evidence from previous testing, before making a judgment.

Then, move on to the next phase!

Determine the Necessary Sample Size

Avoid conducting tests that produce statistically insignificant findings, as much as possible. This will improve your chances of obtaining useful data.

Before you begin, figure out the sample size you’ll need. You can do the math using this calculator.

Input the numbers based on the results you anticipate from your test, and see how the statistical significance appears. If you’re at least at 90%, you’re in pretty good shape.

Of course, this isn’t a foolproof solution, but it will increase your chances of success.

Create the Variations outside of Convert

Once you’ve calculated the required sample size based on your informed guess, it’s time to make your Variations!

Since Split URL Experiences do not offer a Visual Editor or Code Editor, you’ll need to design your variations outside of Convert, making them publicly accessible so that Convert can split the traffic between them.

Create a New Split URL Test

You can create a new experience in your “Experiment Dashboard” under “New Experience”, in the top right. There are 6 experience types to choose from. Select “Split URL”, and add your Experience Name and the URL of your Original site:

Add Variation URLs

You’ll see that the Original URL is filled in on the next screen. This is where you will add the Variation URLs, as well.

You can name each Variation and add more than one Variation using the (plus) icon.

The traffic will be split evenly between your Original and all other Variations.

Need Regular Expressions? (This is Optional.)

Let’s say you use Google Adwords or any other system that passes variables to your site, and you want to keep that. You can turn on variables with the option “Transfer Original URL Variables to the Variation URL”.

What this does is create a regular expression. Simply explained, it adds the following code after the Original URL:

/{0,1}(([\?&]{1}[^=]{1,}=[^=]*[&]{0,1})*)$

and it adds $1 to the Variation.

Essentially, the regular expression takes everything that follows a question mark on the URL and pastes it into the Variation where there is a $1 sign.

Verify Targeted Pages

When you click “Continue”, you will be directed to the Experience Summary.

The Site Area is where you will configure the criteria that trigger your experience.

The most basic configuration triggers the experiment based on a URL, such as “https://www.convert.com”.

This setting is automatically configured when you first create your Split URL Experience and will be configured to the URL used to create it.

Make sure to verify the pages where the Split URL Test will be running and adjust the Site Area accordingly.

Verify Targeted Audience

Next, you’ll need to verify the Audience and make sure it’s the group of visitors you want to target. If not, you’ll need to specify this by going into its configuration.

Verify Goals

You can make lots of different changes in “Experience Summary”, but for now, we will just focus on verifying our goals are set up properly.

Here are some of the common goals used in Split URL Tests:

• Conversion rate
• Clicks on buttons
• Clicks on links or CTA
• Newsletter sign-ups
• Adding to cart
• Revenue per visitor
• Engagement
• Page visits
• Time spent on page
• Bounce rate

Add the Domains to the Project Configuration

If your new Variation URL is different from the Original, you’ll need to add it to the Project Configuration. Make sure to add all of the various domains that include your Variation URLs.

To do this, click on “Configuration” in the top menu, then click the “Active Websites” section. Lastly, add the domain URL and save it.

Add Convert Script on All Domains

Finally, make sure to install the main Convert tracking code in the <head> section of your pages, before any other loaded resource.

QA your Test

Before launching your test to the public, double-check that everything is working as it should.

Check to make sure you’re being randomly directed to either the Original or the Variation, by visiting the Original URL a few times with an incognito browser window.

Try converting on both pages a few times, to ensure that the software is accurately tracking and reporting goals.

If everything is working correctly, you can now launch the Split URL Experience for your website visitors!

Analyze Results

When reviewing your results, ask yourself the following:

• How did your test go?
• Was your hypothesis correct?
• Are the results statistically significant?

If all went well, you should be able to identify a clear winner!

Now, you can either choose to stop testing and set the winner as your new Original, or you can keep testing and iterating.

If you don’t see a clear winner or if your results aren’t statistically significant, try not to get discouraged. This actually happens more than you’d think!

Identify any errors you might have made the first time and run a new test.

Hybrid Test: Split URL Test + Deploy

Let’s say your Split URL Test has been created, configured and tested, and everything is going according to plan. Say your traffic is being evenly distributed, your query parameters are being transferred, your audiences and specific goals are all set, and now you want to try out new styles for your experience pages.

Since Split URL Tests lack the capability to customize and style pages through the Visual Editor, one option you might consider is setting up a “hybrid” experiment (Split URL + Deploy Experience).

How to Set up Your “Hybrid” Experiment

Start by creating your Split URL Experience as directed in the previous section.

Get the Experience and Variation IDs

Once you have finished configuring the Variations of the experience, take note of the Variation IDs from the Experience summary page. Copy the link from the preview of the Variation:

The link copied will look something like this:

https://url.com/?convert_action=convert_vpreview&convert_v=100368636&convert_e=10036216

●	v=100368636 -This is the Variation ID: 100368636
●	e=10036216 - This is the Experience ID: 10036216

Set up a Deploy for Your Variation Page

Now it is time to create a deployment for your Variation page. Use the Visual Editor to apply styles and customize the page.

Set up a Custom Audience for Your Deploy

If you only want visitors from the Split URL Experience and the specific Variation to see your deployment, you must include a Custom Audience:

JS Condition 1: (replace with the Split URL Original Page URL)

document.referrer == "https://urlFromtheOriginalPage.com"

JS Condition 2: (replace with the Experience and Variation ID previously noted)

convert.historicalData.experiments[splitURLExpID].variation_id == "variationID"

This is it! You have successfully applied styles to a Split URL Experience!

Set up a Multipage Split URL Test

Now, let’s say you want to do a Split URL Test for multiple pages on your site (such as product pages). Reference this article on multipage tests and the chart below:

Run Multiple Split URL Tests Simultaneously

There may come a time when you need to conduct two or more Split URL Tests on the same website. Here are your options:

1. Run the tests simultaneously, without having to worry about them interfering with one another.
2. Carry out the tests at the same time but with different audiences
3. Perform the tests in order (complete test 1 before moving on to test 2).

Although Option 3 is the safest, it will severely limit your abilities with Split URL Experiences.

It is totally possible to run a few simultaneous Split URL Experiences on the same page or set of pages, but be aware that bucketing in one experiment can impact the data of another experiment that is running simultaneously.

Below are the two key techniques to help you run simultaneous experiences using separate audiences.

General Collision Prevention

Before setting this up, you must first understand the order in which experiences are executed. Experience conditions get tested sequentially on a page, taking into account their experience ID.

To achieve this, you need to set it up so that the Split URL Experience with the lowest ID number only uses 50% of the traffic. This can be done in the “Stats Settings” section of the “Experience Summary”.

When completing your second Split URL Experience, go to “New Audience”, under “Visitor Data”. Then, go to where it says “Bucketed in Experience” and type in “No”. By doing this, you will ensure that only new visitors are tested, which will prevent the same visitors from being tested twice.

Exclude Visitors from Other Split URL Tests

If you have more tests running on your site, you can set the second Split URL Test to specifically prevent the tested visitors of the first test from being bucketed on the second. This is done instead of the general collision prevention previously mentioned.

Check out this article for more information!

Set up Split URL Tests for Shopify Themes

If your website was designed on Shopify, you may want to take advantage of Convert’s Split URL option. This way, you can test two or more different versions of Shopify themes (applied to your website) against one another.

Shopify now offers a wide range of themes, so it would be great to know how one Shopify theme on a website would perform against another.

For more details, here’s an article that explains the full process of Split URL Testing for various Shopify themes.

Below you can see the Split URL Experience report between the two different themes:

Use an A/B test as a Split URL Test (Yes, that’s Possible!)

It’s actually totally possible to turn your A/B Experience into a Split URL Experience!

Convert has a JS function that allows you to be redirected to a new page, while still storing stats for that specific Variation:

convert.redirect("URL_here");

For example, instead of using:

document.location.href="http://www.mysite.com/my_variation_page.html"

Use the following:

convert.redirect("http://www.mysite.com/my_variation_page.html");

You can even pause the Original from the reports section and have all your users directed to the Variation. In this case, 100% of traffic will see the Variation.

How to Determine if your Split URL Tests are Statistically Significant?

So far, you’ve learned how to set up numerous kinds of Split URL Tests, but how can you be sure your results are trustworthy?

Let’s assume you’re conducting a Split URL Experience and sending 200 visitors to each Variation of a landing page. The Original captures 50 conversions, whereas the Variation captures 40.

You may be tempted to abandon the Variation entirely, in favor of your Original, but keep in mind that the test is still not finished and the results are not yet statistically significant.

Your findings are almost just as likely to be random as they are to be a result of the effectiveness of one page over another.

Convert provides a calculator to help you figure out the statistical significance of a Split URL Experience. The only four data points required are Original visitors, Original conversions, Variant visitors, and Variant conversions. Simply enter the information into the calculator’s left-hand side, to receive your answer.

Now that you understand how to set up Split URL Experiences and verify their statistical significance, let’s go over a few common mistakes to avoid when running your tests.

Common Mistakes to Avoid when Running Split URL Tests

Uneven Traffic between the Original/Variation

Let’s say your website receives 15,000 visitors each week, and you begin running a Split URL Test with an Original and a Variation. In an ideal world, each of these different versions would receive 7500 visitors (half of 15,000).

However, it’s more likely that the numbers in each variety will be in the range of 7490 to 7550. (Minor variances are typical and due to pure chance.) On the other hand, it is quite concerning if, say, one of your Variations captured 6000 visitors and the other received roughly 9000.

That’s where the SRM test comes in. Instead of relying on intuition, it uses the Chi-square goodness of fit test. For example, it can determine whether 7300 or 7800 visits are “normal” in comparison to the other amount of visitors received.

In December of 2021, the Convert team introduced our own SRM method, which Convert users can enable by going into Project Configuration > More Settings.

This will allow them to see the SRM tags in the reports:

Although reasons for SRM issues can vary, here are some ways to help eliminate SRM tags from your tests:

Wrong Page Targeting

Let’s assume you want to use the Split URL Experience to target all pages of your website (and not just the homepage) while using the same query parameters in both the Original and the Variation.

• The Original URL is https://www.convert.com
• The Variation URL is https://www.convert.com?v1=true

This is not as simple as it seems and must be set up correctly in order to avoid SRM tags.

First, define your Site Area with “Page URL contains https://www.convert.com” and input “query string contains v1=true” into the exclude section.

While you define the Split URL Variations, you use a regex formula to catch all pages:

([a-z]{1,2}tps?):\/\/((?:(?!(?:\/|#|\?|&)).)+)(?:(\/(?:(?:(?:(?!(?:#|\?|&)).)+\/))?))?(?:((?:(?!(?:\.|$|\?|#)).)+))?(?:(\.(?:(?!(?:\?|$|#)).)+))?(?:(\?(?:(?!(?:$|#)).)+))?(?:(#.+))?

Script Not Installed on All Pages

It’s important to always double-check that the Convert script is installed appropriately on both the Original and Variations.

One common issue we discovered, after troubleshooting customer requests, was that the Convert script was missing from one of the Variations. This resulted in uneven traffic distribution and thus SRM tags.

Make sure to include the script on every page where the Split URL Test will take place.

Bot Traffic Targeting One Variation

Another common finding in Split URL Tests with uneven traffic distribution, is that one of the Variations will receive visits from unusual user agents (which we suspect to be bots).

For example, the Original variation might receive totally normal traffic, while Variation 1 receives hits from user agents like the following:

• Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.11 (KHTML like Gecko) Chrome/23.0.1271.97 Safari/537.11
• Mozilla/5.0 (Windows NT 5.1; rv:27.0) Gecko/20100101 Firefox/27.0

We suspect these to be bots since they are using much older versions of Chrome and Firefox browsers. (Chrome browser now is around 99, as opposed to 23, shown above.)

Because we were unable to find an official bot with this user agent header, we suspect that this is advanced targeting made to appear as though it’s a human.

In such cases, you can exclude specific bots from your Split URL Experiences by following these steps.

Redirect Loops

Let’s say you’ve created a Split URL Test on your website and you discover that your users are internally redirected in a loop.

When a visitor arrives on the test page, they are forwarded to the Variation URL according to the traffic allocation you provide. On the Variation URL, visitors can click any element/link to return to the Original page. But, because the test is set up on the Original URL, the visitor is forwarded to the Variation URL, and the process repeats indefinitely.

These infinite redirect loops must be prevented ahead of time, as they will affect your site performance and experience stats.

Let’s say you want to target just the homepage (https://www.convert.com) with the Split URL and pass all query parameters the visitors might have.

• The Original URL is https://www.convert.com
• The Variation URL is https://www.convert.com?v1=true

To avoid any redirect, you’ll need to set the Site Area to “Page URL matches exactly https://www.convert.com” and input “query string contains v1=true” in the exclude section.

This ensures that the conditions of the experiment will still match if you end up at “https://www.convert.com ?v1=true”.

Double Traffic Recorded in Original in Your Analytics Tool

When running a Split URL Test, all of your traffic will go to the Original URL, which the Convert script will then redirect to a Variation URL (if the Variation is to be presented to that specific visitor).

In some instances, your analytics software (like Google Analytics) may show that your Original URL is receiving all of your traffic, even though the Variation URLs appear to be receiving the right portion of the traffic.

This can make it look as though your page views have increased, but in reality, the Original is displaying the sum of all the other percentages, plus its own.

Here’s how you can fix this:

Order of Tracking Codes

Make sure the Convert experiments tracking code is placed on the page BEFORE your analytics software tracking code. This will solve the issue most of the time.

Wrap Analytics Tracking Code

To prevent traffic from being recorded on the Original URL, wrap your analytics software tracking code whenever the Variation is presented. Here’s an example:

if(!convert || !convert.isRedirect) {
  // your analytics tracking code here

SEO Considerations of Split URL Tests

When doing Split URL Testing, duplicate content is something you may want to consider from an SEO standpoint.

Duplicate content is defined as blocks of content that are the same or extremely similar across several web pages, with only slight variations in graphics, design, or language. Duplicate material negatively impacts the search engine user experience, but luckily there are ways to account for it when needed.

Consider how a search engine perceives duplicate content. Search engines index pages to keep search results relevant, and will avoid showing pages with comparable information, as part of the user experience. Their algorithms will essentially group all pages with comparable information, displaying the Original or best content (based on their findings), while filtering out the rest.

In the case of fraudulent SEO methods, search engines must also deal with content repetition. Google has been known to remove websites from search results if the algorithm detects that duplicate content is being used to manipulate search results. Google rarely, if ever, imposes punishments, but it’s best not to leave it to chance– just in case!

Let’s have a look at some of the suggested solutions.

Do Not Use Googlebot User-Agent in Your Audiences

You won’t receive a penalty, as long as you aren’t basing your traffic allocation on a search engine versus human distinction (using Googlebot user-agent in the Audiences feature). Google doesn’t care which version of your website its bot sees. What matters is that it gets the same user experience as a random visitor.

Use rel=”canonical”

If a Split URL Test has multiple URLs, place the “rel=canonical” link attribute on all of your alternate links pointing to your Original page. If bots are indexing your website, this will direct them to your Original page. Experiments involving redirects should be fine as long as they don’t redirect to unexpected or unrelated content.

Use 302s for redirect

Google advises against utilizing a permanent 301 redirect, favoring a temporary 302 redirect. This informs search engines that the redirect is only temporary (meaning it will only be active for the duration of the experiment). It also tells them to preserve the Original URL in their index, rather than replacing it with the redirect’s location (the test page).

Only Run Split URL Tests for as Long as Necessary

If there is enough traffic to justify it, you can run your Split URL Test and then turn it off if it fails to reach a conclusion within your expected timeframe (or immediately after a conclusion is reached).

Summary

Split URL Tests are an excellent way to evaluate conversion and engagement rates across two different versions of a landing page. With Convert Experiences, you can set up a Split URL Experience in a multitude of ways, depending on your needs.

The results of these tests will allow you to optimize your designs and layouts for your target audiences and will be critical to your success as you develop your CRO knowledge and programs.

Happy testing!