Convert Experiences: Helping Organizations of All Sizes Run More Tests Across Diverse Channels

Our crew never rests—we’ve been cooking up tons of new features to help you leapfrog ahead with your experimentation programs. After navigating 2023’s industry shakeups (Google Optimize sunset, anyone?) and major growth, we didn’t rest on our laurels—in 2024 we’ve launched even more product updates to equip you with peak testing technology.

Convert Says: Experimentation Is Human Nature™

We say this because experimentation is a cycle that is ubiquitous in our daily lives.

We observe a set of events, suspect a causal link, hypothesize why this link exists, and then take action. This cycle of observation, hypothesis, and action is present in everything we do, from switching to a ‘better’ workout routine or tweaking a cooking recipe. It’s like climbing a mountain of wins.

All human progress can be traced back to curiosity and experimentation. We are curious about meaning and purpose. This compels us to understand why things happen the way they do.

That’s why we see the same in businesses run by humans—experimentation in marketing, sales, product development, customer experience, pricing, etc. (Not like I’ve seen any business run by aliens but I bet they would do the same).

Hence, any platform or tool that wants to support proper experimentation should:

Free up resources (funds) for people and processes

Your testing tool is just one part of your experimentation program. And it is not even the most important part.

As Steph Le Prevost points out below (and many other experts concur), people and processes are the bedrock of validation and causal relationship-seeking culture.

In our “How to Optimize Each Stage of Your Experimentation Journey” event, Jeremy Epperson also argued that tool choice should not be the first consideration. Instead, the focus should be on the skills, process, and team building of how we actually execute against experimentation.

Allow for testing across all growth channels and motions

Here’s a bit of context to grasp what this means:

Elena Verna emphasizes the importance of leveraging all growth motions and levers in your growth model. And experimentation plays a role in all three growth motions.

• Product-led growth: To identify which product features drive user engagement and retention
• Marketing-led growth: To find the most effective marketing strategies or campaigns that drive user acquisition and conversion
• Sales-led growth: To find the most effective sales tactics/strategies — such as what pricing model increases the average deal size or close rate

Experimentation is, after all, an amplifier — improving what’s already working, spotting ideas that aren’t worth rolling out, and unearthing innovation to revamp what’s effective but gradually regressing to the mean.

All of which are elements driven by a comprehensive experimentation approach that permeates every aspect of your business and product. This is also known as full stack experimentation.

If your experimentation tool is built with your business’s best interest at its core, it should be able to support testing across all these growth motions.

Shorten the time from observation to insight to knowledge

This may manifest in exploration tools like heatmaps that don’t just show “what’s happening”, but also connect the “why it’s happening” to actionable next steps (i.e., ideas to test).

It also involves giving you the power to unlock data from any touchpoint along the customer’s journey, even if it is a 3rd party platform.

Simplify test deployment

This can be achieved by offering hyper fast response times, so you and your team aren’t left twiddling your thumbs through inordinately long waits and then subjected to frustrating scripted fluff responses.

Instead, your tool should increase time to action/learning; or slash the time you take to launch a test and get back results — an important program efficiency metric.

It should stay one step ahead of privacy issues which add a layer of hesitation to running tests because of either skewed results, steep fines, or heavy criticism (like in the case of Facebook’s infamous 2012 experiment).

At Convert, we’ve worked extensively on this aspect, as detailed in our blog post on respecting visitor privacy during experimentation.

All the Recent Additions to Convert Experiences (And Why They Matter)

Now, let’s get into our new additions and why you should be excited about them. Starting with the most recent:

Environments

Do you want to run tests on multiple domains within one project? Environments are here for you.

Think of instances where you’ve wanted to run a test in different testing environments—staging, development, or production—within one project but couldn’t. Now you can.

Convert’s Environments give you the flexibility to launch experiments exactly where you want them regardless of domain, which is a quality assurance dream come true.

Here’s how it works: Each environment gets its own tracking code specific to its domain. Select the environment you need, copy the tracking code, and place it on the corresponding site. This way, experiment data is kept isolated in the environment in which it is running and computed without lag.

Convert’s Environments make it easy to ensure that you have the flexibility to run isolated tests before going live. You can set up multiple environments – depending on your plan.

QA Overlay

You want your experimentation quality assurance to be quick, intuitive, and entirely within reach of your testing tool. We made it happen.

Say goodbye to toggling between screens, tabs, and tools to guarantee the integrity of your tests. With our new QA tool, you have a one-stop overlay for validating your experiments in a live environment.

Need to preview different test variations? Just click or scan a QR code, and you can switch between original and variant setups on the fly. Catch bugs, mismatches, or rendering issues before they impact real users across devices.

This QA tool gives you the data that matters. See which experiments are running, track your audience targeting, monitor active goals and access live logs with real-time updates of all your variations.

You can also use the status overview within the QA Overlay to verify your tracking code installation, how in sync it is, and if your experiments are actually running so you can roll out tests with confidence.

And if you’ve got multiple projects running, the panel (which can adjust to your preferred size) lets you validate each one in a single QA window. Get a clear view of which experiences are active, which are in draft, and what goals have been triggered—all in real time.

Now, QA within your favorite A/B testing tool, Convert Experiences, has gotten even more powerful.

Observations

Imagine you’re high on inspiration, getting hit with new test ideas from every angle. Some are groundbreaking, and others deserve a closer look, but they all need prioritization.

Convert’s Observations feature makes it easy to manage these ideas. It’s a centralized, flexible repository designed to store and tag every test idea (or observation) right in your project dashboard under the Compass tab.

When you’re ready to use these ideas, you can make them a Hypothesis (also in the Compass tab) so you can refine and pass them through a prioritization framework and, when the time is right, act on your best ideas.

You can tag each test idea by the site area you intend to run it on, metrics of interest, and other criteria, creating a smart, searchable catalog of potential experiments.

And if any idea feels like it’s ready for the next step, you can link it directly to a new or existing hypothesis. How neat is that? Set objectives, track progress, and once you’ve proven a hypothesis, convert it into an objective for easy access in your Knowledge Base (another new feature).

Knowledge Base

Convert’s Knowledge Base is your living library of insights, turning each test—regardless of outcome—into a reusable asset.

It becomes your single source of truth. Tested and proven hypotheses automatically find a home here, a repository of your completed tests for maximum learning impact.

You can see exactly who proposed each idea, when it was added, and how it was tested. With tagging and filtering, finding a specific test takes a few clicks, so your experimentation history is easily accessible to you and your team.

This way, your experiments evolve into true learning tools, inspiring more tests, and greasing a really efficient experimentation flywheel with solid A/B testing meta-analysis.

With Knowledge Base, you and your team find it easy to stay on track with your experimentation goals and optimization roadmap, so you can:

• Avoid repetitive testing
• Go from spaghetti testing to strategic experimentation and
• Build on the success of work you’ve already done

Not to mention, it’s a productivity booster that’ll improve your testing velocity and impact.

Live Duration Insights®

Guessing when your test will reach completion is a thing of the past. With our new Live Duration Insights®, you have real-time visibility into how close your test is to reaching statistical significance.

This tool is for users who want to stay in the loop and know exactly how the final test results are trending.

With clear completion estimates, you can prepare in advance—with a measure of confidence—for changes to make based on the final test results as soon as they become ready. You can make decisions whether to stay the course, make adjustments, or pivot as results start to solidify. This can come in handy when a variation is clearly making you lose money.

If you want this transparency and want to sleep better at night in high-stakes testing situations, you should definitely give Live Duration Insights® in Convert a spin.

Dynamic Triggers

It’s too easy to compromise the integrity of your tests in certain situations, such as when tested audiences don’t truly experience the test, but their subsequent actions get included in the calculation of test results.

Here’s what I mean: Let’s say you’re testing a change on your Shopify product page. You want to learn how adding a customer reviews section below the product description impacts add-to-cart rates.

However, a sizable fraction of people who visit this product page don’t scroll all the way down to see the reviews, so it doesn’t have any meaningful impact on their decision to click on the ‘add to cart’ button.

Can you trust that result?

What if you could trigger the test only when people see the tested element, i.e., truly experience the test?

That’s what we’re providing with Dynamic Triggers (AKA manual activation) — the ultimate flexibility to launch experiments only when specific, context-aware conditions are met. The goal: more reliable test results.

These interactive triggers make your tests appear precisely when you want them to, based on user actions, time delays, or specific interactions like scroll depth.

Launch tests when visitors:

• Spend a defined amount of time on the page
• View a specific element
• Click a button
• Hover over a section
• Scroll to a particular area, etc.

For advanced setups, Dynamic Triggers can even respond to customer JavaScript events, giving you even more nuanced control to sync tests with other dynamic elements.

Dynamic Triggers make your tests intentional and run them in perfect harmony with user journeys. It is available on both Convert’s Growth and Pro plans, with the Pro plan offering the full complement of dynamic triggers — the likes of which you need to pay extra to access in most of the other A/B testing tools.

Import and Export Templates

Now, you can move your experimentation data in and out of projects. Convert Import/Export templates enable you to seamlessly import goals, audiences, experiences, and other key elements across projects—even across accounts and, check this out: tools!

We don’t believe in boxing you in. You should have the freedom to move your data however you please. It’s yours.

This feature is also great for Shopify merchants with multiple stores and users with multiple websites. So, if you want to replicate an experience, there is no need to build it from the ground up each time. You can take the data from one and import it into another.

These exports come in a JSON format, making data transfers smooth and ensuring your configurations are preserved across platforms. Pick what you need—individual goals or the bulk of them, audiences, etc.—and move them in a clean, organized fashion.

Copy and paste your setups, backup your setup, be flexible with your data, keep projects across accounts and platforms aligned… There’s so much you can do with Import and Export.

Here’s the kicker: As of the time of writing this, no other A/B testing tool does this. When you start seeing this pop everywhere else, remember who started it ;). We love being a pacesetter in our industry.

Sequential Testing

Do you want to use Frequentist stats, but fixed horizon tests aren’t agile enough for the fast decisions you want to make? We’ve got you covered.

One- and two-tailed tests are no longer your only choices when you go to Stats & Settings in the Experiment Summary window in Convert.

We’ve added Sequential testing!

With this set, you can evaluate the results of your A/B tests at any point after reaching a minimum number of visitors.

This minimum number of visitors, a tuning parameter you can edit, ensures you can make faster decisions without risking premature conclusions. It still maintains statistical rigor with a stringent error rate control, so you can trust the findings from your tests.

This is huge because, with sequential testing, you can deploy winning variants faster or shut down variants that are unlikely to outperform the control. That earns more revenue for you as you deploy a winner in one case and stop burning money in the other.

AI Wizard

This was our first dive into the world of Generative AI. AI Wizard is a widget that sits in Convert’s Visual Editor as your partner for generating copy alternatives with speed.

Convert’s AI Wizard in the Beta Visual Editor

In the Visual Editor, click on an element on your page, such as your website headline, and you’ll find the AI Wizard tab in the left sidebar. Just click on it.

The AI Wizard helps you rewrite headlines and paragraphs of text using various models: the Seven Persuasion Concepts, the Fogg Behavior Model, and the most popular copywriting formats. This means you can swiftly craft alternate versions of your copy for A/B testing.

But let’s be clear: The AI Wizard is not here to replace strategic research, theme identification, or a seasoned copywriter’s strategic deliberation over word choice. Rather, it’s your companion for surfacing different angles to explore your existing web copy, all based on proven copywriting, persuasion, and behavior models.

And it is available to all Convert plans purchased in 2023 and beyond.

Full Stack Experimentation

Full stack and server-side testing are here to stay.

Because server-side testing can get around multiple cookie and tracking issues. And full stack equips organizations with the ability to not only test UI changes but also permeate the willingness to let data guide decisions when it comes to investing in features and even in determining the business model adopted.

We dive into all the details in our full stack experimentation guide but here’s the rundown of why server-side and full-stack are so important. They:

• Fuel continuous learning and delivery of higher quality products,
• Provide a comprehensive view of the customer experience across channels, devices, and interactions,
• Help manage the risk of focusing solely on a single success metric,
• Accelerate innovation, reduce time to value, and minimize operational and performance risks,
• Enable deeper testing of website aspects such as logic and functionality,
• Allow for product feature testing, enabling the testing and refinement of new ideas and features,
• Offer rich aggregation of business-wide insights for data-backed decision-making,
• Allow testing of components coming from the backend of websites,
• Allow testing of certain aspects of mobile apps, and more.

Companies like Etsy, Microsoft, Uber, Amazon, Facebook, LinkedIn, Google, Twitter, Netflix have used full-stack experimentation to fuel innovation and improve the customer experience on their sites. They understand that moving away from front-end changes only can open up a sea of strategic possibilities — allowing development to more directly support and inform the evolution of the business. We understand that too. And if you don’t already, you should too.

That’s why we’ve decided to take things to a whole new level with our Full Stack features—with server side experimentation, feature flags, and rollouts. Convert is truly a comprehensive experimentation platform that allows you to test and optimize across your entire tech stack, growth channels and motions.

You can now roll out new features to a subset of users, test their impact, and then decide whether to roll them out to everyone or not. You get to see how your users react and then decide whether to go for a full release.

Our SDKs allow you to integrate Convert’s robust features into your own applications. We currently support JavaScript (most popular JS frameworks are supported) and PHP SDKs.

Customizable Bayesian & Frequentist Stats Engines

You can now tailor your stats engine to your preferences.

For instance, for Frequentist, you can choose one-, two-tailed, and sequential test types, along with comparison corrections like Bonferroni and Sidak and your desired confidence level.

Alternatively, opt for Bayesian reasoning to focus on the Chance to Win instead of the more traditional Confidence value.

Convert’s Customizable Bayesian and Frequentist Stat Engines

You now have the power to pop the trunk and mod the engine running your experiments — putting even more control in your hands.

Now, you may be wondering: When do you use Frequentist and when do you go with Bayesian? Contrary to popular opinion, it isn’t about the sample size. It’s about your comfort with uncertainty and your need for actionable results.

Bayesian is great when you’re comfortable with a degree of uncertainty and need to make decisions quickly. Frequentist is your go-to when you need more controlled results and are willing to do proper test sample size prepping and resist the temptation to peak.

Frequentist is the sturdy oak tree, while Bayesian is like the flexible bamboo.

With the power to choose the stats engine you want for the experiment you’re running, Convert is supporting you by shortening the time from observation to insight to decision-making.

Visual Editor Updates

Now, each individual change is connected on a DOM level, which means less duplication, easier re-editing, and deletion of specific past changes without having to edit the entire code.

This is a game-changer for those who want to make quick and precise changes to their A/B testing experiments. You can fine-tune the elements of your website with surgical precision.

This further simplifies test deployment for CRO teams. You spend less time and effort in launching and modifying tests, and free up time for action and learning.

Locations

Site Area, your familiar tool for specifying the exact places where your experiments trigger, is now Locations.

Here’s what this means:

• Old experiences with “Site Area” will continue to run without changes.
• New experiences will have “Locations” (instead of Site Area).
• Locations are reusable. They can be saved and used across experiences to save time.
• Site Area has been phased out.

We hope this helps you build reusable and more structured CRO programs for your team and clients.

And in the spirit of freeing up resources for people and processes, we’re thrilled this means you get to save time you would otherwise have spent building new locations for every experience.

API V2

The API V2 is a major upgrade that allows you to easily access account information as well as reporting information, while giving you access to the full experimentation lifecycle.

Everything that can be done through our UI can also be done via the API.

You can

• create an experiment programmatically,
• run custom analysis on experiment results,
• build customized integrations and workflows,
• create feature flags from an automated script,
• build custom dashboards of feature test results,
• connect your experiments to other project management tools, and more.

You can create projects, or completely automate the whole experiment pipeline without ever touching the Convert UI, up to you.

Native GA4 Integration

We’re not just keeping up with the times, we’re leading the charge with our native GA4 integration. This isn’t just a minor update, folks.

With this integration, you can seamlessly connect your Convert Experiences with GA4, making it easier than ever to track and analyze your experiments.

No more juggling between platforms or wrestling with incompatible data. Everything you need is now in one place, working together in perfect harmony to shorten your time from observation to insight to knowledge.

But what does this mean for you? You can now leverage the power of GA4’s advanced analytics capabilities right from your Convert Experiences dashboard.

You can track user interactions, analyze user behavior, and gain deeper insights into your experiments, all without leaving Convert Experiences.

And the best part? This integration is based on Google’s official integration API documents, which means it’s secure, reliable, and designed to work seamlessly with GA4.

Plus, Convert is one of the first A/B testing providers to have an official functional integration with GA4. So, you’re not just getting a cutting-edge feature, you’re getting a trailblazing solution from a trusted vendor. Get the details in our official announcement.

GA4 Conversion Events (Goals) Import

Imagine you’re a chef, and you’ve just spent hours carefully preparing a feast. But when it’s time to serve, you realize you’ve left all your dishes in the kitchen.

That’s what it’s like setting up conversion events in GA4 and then not being able to use them in your A/B testing tool. But with Convert Experiences’ new GA4 Goals Import feature, that’s a thing of the past!

Just like a waiter whisking your dishes from the kitchen to the dining table, our GA4 Goals Import feature allows you to easily import the GA4 conversion events you’ve set up within Google’s ecosystem into Convert. This means you can gauge the success of your variations and experiments against these goals, all within Convert Experiences.

What’s better: This feature is packaged within our one-click integration that requires Google authentication. It’s like having a personal assistant who not only brings your dishes to the table but also does them afterward!

That way, you can move from observation to knowledge even faster. Convert’s integration with GA4 gets even better…

GA4 Automated Revenue Tracking

It’s common to find your A/B testing tool reporting one revenue value while Google Analytics reports another. That’s not a good look when we’re talking about money.

With Convert’s new GA4 Automated Revenue Tracking feature, that’s over. Our tool will use the total revenue collected by GA4 as the basis of its own revenue reports and Average Revenue per User (ARPU) calculations.

This means you’ll get trustworthy results that align with your GA4 data, saving you hundreds of hours of troubleshooting and test design reviews. Yet another way we help you get from observation to insights with speed.

27-Second Google Optimize Data Importer in Convert Experiences Tools Extension

We know you haven’t forgotten Google Optimize yet. It offered a low barrier to entry for many testers but also encouraged a lot of random, one-off efforts.

Our goal at Convert was to comprehensively replace Google Optimize—for the eager experimenter—with a solution that doesn’t require as many hacks, won’t break the bank, and will be there for the foreseeable future. (In fact, one of our first customers is still with us — paying $9 per month).

So, we’ve built a data importer that allowed you to quickly and easily migrate your Google Optimize data into Convert Experiences.

All of this baked into the Convert Experiences Tools Chrome Extension, which also helps you optimize your experience development, testing, and debugging through some key features.

The extension gives you deep insights with detailed logs, instant access to Convert’s installation snippet, and smoother workflows with the Visual Editor.

You can manage experiments directly in your browser, make code changes on the fly, verify your target audience, and troubleshoot conversion issues with ease.

Convert Assistant

It’s a custom GPT you can access via ChatGPT with current knowledge of our extensive support database and articles. You no longer need to sift through pages of documentation or wait for support responses. Simply ask and receive.

To use Convert Assistant, you need a ChatGPT subscription. It allows you to use GPTs. 

Then, you can reach out to us via support@convert.com or our other help channels for direct access to the GPT. Because it isn’t publicly available yet.

Then, whenever you’re curious about a specific feature or how to optimize its use, ask the Convert Assistant GPT. It will give you specific answers to your questions in seconds.

Facing an issue or need guidance on a particular topic? Convert Assistant taps into our support documents and blog articles, offering you precise knowledge in real time.

Beyond Convert-specific details, the Assistant is well-versed in the broader realm of experimentation. Whether you’re new to the scene or a seasoned pro, it’s like having a quick guru by your side.

Conclusion

In lieu of a conclusion, we’ve drawn upon insights from Ton Wesseling and Jonny Longden’s thought-provoking LinkedIn posts, which provide guidance for selecting a suitable experimentation tool vendor.

So, if you’re wondering what to consider when making such a choice and if Convert is the right choice, here’s what you need to know:

1. Pricing Structure: Understand the pricing structure of the tool. Is it cost-effective for your organization’s needs?

Convert provides flexible self-service pricing. Plus, we don’t force you into an annual plan and our pricing is significantly more affordable compared to what’s popular out there. On top of that, we give you a free 15-day trial with no commitments.

2. Integrations: The tool should have the ability to integrate with other platforms, particularly data integrations. API capabilities are also important.

Our API V2 is out and we’ve got over 90+ integrations with tools in your martech stack.

3. Support Availability and Quality of Documentation: Good customer support and comprehensive documentation are crucial for troubleshooting and understanding how to use your experimentation tool effectively.

Besides our comprehensive support resources, you can reach our responsive support team from inside our platform in just 2 clicks.

4. Data Quality: The tool should provide high-quality data, including SRM (sample ratio mismatch) aspects.

Convert Experiences comes with a useful SRM checker. Once enabled, the tag will look like this:

5. Transparency on Statistical Models & Ability to Run Your Own Stats Calculations:

The tool should provide clear information about its statistical models to ensure trustworthiness. It should also allow you to perform custom statistical calculations.

We’re very open about a lot of things. This one is no exception: Here are the statistical models we use.

Plus, Convert’s A/B testing significance calculator supports revenue metrics like AOV & ARPV.

6. Server-side Solutions: The tool should offer server-side testing capabilities. Our full stack testing feature allows you to test both front-end and back-end aspects of your product. We support both server-side and client-side testing.

7. GDPR Compliance and Strict Cookie Policies: The tool should be compliant with GDPR and other relevant data privacy regulations. We’ve been the forerunner of data privacy, security, and compliance for many years now. We’re GDPR and CCPA compliant.

8. Ability to Influence Product Roadmap: The tool should allow for user feedback and influence on its product roadmap. We maintain close relationships with our users and agencies who use our product. We listen and use your feedback to evolve Convert.

9. Targeting and Segmentation Functionalities: The tool should offer robust targeting and segmentation options.

Convert offers an advanced targeting engine with 40+ stackable filters, and the flexibility to use data locked in 3rd party systems. Plus, after your experiment starts receiving traffic, you can access the advanced post segmentation feature:

10. Load Time / Flickering: The tool should have minimal load time and prevent flickering issues. We have super fast loading times and solved the flickering problem years ago, back when it was still a luxury.

Anti-flicker technology has always been baked in our tool at no additional cost.

11. Ease of Use / UX: The tool should be user-friendly and manageable, even with multiple accounts/tests. We have countless reviews praising our “idiot-proof” UI.

12. Consistency and Predictability of Traffic Volumes: The tool should be able to handle consistent and predictable traffic volumes.

14. Visual Editor and Code Editor Feature: The tool should provide both a What You See Is What You Get (WYSIWYG) editor and a code editor.

Convert’s robust Visual Editor is usually one of the first features our users fall in love with — and the code editor features give you extra flexibility in building tests with custom JS and CSS.