Fundamentals of Product Experimentation for Beginners

Growing a successful product means constantly adapting to your users’ needs. Product experimentation is key to this. It’s a tool for testing and improving your product while you continuously develop it.

Using this tool, however, could take a bit more effort than you’d expect at first. In this product experimentation handbook, we’ll show you how to start, what metrics matter, and how to use a product experimentation framework effectively.

So, product managers, growth marketers, developers, founders, or anyone looking to enhance user experience and boost product adoption and retention, hang on to this.

In the end, you’ll be able to use product experimentation to validate each step of the unending journey toward the better version of your product for your users.

Let’s get started!

Why Is Product Experimentation Essential in Today’s Climate?

Tech giants like Booking.com, Netflix, Meta, Google, Microsoft, and Amazon have been testing hypotheses to enhance their products and services for decades. Nowadays, they conduct up to 10,000 tests each year.

Here’s what they understood:

1. Relying solely on intuition and best practices for product development decisions isn’t enough. A data-driven approach is crucial for understanding users and making informed improvements.
2. Before making any changes or launching a new product, it’s vital to validate assumptions and test hypotheses.
3. Embracing the scientific process is essential for adapting to market shifts and changing user preferences in a calculated manner—as opposed to haphazardly.

This is one way they’ve been able to cement their status as tech giants and grow insanely fast.

Of course, many factors contribute to the value of a company. Still, you can’t deny the power of experimentation-derived insights that support better decisions — whether in evolving the product or avoiding damaging changes. If you deny it, I’ll refer you to Blockbuster and Digg.

For most businesses today, SaaS companies especially, product experimentation is a must-have. You’ve got considerable competition and rapidly switching market preferences to contend with while building products. How would you know your product development is on the right track to meet business goals?

With experiments, you can establish a causal link between ideas and outcomes, so you’ll know precisely what to build, upgrade, or tear down. It also empowers your team with a strong, well-rounded understanding of various strategies and their impact, and helps you build on previous knowledge through iterative learning.

Ultimately, product experimentation accelerates innovation and product evolution. As ideas are implemented based on solid evidence from testing against key product health metrics, you get to move faster in the right direction.

Product experimentation is also a cornerstone of product-led growth (PLG). Here’s how…

Product Experimentation Meets Product-led Growth (PLG)

Product-led growth is a customer-needs-focused strategy for growing your business that depends on your product to do the heavy lifting. With such a heavy reliance on the product to acquire, activate, and retain customers, it must be truly solid.

That’s where product experimentation comes in. To acquire customers, for one, your product needs built-in virality — its own ability to:

• Drive outstanding outcomes (like ChatGPT did with human-like interactive AI)
• Get users to discover value quickly (like Canva does with thousands of professional-looking visual content templates), or
• Invite others to partake as well (similar to Dropbox’s “invite friends for more storage space”)

The easier it is to realize these Aha! moments with products, the easier it gets to activate users, i.e. get them hooked, build a habit, re-engage, and retain.

Product experimentation makes this happen in the way it illuminates the customer journey, helping your organization build that understanding grounded in evidence.

Another way it puts your customers front and center is by fuelling this core tenet of PLG: continuous improvement. You get to create seamless and satisfying user experiences one validated step at a time.

This leads to a lot of benefits including lower customer acquisition costs and scaling faster, regardless of headcount. All with the insights from product experiments that support product-led growth.

Ruben de Boer puts this into perspective as an emerging trend among modern organizations:

Product-led growth will get experimentation more and more into the product teams. I see this trend happening in many organizations, and I am confident more organizations will follow.

Experimentation is indispensable when products focus on a growth outcome instead of simply rolling out features.

Product-led growth organizations are geared toward creating impactful changes that influence customer behavior, satisfaction, and revenue. To realize this, product teams need to make the right decisions for growth. For this, they need to experiment.

Imagine if just 25% of product updates make a positive impact on the outcome, it’s crucial to know the 75% that shouldn’t see the light of day. Not experimenting will jeopardize the end goal.

Furthermore, experimentation allows for fast innovation cycles, meaning the product will continuously evolve to meet market demands, keeping users engaged and attracting new ones.

Thus, experimentation is crucial for successful product-led growth.

Ruben de Boer, Lead Conversion Manager, Online Dialogue

Product experimentation offers a measure of assurance in maintaining a great user experience by selectively rolling out impactful ideas while holding back ideas, features, and upgrades that could muddle the user’s journey and disrupt the delicate sequence of positive interactions leading to product endorsements.

Instead of planning a roadmap for an entire year, experimentation allows for a flexible roadmap as it offers insights, validates assumptions, and drives quick iterations and innovation.

Experimentation provides data-driven insights that reduce reliance on gut feelings or assumptions. It enables teams to make informed decisions based on continuous user behavior and feedback, data analyses, and experiment outcomes. Product teams can create and test better features and designs as they better understand the users’ preferences, gains, and pains.

Because this learning happens continuously, the product direction changes based on the insights to match the users’ needs best.

While experimentation significantly influences the product direction, it is key for product teams to balance these insights with the company’s strategic vision. Integrating the learnings with the company goals ensures that the product meets user needs and aligns with the organization’s growth and innovation objectives.

Ruben de Boer

Web Experimentation vs Product Experimentation

Web experimentation and product experimentation, though quite similar (and often intertwined), serve distinct purposes within the sphere of optimizing user experience and product development.

We could think of website experimentation as a small part of product experimentation, in the same way (controversial, I know) that we could understand CRO as a small part of experimentation, or UI/UX design are parts of Product design.

Strictly speaking, web experimentation would deal solely with the website itself, and the digital interface, whereas product experimentation could involve testing all aspects of a company’s offering, from improving features to introducing new ones, its pricing, delivery, and aftersales, etc. It’s full-stack experimentation on steroids.

Some lines may seem blurry, like whether improving the algorithm on a product like Spotify’s recommendations is a web experiment or a product experiment, but from my point of view, if it’s not to do with the interface itself, it’s product experimentation.

Therefore, and linking back to a previous Convert article, we could argue that innovation (Exploration) is much more likely to happen with product experimentation, whereas exploitation is largely in the realm of web optimization.

Properly executed, product experimentation can lead to truly transformative outcomes for a business as its depth and breadth are much larger, but it can be more costly and resource-intensive since the risks are much higher.

Web experimentation has a much lower barrier of entry, so it can be a great gateway for companies to evolve in their maturity over time and embrace a culture of experimentation over time. But that takes time.

David Sanchez del Real, Head of Optimisation at Awa Digital

Here’s a breakdown of their differences based on their focus and scope:

Scope

Web Experimentation	Product Experimentation
Tests variations in web interfaces such as landing pages, forms, or other parts of a website to optimize specific outcomes like click-through rates, conversions, and other web metrics.	Tests the entire product, including features, workflows, and even pricing models. It’s about optimizing for the best version of the product that meets the user’s preferences while achieving business goals. It’s like a more expansive form of full-stack experimentation.

Focus

Web Experimentation	Product Experimentation
The UI and UX of the website for higher conversion, better engagements, or other web-specific goals.	The entire user experience within the product, which can also include the website, for higher retention, revenue, and growth metrics.

Depth of Analysis

Web Experimentation	Product Experimentation
Usually involves marketing- and sales-level analysis of the business.	Goes deeper into how different aspects of the product affect overall user behavior and business outcomes.

Example

Web Experimentation	Product Experimentation
A/B testing a different form layout on the same landing page to see which produces the higher conversion rate.	Testing a different onboarding flow within an app to see which results in better user retention and engagement over time.

Kelly Anne Wortham puts a deeper spin on this:

Product testers should keep in mind the important difference between marketing/website testing, which has always been about the ability to “fail faster,” and product testing, where failures are incredibly costly. Because of those costly risks, it’s more important with product testing to fully understand the customer journey—qualitatively and quantitatively—before you make product decisions.

To do that, product testers should take these steps:

1. Utilize extensive research first – both qualitative and quantitative
2. Practice customer empathy to define what success looks like
3. Use rapid experimentation methods to gather evidence to support your build.

The standard product life cycle is Build-Measure-Learn (hopefully).

However, a sound product experimentation framework turns that product lifecycle around by first asking the question: What do I want to learn?

Then: What would I need to measure to learn that?

Last: What would I need to build to measure that?

Then, and only then can we start to Build-Measure-Learn guaranteed.

Kelly Anne Wortham, Founder of Forward Digital Org

Product Growth Stages: Key Metrics that You Can Impact with Experimentation

Inspired by Elena Verna’s ‘dirty dozen’ of PLG B2B SaaS health metrics, we’ve outlined key PLG metrics across the entire user journey and illustrated how strategic adjustments in onboarding processes, feature testing, marketing approaches, pricing structures, and more can enhance the user journey.

Let’s take a look at how you can leverage experimentation to attract, engage, and convert users more effectively, driving growth and revenue with precision.

Growth Stage: Acquisition

PLG Metric	How to Impact With Experiments
Web traffic: Measures the volume of new potential customers visiting your website.	Run experiments on marketing channels and page elements to determine what attracts higher traffic.
New accounts signups: Measures the number of accounts that have completed the signup process and begun onboarding.	Test different signup forms, social proof, and signup incentives to increase the number of users who complete the signup process.

Growth Stage: Activation

PLG Metric	How to Impact With Experiments
Activation completion rate: Tracks the percentage of users who complete the onboarding checklist.	Experiment with various onboarding guides, tooltips, and interactive tutorials to see which increases the completion rate.
Time to value (TTV): Calculates how long it takes a user to derive value from the product after signing up.	Test and measure how changes in the onboarding process affect the speed at which users find value in the product.
Activation velocity: Measures how fast users reach the activation stage in their user journey.	Experiment with steps to reach activation, providing more direct paths to key features or incentives, and targeted communication to boost speed to the activation stage.

Growth Stage: Engagement

PLG Metric	How to Impact With Experiments
Product stickiness: Measures how often users return to use the product—daily, weekly, or monthly (e.g. DAU, WAU, MAU)—which indicates engagement levels and retention potential.	Test different features, content, or engagement strategies to keep accounts within the desired frequency of use.
Product adoption rate: Indicates the percentage of users who are regular users.	Experiment with features or incentives that encourage existing users to explore the product’s value more regularly.
PQL (Product-qualified leads): Leads with high engagement levels which indicates they’ve experienced value with the product and have a higher likelihood to convert to paid plans.	Test different criteria for what defines a PQL to refine the sales team’s focus on the most promising leads.

Growth Stage: Monetization

PLG Metric	How to Impact With Experiments
Conversion to paid accounts: Measures the number of new accounts that upgrade to paid subscriptions.	Experiment with different pricing structures, trial lengths, or premium feature access to see what converts more accounts to paid plans.
Average revenue per new user: Tracks the average revenue per each new account, which tells you the value of the initial conversion.	Test upselling strategies, bundle offers, or personalized recommendations to increase the average revenue per new user.
CAC payback period: The time it takes for your organization to recoup the customer acquisition cost.	Experiment with more targeted customer acquisition strategies to reduce upfront costs and shorten the payback period.
Expansion revenue: Measures revenue from existing customers that result from upgrades, add-ons, or adding more users to an account.	Test different in-app prompts or email campaigns to encourage existing customers to purchase add-ons or upgrades.

As these PLG metrics tell the health story of your product and growth trajectory, experimentation helps you make data-driven decisions to rewrite the story.

Let’s put things in perspective. Here’s an example of a successful product experiment, where a marketplace drove millions of additional deals by running product tests in the activation growth stage.

They observed an unused feature called “save search” correlated with higher user engagements via messaging. So, to drive user engagement they had to push this feature into the spotlight.

First, they tested tooltips triggered on users’ first search, informing them of the feature. They tested a floating ‘save search’ button that followed the user as they scrolled through search results, increasing its visibility and accessibility. They also tested a prompt to save a search if the same search occurred thrice.

This resulted in a 300% increase in the save search feature usage. And that translated into a 44% increase in buyer-to-seller message exchanges.

Sometimes, product experiments ‘fail’ in the technical sense but reveal valuable insights.

Example: A company tested adding a new feature that was highly requested by many users in the hopes of increasing trial-to-paid (TTP) conversion, but it failed.

Doesn’t make sense, right? But they learned not to take test results at face value. When unexpected results show up like this, you can segment users to see a more detailed story.

For them, they learned that the segment of their users that resulted in no change in TTP rates wasn’t aware of the feature and never realized its value. They also learned that even people who didn’t use the feature but were aware of it were more likely to convert to a paid plan because of the perceived value. So, they prioritized notifying users of the feature.

Product Data & Analytics: What Do You Need to Have in Place Before You Can Experiment?

With product experimentation, you never want to take shots in the dark. So, your foundation must be built on critical elements to ensure your experiments are, as Ruben said, “structured, measurable, reliable, and yield actionable insights”.

Here are those elements:

1. Rally Your Team Around Clear Goals

Right at the top, create clear and measurable objectives for your product experimentation program—goals built around the product and your unique definition of success.

Clear objectives and measurable key metrics are essential to get everyone on the same page and working towards a common outcome-driven goal.

Ruben de Boer

Next, you want to make sure your entire team is on the same track in terms of the goals and KPIs governing your product experimentation program. Not only is this great for marching forward in unison, but it also makes it easy to get the support you need to succeed.

First, it is essential to determine the most important goal and corresponding KPI for your product. What would make the product more successful, what is a good metric of that? Sometimes it is a kind of usage metric like activity rate, daily returning visitors, or related, especially for subscription products.

If it is more commercial like a booking kind of platform, the number of bookings (or percentage of users booking something) will of course be very important. Also, it is important to align these goals with the team working on it, since it might not be that straightforward for everyone as it is within eCommerce.

Lucas Vos, Senior Conversion Specialist, RTL

When conducting product experiments, it’s crucial to establish well-defined success metrics, test multiple variations simultaneously, and be patient. A common pitfall is to conclude an experiment prematurely. Statistical significance is a thing. In my experience, 9 out of 10 experiments I’ve overseen saw shifts in the winning variation within just a couple of days of data collection.

Henning Heinrich, Head of Growth at Glassfy

2. Acquire Experimentation Tools with Server-side Testing Capabilities

You need testing tools that’ll accommodate experimenting across platforms and layers of your product.

Check your tool stack. Because digital products most of the time are active on more platforms than one, you might be having a challenge with your experimentation tool. Some of the standard tools (like Optimizely) support app testing for example, but it might be more complex to set up within those tools.

With that, it could be difficult to test the same changes on multiple platforms at once. Therefore, server-side testing might be something you want to investigate, as it should open possibilities to test the same changes on multiple platforms at the same time.

But to start, you can also consider Firebase to test on Android and iOS apps. That could be very useful through triggering different configs within your apps.

Lucas Vos

Examples of such tools with server-side testing capabilities are:

• Convert Experiences
• Split
• LaunchDarkly
• AB Tasty
• Kameleoon, and more

And when implementing these tools, make them available to the rest of the team.

To get the best ideas and insights, everyone must be able to contribute to the process. Therefore, access to tools and data must be available for everyone.

Ruben de Boer

3. Implement Proper Data Collection and Analysis

Make sure your data collection process is strong and your analysis sharp. With reliable data, gaining your team’s trust and involvement becomes easier.

All data measured must be accurate and reliable to draw trustworthy conclusions. Furthermore, the data must be complete. All useful metrics must be available.

Ruben de Boer

Yes, this doesn’t translate to gathering every bit of data you can find. That can become a roadblock if done incorrectly.

Streamline your data collection process. While collecting more data may seem appealing, the potential drawbacks such as increased errors, compromised data quality, and higher costs often outweigh the benefits. It’s essential to consider what specific data is truly necessary for informed decision-making and to prioritize it rigorously.

[…]

In addition to the aspects of data visualization and analysis, product analytics offers an exciting opportunity to automate product-led growth (PLG) initiatives, such as activating power users in referral campaigns. Achieving this involves properly segmenting user cohorts and integrating with the right tech stack from the get-go.

Henning Heinrich

4. Imbibe a Culture of Learning and Improvement

Cultivating a culture that views failure as a learning opportunity encourages experimentation and innovation.

It’s also a great catalyst for getting hands on board because, with product experimentation, the involvement of multi-disciplinary teams is crucial to progress faster with a comprehensive approach.

To gather the most reliable and useful insights, you need specialists in UX research, UX design, data analysts/scientists, developers/engineers, and possibly a psychologist.

[…]

In addition to having the right team and tools, fostering a culture where failure is viewed as an opportunity to learn and improve is essential. Also, strategy, stakeholder buy-in, and education are crucial things that determine the success of product-led growth.

Ruben de Boer

An experimentation culture also means you make a habit of gathering user feedback as well as quantitative data. And you use them to generate problem statements and hypotheses to experiment on.

A process and tooling must be in place to gather user feedback. This can be done through surveys, interviews, and usability testing.

Ruben de Boer

There’s more. An experimentation mindset makes you, your team, and your leadership respect the learnings from failed tests—which is something to contend with in every experimentation program.

Fostering a culture where failure is viewed as an opportunity to learn and improve is essential. Also, strategy, stakeholder buy-in, and education are crucial things that determine the success of product-led growth.

Ruben de Boer

Finally, Lucas leaves us with some details you don’t want to miss if you’re testing apps:

When doing app testing, please consider that releasing a new app version with your experiment needs some time to get adopted because people need to update their app. Depending on the platform and product it might take more than a week for a serious ramp-up of usage of the new version, providing the necessary users for your experiment.

What happens within an iOS app can really be different within an Android app. At some point, the groups can be just different, as there are some minor differences in their characteristics. Test changes on both platforms.

Lucas Vos

The Structure of a Product Experimentation Team

To start product testing, a company needs a solid team composed of product managers, data analysts, designers, developers or engineers, and user researchers.

They’ll collaborate to plan and run experiments, typically following one of three common models below:

1. Centralized Model

Here, the experimentation team is at the center of all experiments in the organization. They sort of provide experimentation services to other departments and business units in the company.

What’s neat about this structure is the way it keeps all experimentation data in one place. It’s also easy to stick to program objectives, compliance, and culture. However, it can lead to delays in test deployment and may hinder others from getting involved.

2. Decentralized Model

In this model, experimentation resources are spread out among various teams and departments. This will allow them to independently hone in on specific areas like customer acquisition, conversion, or retention.

Individual teams get to own their experimentation process, but that can create silos and data inconsistencies.

3. Center of Excellence Model

This hybrid model merges the centralized and decentralized approaches. Individual teams handle their own experiments, with a central ‘Center of Excellence’ (COE) providing support and standards. It grants teams flexibility and autonomy yet ensures consistency and promotes cross-team coordination.

But this comes with its own challenges as well. Lines of responsibility can be blurred here. Teams may struggle to identify jurisdictions or decide when to seek guidance from the COE. There’s also the need for more funding for a bigger program.

Which product experimentation team model is right for your organization?

That’s up to your organization’s capabilities and goals. The centralized model is the best way to introduce product experimentation. But as your experimentation program matures, you’ll probably want to switch over to the center of excellence structure.

Product Experimentation Framework

A product experimentation framework is the basic structure that guides the tests you run to improve the product experience.

Melanie Kyrklund provides valuable insights into the broader context of product experimentation:

The process or framework for running product tests. How is this different from regular website testing?

One of the goals of product experimentation is to de-risk development by providing greater certainty on where to focus resources. Experimentation is a process that allows product teams to reach the goals they have set — which are anchored in business and customer needs and measured via KPIs and metrics specific to the outcomes and product being worked on.

Greater integration between engineering and experimentation processes is required to facilitate the validation of features of higher complexity, making server-side testing more prevalent in this field. Experiment development and QA processes need to take into account that a lot of code will be thrown away. Cost considerations are more important when prioritizing and building experiments server-side vs client-side.

Product teams will venture into testing more disruptive and complex changes compared to marketing teams. Product strategies are broken down into MVPs first so that key assumptions can be tested before considerable investment is put into building features. Testing is continuously used to validate features and strategies as they take shape. These types of tests fall in the “exploratory” and “validation” streams depending on the maturity of a feature. The third bucket at the end of this process is “exploitation” where features are fine-tuned and lucrative elements are optimized.

Inversely, exploitation is more the remit of website testing. It typically falls under the remit of marketing and commercial teams and is aligned with marketing and growth objectives. It is concerned with the optimization of key marketing journeys (landing pages and routes into conversion funnels) and the exploitation of promotional and merchandising vehicles. It is more likely to leverage client-side technology.

Melanie Kyrklund, Global Head of Experimentation, Specsavers

Next, we’ll delve into a 5-step framework for conducting impactful product experiments. We draw inspiration from the framework designed by Diksha Shukla, experimentation pro.

1. Define KPIs

Build a dashboard tracking key product health metrics and KPIs to monitor user challenges, backed by user research. These KPIs must be relevant to your product, users, and business goals. You can reference the PLG metrics we mentioned above to guide you.

The metrics you choose must be relevant to your desired business outcomes and should be sensitive to the interventions made through your product experiments.

Ask yourself, “If this metric changes either positively or negatively, would that benefit or harm the business?” That thought process will steer you toward choosing impactful and actionable evaluation metrics, providing a clear metric framework for deciding on what to experiment.

2. Conduct Research

Understand your users’ pain points by examining their emotions, motivations, and situations through interviews, polls, and surveys. Use both quantitative and qualitative data for a complete picture. This research will provide the behavioral data you need to develop informed hypotheses for your experiments.

This step is a crucial foundational stage for ensuring your test design is based on solid data and clearly defined metrics. The statistical rigor of your experiments begins here.

3. Identify Gaps and Formulate Hypotheses

Let your data help you find problems. Where are users dropping off? What experience is causing that? What solutions can you formulate to fix that? Brainstorm together to create testable hypotheses based on your research. You can also use a hypothesis generator to put it all together.

However, you need to be alert to product experimentation pitfalls that appear here.

One way to ensure you’re not messing up the statistics of your tests is by formulating a correct null hypothesis.

4. Prioritize Test Ideas

What test ideas in your backlog should you test first and why? Consider one of these prioritization frameworks:

• RICE framework: Consider reach, impact, confidence, and effort to prioritize features.
• Value-effort matrix: Evaluate features to test based on their potential value versus the effort required.
• Kano model: Prioritize features that provide the most user satisfaction (although this ignores cost and feasibility)
• MoSCoW method: A straightforward one that categorizes features into must-haves, should-haves, could-haves, and would-like-to-haves.
• Weighted scoring model: In this model, you assign scores to features based on specific criteria such as feasibility, strategic alignment, customer value, etc.

Choose the one that fits your specific experimentation scenario, product needs, and the nature of the features. Because the ultimate goal remains to make data-driven decisions, not follow a process.

5. Test, Learn, Iterate

With an experimentation tool that suits your needs, design your test and launch it. Then, collect results, learn from them, and iterate. By iterating, we mean the opposite of spaghetti testing. Using a structured process, delve deeper into the insights you’ve acquired from your tests (whether they won or not) to understand the ‘why’ behind the results.

It is common to make mistakes when analyzing test results. So, keep a straight head with your numbers. For instance, it can be super tempting to sneak a peak at how your test is doing early on. Don’t do that. Wait until your sample size is adequate before you make any calls.

And if you’re testing multiple features or changes at the same time, be aware of network effects. Control for multiple comparisons.

One more thing: watch out for those ‘outstanding’ results. Sometimes a bump in your numbers is only due to novelty effects and seasonality. Holidays or weekends can skew your stats, so take those into account and focus on lasting impacts.

Displaying Value Through Your Product Experiments

You don’t want to run product experiments just because it’s cool to do so. You want to bring tangible value back to the table. To start, keep the cost of testing in check.

How do you do that?

• Add a “cost to implement” criterion in your prioritization framework and pick test ideas that bring the most to business goals at a lesser cost
• Use A/B testing tools with built-in full stack experimentation capabilities
• Be careful about variants that could have a huge negative impact on revenue, because as Dennis Meisner wrote, their exposure impacts business figures
• Prioritize time efficiency, where you value quick experimentation cycles, high-velocity testing, and faster iterations. Also, in relation to the point above, the shorter the time a less-performing variant is exposed, the lower its impact on revenue.

After you start off on such a strong foot, turn your attention to sample efficiency. Sample efficiency in product experiments refers to the ability to pull reliable results from the least amount of resources. That’s another way to make product experiments more valuable.

So, to enhance your sample efficiency, you need to craft strategic hypotheses based on SMART business goals and prioritize the high-impact ones. It goes back to everything we’ve discussed so far.

You should also implement efficient sampling techniques such as sequential testing, which adjusts sample size based on data gathered as the test progresses. It makes tests run faster while reducing the sample size required for a statistically significant result.

It’s also sound practice to document every experiment in your experiment repository to maximize the insights you gather and continually improve sample efficiency.

With a reliable record of experiments, proving the ROI of testing becomes much easier. Use the metadata from your tests to show value by calculating:

• The time it took your experiments to pay back its initial investment
• The percentage of the output of benefits to input of resources of your experiences (translation: ROI of experiment = (Profit from the experiment – cost of the experiment) / cost of the experiment)

You can also benchmark the metrics you improved against industry averages and internal records.

This excites stakeholders about what you’re doing in experimentation and its impact on product direction. As you track learning and insights (with an unshakeable focus on metrics that matter), you can display how experiments have influenced the changes your product has gone through.

How to Embed Experimentation in Your Product’s DNA?

Because experimentation isn’t a one-off strategy, you’d want to build it a permanent home in your product development—right in its DNA. To do that, you first have to cultivate a culture of experimenting with your product.

This leads to innovative progress, continuous improvement, and sustained competitive advantage. Three things you need to survive as a business today.

Encourage a culture of thinking big, starting small, and failing fast. Be unafraid to come up with big ideas that may break things. Experimentation allows you to give these ideas (which are definitely based on proper research) their fair trial.

And when tested correctly, that’s only when you can say for sure that your idea was whack instead of just dismissing it in your head.

Plus when you start, start small in workload and ego. Identify the specific problems your users face and create a diced-up plan to solve them. Those are the ideas you test. Prepare yourself and your team to have your brilliant ideas shattered by the results and open up to ideas you’d otherwise dismiss.

This fosters a culture of continuous learning, where you encourage regular retrospectives, host knowledge-sharing sessions, and create a safe space for sharing failures and leanings.

If you’re responsible for leading your team in experimentation, promote vulnerable leadership. Be willing to admit mistakes and encourage others to do so with trust and honesty.

Celebrate big learnings too, not just big wins. With continuous experimentation and learning habits, your product’s DNA will be infused with experimentation quickly.

In summary, product experimentation as a continuous process of making data-driven decisions in developing your product around your users’ preferences is possible when you:

• Involve leadership in experiments so they back teams and assess results
• Value experiment outcomes, both wins and losses alike
• Build and keep a team of people with an experimentation mindset
• Measure product experimentation impact with the right metrics
• Test fast and iterate often
• Use the right testing tool