Practical Hypothesis Building: The Way the Experts Do It

What is a Hypothesis?

A proposed solution to a problem, whose validity requires evaluation.

Why is a Hypothesis Critical to Your Optimization Program?

The innate desire for improvement is what drives an optimization program, but testing just for the sake of testing is a setup for failure. An experiment needs a primary goal that determines success.

Formulating a hypothesis forces us to think about what that goal is, how we should try to achieve it, and what results to look for. A good hypothesis provides structure.

A poor hypothesis creates confusion; garbage in, garbage out. The hypothesis is the foundation of an experiment, it contains all of the information necessary for building a split test.

How do you Construct a Split Test Hypothesis?

There are three main components of a split test hypothesis: Comprehension, Response, and Outcome.

To make components easy to remember, they follow the CRO acronym that all of us are very familiar with.

Each component is comprised of two to four short written snippets describing data collected via research, design, and measurement. Assemble the snippets together into sentences and you’ve built a solid hypothesis. Here’s an example, with each data snippet marked in bold:

• Comprehension: We have observed a reduction in multi-item purchases by comparing the last 6 months of purchase
  data.

• Response: We want to promote paired products with an inline upsell at the cart page on mobile phones for returning
  users with an item already in their cart.

• Outcome: This should lead to single item purchasers more easily finding and purchasing complementary products, which will be measured by average order value (AOV) and backed up by average order size, multi-item purchase count, order conversion, and revenue.

You can see how each snippet is a description of an important piece of data that was either collected from research, proposed via design, or outlines measurement criteria.

Assembled together, they read as a complete hypothesis. Now, let’s dive into each component.

Comprehension

Comprehension comes from research; it is identifying and understanding a problem you want to capitalize on. For example: “90% of users are abandoning from the signup form after the 3rd input.” Comprehension is composed of two snippets: the observation, and the observation method.

Observation

To identify a problem, first you must observe it. Dig into some data and get your hands dirty. Find something about your site that is not performing as well as you want it to. The two main avenues for research are analytics and feedback. Analytics is numeric, quantitative data on a large scale; typically numeric, statistical data that’s passively collected for later aggregation. Feedback is descriptive, anecdotal data that’s actively collected and representative of user behavior.

Observation Method

The tool, technique, or process you used to identify the problem. Incorporating the observation method into the hypothesis helps prove that you did your research. Examples of observation methods for each type of research include:

Analytics

• Techniques: heatmaps, page funnels, session replay
• Tools: Google Analytics, SessionCam, FullStory

Feedback

• Techniques: surveys, usability testing, product reviews
• Tools: Hotjar, Qualaroo, TurnTo

Response

Response comes from design; it is proposing and targeting the change you want to make. Often, this can be approached by starting with a question, like “what will happen if we increase the size of product images in the category grid display?”

The response is the “meat and potatoes” of your experiment; the core that will be configured in the split testing software. The snippets that form a response are: variation, location, and audience.

Variation

What is being proposed as an improvement. This is the theoretical part of an experiment. Whatever change you think will outperform the current state of the site, big or small: stronger call-to-action text on a button, larger images, reduction of form inputs, redesigned header navigation. You get the picture; a short description of your idea for outperforming the baseline.

Location

Where the experiment will take place on your website; types of pages, URL targeting, device scope and/or screen breakpoints.

Audience

Who should see the experiment; specific details regarding user’s status in relation to providing meaningful behavioral feedback.

The snippets of a response are how you scope an experiment and fine tune the signal-to-noise ratio of the result data.

For example, if the variation is changing the position of the estimated order total of an eCom site, then the targeting might be the cart page and then the audience would only be users who have an item in their cart. You would not want to collect data from users that interact with an empty cart page, since they wouldn’t see the UI elements that display a cart total anyway.

Looking for an A/B testing tool that lets you get very sophisticated with your targeting and audience building? Check out what Convert Experiences is capable of.

Outcome

Outcome comes from measurement; it is how success is defined and evaluated. The snippets of an outcome are: desired behavioral change, the primary KPI, and secondary KPIs.

Desired behavioral change

The change in user behavior that you want to be caused by the response; either encouraging a positive action or discouraging a negative action.

Primary KPI

The metric that determines an experiment’s success or failure. Measurement of a user action that the experiment is trying to affect. An experiment’s raison d’être.

Secondary KPIs

Metrics that back up the Primary KPI. These help you flesh out the story being told by the primary KPI, and/or check for unexpected user behavior. Secondary KPIs are often conversions at each stage of the funnel between the targeted area and primary KPI endpoint.

For example, if your primary KPI is revenue from orders and the experiment targets a product page, then the secondary KPIs would be add-to-carts, cart pageviews, checkout pageviews, and order conversions.

Hypothesis Builder

Over at Corvus CRO we’re all about building little tools to help make the lives of CRO experts easier. To that end, we’ve constructed a hypothesis builder to help simplify and standardize the construction of split test hypotheses. It is a web form that takes text input for each of the necessary snippets and assembles them into a full hypothesis. You can use the tool to proof out and build your own hypotheses. There’s even a “copy to clipboard” button so you can easily grab your constructed hypothesis and paste it into your project management tool of choice. We hope you get as much use out of the tool as we do!

Collecting each snippet can be a time consuming process, but the payoff is solidly constructed experiments that will provide more overall value in the long run.

An ounce of prevention is worth a pound of cure; a little more preparatory work up front significantly reduces later overhead.