Personas in A/B Testing: A Customer Research Blueprint for High Quality Experimentation

When was the last time you used your buyer persona?

Whether you’re using a buyer, user, or customer persona, you can agree that the most exciting time in its life was its creation.

After that, it gets pasted on a wall in the office or stuck to flashy slides, never to inform marketing efforts or inspire experimentation hypotheses.

But personas can be so much more than semi-fictional depictions of your ideal customer. You can use personas in A/B testing as a powerful customer research tool to trigger a high-quality experimentation feedback loop.

One where research feeds experiments, experiments generate insights, and insights (about your customers, users, or buyers) lead to more robust hypotheses.

Let’s show you how.

Personas: Much Maligned But Useful?

During the early days of software, products were not as user-friendly as they are today. It was so bad back in the 90s that Alan Cooper had to write and publish a provocative take on the matter.

The first edition of his book, “The Inmate Are Running the Asylum” came out in 1999, and in it, Cooper introduced the use of personas for designing digital products.

That was the first use of the term “buyer persona”. And that book earned Alan Cooper recognition as the founding father of personas. The first buyer persona he created was called Kathy, which he used to illustrate a user-friendly interaction design of digital products.

Since then, personas have permeated the customer-centric facets of business. Top of those facets is marketing and communications. Because you need to know your audience (customer or user) to communicate with them effectively.

When you ask marketers in, say, the home services industry, who their ideal customers are, you will hear things like “homeowners aged 35 to 50 living in Palo Alto”.

And if you ask a B2B SaaS marketing chief who their ideal customer profile (ICP) is, their response might be, “CXOs from Enterprise EdTech companies”.

The point of personas is to give people a better understanding of your customers by making them seem like people they know.

So, to fill in those blanks, some marketing teams build semi-fictional profiles of these target customers. Research like customer interviews and surveys provides some ‌details.

Stereotypes of the target demography, data pulled from the organization’s CRM, or just blunt assumptions cover the rest. For example, “a 45-to-55-year-old businessman who reads Forbes every morning.”
Here’s an example of a detailed buyer persona by GoDaddy with a cool narrative:

What’s the Problem With Personas?

Personas have come a long way from when Cooper introduced them in the 90s. But the problem with personas in the early 2010s (and maybe even now) is the fact that they are nothing more than an engaging read — kind of like marketing fiction.

Why?

They’re Mostly Brainstormed Internally

This is often without a word exchanged with real buyers. Although some teams make an effort to interview and survey actual customers, this isn’t the case most of the time.

Instead, we get biases validated and made available for everyone to use as the “truth” just because it’s in a pretty set of slides.

Traditional Personas Perpetuate Stereotypes

Naughty Nancy. Peeved Peter. These stereotypes, by definition, do not explore the real motivations, frustrations, uncertainties, and doubts of buyers. They merely project opinions about certain segments of people onto them.

Amanda does a great job of putting the spotlight on why personas border the ridiculous when created in-house and without user research. But this type of personas is usually the first item you see in many content marketing strategy docs.

There’s an Obsessive Focus on Demographic Data

Demography doesn’t define real people. It also doesn’t define why they buy. Depending on demographic data is the most non-nuanced and rudimentary research possible.

Two individuals of the same age and income bracket may have drastically different tastes and sense of style. And they may face completely different conversion roadblocks on your website.

We asked John Ostrowski (Positive John) to share his take on why marketing teams and businesses have failed to implement personas.

Here’s what he shared:

This is not a how-to, it’s about what it is, what it isn’t, and what went wrong.

Consider how the word “force” was used in the English language for centuries before Sir Isaac Newton defined it mathematically.

Today it is sometimes used interchangeably with terms like “energy” or “power” — but not by physicists and engineers.

When aircraft designers use the term, they know precisely what they mean in a quantitative sense (and those of us who fly frequently appreciate their effort at clarity).

Still, every marketer will tell you a slightly different perspective of what a user persona is or isn’t.

So I will give you yet another one.

Wikipedia definitions to level the playing field

A persona, (also user persona, customer persona, buyer persona) in user-centered design and marketing is a fictional character created to represent a user type that might use a site, brand, or product in a similar way. 

Marketers may use personas together with market segmentation, where the qualitative personas are constructed to be representative of specific segments.

Personas are useful in considering the goals, desires, and limitations of brand buyers and users in order to help to guide decisions about a service, product or interaction space such as features, interactions, and visual design of a website — This is where the problem begins 👀

In most cases, personas are synthesized from data collected from interviews with users.

They are captured in 1–2-page descriptions that include behavioral patterns, goals, skills, and attitudes, with a few fictional personal details to make the persona a realistic character.

This is what comes to mind when I hear the word “persona”:

Fluff.

So where did marketing-created buyer personas go so wrong?

The way I understand this question is, why do marketing-created personas get such a bad rep?

At its core, a persona is a piece of evidence, a measurement that you get out of a research process of user interviews and segmentation.

It went wrong when agencies overpromised the outcome and delivered another slide deck to secure their retainer.

If product leaders can’t identify which decision will be affected by coming up with personas, it has no value.

The reason why it’s common for personas to be shelved.

I’d like to share Jeremy Epperson’s perspective on this. Despite coming from an agency perspective, he seems to share my vision of where personas went wrong.

When asked about User Personas and how to use CRO to have a persona strategy that stands the test of time, here’s what he had to say:

I don’t use the term persona anymore.

Like I personally will not say that term and I don’t let others say it around me.

[…]

We pay a bunch of money, we create the persona, and then it gets shelved.

And it’s never activated in testing, right? We never challenge the assumption.

This is like research for research’s sake.

Right back to that point is like okay, we’ve conducted research, but it hasn’t changed how we test it. Hasn’t changed how we market it, hasn’t changed our positioning.

So personas are like stacking a bunch of attributes that you really don’t segment or optimize on.

But can personas be useful? Consider these stats about buyer personas:

• According to Mark W. Schaefer, 90% of a company’s sales usually come from 3 to 4 of their buyer personas.
• One MarketingSherpa case study showed how personas dramatically boosted digital marketing results, increasing length of visit by 900% and marketing-generated revenue by 171%.
• Another case study revealed buyer personas led to a 97% increase in lead generation and a 55% increase in website traffic from organic search.
• Email marketing campaigns using buyer personas experienced 2X the open rate and 5X the click-through rate as those without them.
• Personas inform personalized emails that boost conversion rates by 10%.

So personas don’t have to disappear. They simply need a revamp.
We can update them in real time with insights collected from scientifically conducted and relatively unbiased research (pre-test). And post-test, we can update them with learnings from the test results that focus on customer experience optimization (CXO).

The Jobs to be Done Framework: Resuscitating Tired Old Personas?

The Jobs-to-be-Done (JTBD) framework comes from product development. It is an approach where the focus of product design and development is on the “job” customers would want to “hire” your product to complete.

That means the design or development of the product isn’t about the product itself; it’s about the customers’ motivations for getting the product.

For example, you won’t buy a toothbrush because you want to own one. You’ll buy it because you want to maintain good dental hygiene.

Similarly, you don’t use Google Analytics because it’s what all the cool marketers are using. You use it because you want to understand what website visitors are doing on your website and how well your marketing campaigns are performing.

So, how does this compare to personas? Are they mutually exclusive? Can they be reconciled? 

Here’s John Ostrowski’s take on Personas vs. Jobs to be Done:

---

They’re not mutually exclusive because they are tools to solve different things, as Nielsen Norman suggests.

Have Jobs to be Done made personas obsolete?

Absolutely not.

So product teams will do user research and write up a list of these jobs to be done, in simple, clear language, so they have them as a reference for what they should build.

The thinking goes: if you can identify that real underlying need, that root cause, you can be better equipped to design something that helps satisfy the need better than you could if you were just focusing on features, demographics, or specific types of users.

Then how are they different?

Jobs to be Done explains the situations and motivations for customers to “hire” your product to solve particular problems. Straightforward and user-centric way to think in terms of features.

Personas identify distinct groups that hire your product in different ways that correlate to their profile and demographic (e.g. income level, geo-location, gender, age, profession), needs, and goals. Good personas foster empathy. Personas build audiences.

#reflection: I tend to believe Jobs to be Done is a tool better suited for product teams solving for conversion and retention while personas are geared toward marketers solving for acquisition.

---

Jobs to be Done can and should be part of Persona 2.0. A document constantly updated with fresh insights and information covering all stages of buyer’s interactions — from acquisition to retention to expansion.

Sort of like a living 360-degree depiction of your customers.

Here’s an example.

Although he isn’t directly talking about personas or JTBD, Paul Randall’s post below shows how Jobs to be Done can mingle and meld with personas. Specifically, where he suggests grouping stages by “doing” words instead of arbitrary sentiments or pain points.

This is a hat-tip to the JTBD framework. The core objectives and motivations and their granular sentiments come from traditional, qualitative persona-centric research. He calls this hybrid the Experience Map.

This framework can give marketers the language to display the subjective superiority of their product and position it as a better choice for simplifying day-to-day tasks.

Lorenzo Carreri explains how Teamflow did it on their pricing page:

Meanwhile, product and success teams‌ can leverage Jobs to be Done from the perspective of direct interactions with the tool or the app — to get customers to first value, ensure consistent value, and place well-timed upgrade and expansion nudges.

Another practical example that connects the dots between personas and the JTBD framework (but in the context of product teams) comes from Reforge. It focuses on the pathway from the customer’s initial state to the desired outcome.

In the middle of this pathway are the job map and job stories. The job map shows how customers make their way towards achieving their goals, while job stories frame the problem and the individual steps to solve them.

This flow tells the story of the journey the target customer goes through from their initial state (with their motivations and uncertainties) to the outcome they want. Plus all the checkpoints in between.
Doing this lets you know who has the problem (persona) and what they’re trying to do (JTBD). That presents a more in-depth perspective, usable across job roles in your organization, and a much more realistic identity of your customer and why they would want your product or service.

How to Build Research-driven Personas (Persona 2.0)

Keep this in mind: Persona 2.0 is to good ol’ regular personas what high-powered lasers are to regular flashlights. You need that mindset to begin because this is a significant upgrade that requires effort to match.

Check out John Ostrowski’s examples from the wild on crafting Persona 2.0s:

Example of Persona 2.0

You can see Jobs to be Done built right into Gitlab’s existing personas (14 of them) in its handbook (available for anyone to check out).
Note: There are two basic kinds of personas at Gitlab, based on data-driven insights that focus on user needs and emotions.

Buyer personas – Focus on the high-level goals of potential customers who may or may not be potential users. Owned by the Marketing team.

User personas – Used by UX professionals and Product Managers (PM) as a mechanism to connect with end users’ needs, motivations, behaviors, and skills. Owned by Product Managers, who are also the DRI on persona-related research efforts.

Gitlab

The great thing about this example is that you can add new personas or update existing ones whenever you like. As it should be with Persona 2.0 — if you want it to function within the feedback loop that powers high-quality experimentation.

These personas should:

• Be informed by research
• Be driven by job title or feature
• Be gender neutral
• Use bullet points and avoid long narratives

Use the Jobs to be Done framework

The Step-by-Step Guide to Building a Persona 2.0

1. Define targeting

Decide on the criteria you’ll use for picking customers for research. Start with the top 10% of your customers, if possible.

2. Design interview questions

Create a template of interview questions specific to the product. You want to learn about the customer, how they make a purchasing decision, alternatives they considered, how they use the product, etc.

3. Send an email to book interviewees

Reach out to the customers you’ve shortlisted for the research. Invite them for an interview at a time convenient for them.

4. Run recorded interviews

Taking notes while the conversation is going on isn’t optimal. Not only do you need to pay full attention, but you also need to keep a record for reference‌. Request consent to record the interviews.

5. Transcribe interviews with person-based or AI-based software

This significantly cuts down the time to turn your recorded interviews into text so you can easily make sense of your qualitative data. You can try otter.ai or fathom.video for this.

6. Tag transcription to analyze it quantitatively

Highlight key points and themes. Use color coding to make these themes easy to spot and assess. You can move them to a spreadsheet.

7. Design user journey from first thought through awareness stages

Here, you’re mapping the journey a user takes when they’re looking for a solution that leads them to discover your product.

Consider watching Vassilena Valchanova’s video below for clarity on the journey map. You can skip right to 6:49 for the exact process — but I strongly recommend listening to it all as it is a very insightful discussion about Jobs to be Done.

8. Summarize quantitative analysis in a couple of personas

Collect and analyze objective, numerical data from the interviews. What are common quantifiable attributes you can find in the responses? How can you use this to group responders into broad categories that could be personas?

9. Run a workshop with the team to communicate findings

Also, suggest tests based on findings. You can A/B test the personas to find out which one best defines your ideal customer.

If persona A buys your product because it solves problem A and persona B buys for problem B, your control will be messaging that addresses problem A and your challenger will be messaging that addresses problem B. Which performs best?

10. Define the next time the exercise should run again

Because this is a feedback loop that keeps updating the insights and persona doc.

ICP Research for Persona 2.0

There is more than one way to conduct solid research to identify frustrations, uncertainties, and doubts and to determine Jobs to be Done. But there are plenty of wrong ways to go about it.

This is why we recommend the battle-tested and industry-wide admired ResearchXL methodology developed by Speero.

Remember, at the heart of it all, it’s the insights that matter.

1. Motivational Data

Start by collecting motivation data. This data answers the questions:

• What made you search for a solution?
• Why are you willing to pay for this solution or keep paying for it?
• What outcome are you looking for when you purchase our product or service?

This is worded into customer surveys and interviews. While interviews give you more wiggle room to dig deeper into what customers are thinking, surveys are easier for pulling in lots of responses with fewer resources.

Responders word their sentiments differently, but they’re usually talking about the same things. So, you can make sense of this qualitative data by keeping count of the common themes in the responses you get.

Be careful not to fall victim to cognitive bias. That is where you’re focused on feedback that supports what you already believe. To avoid this, you need 2 or more people working independently to analyze the same data set.

For tools, Typeform is a great option for conducting online surveys that you can send out to customers. You can use Google Forms too. If you’d rather present your surveys as on-site pop-up polls on your landing pages, or other pages on your website, you can use HotJar or Qualtrics.

2. FUDs (Fears, Uncertainties, and Doubts)

While motivation moves people to buy, FUDs are the psychological friction affecting that motion. Too much friction and you’ve lost a buyer. So, it not only makes perfect sense, but it’s also vital to know what’s causing objections in the mind of your customers.

You can gather this info through exit intent polls. But the challenge with this is that when people are already experiencing fears, uncertainties, and doubts, they’re not excited to respond to polls.

A simple yes/no question can help overcome that challenge. Then once they’ve committed to a response, follow up for an explanation to get more context.

Here’s how Emma Travis would go about it:

You can also use an online chatbot for this.

You can ask yes or no questions such as: 

• “Is there anything holding you back from buying?” or 
• “Do you have ‌questions you haven’t been able to find answers to?”

Good open-ended questions would be: 

• “What is holding you back from making a purchase?” or 
• “Why didn’t you complete a purchase today?”

Sometimes they may need more information than what’s available on your site. You can set that up to notify your customer success team to reply with additional info.

If you sell software, customers might want to see a case study that they can relate to.

Try different questions to see which elicits the most responses. Your next iteration can always be improved with a new version of this research.

3. Behavioral Data

Here’s the first quantitative data in our persona 2.0 research. Behavioral data shows how users engage with your website. It is stored as an “event” when users perform an action and the event is described with “properties” (metadata).

Think about analytics data that report user actions such as page views, sign-ups, clicks, mouse movement, etc.

When you analyze this data, you’re trying to understand the “what” and “how” to give context to the “why” you’ve learned so far.

4. Friction Data

These are the difficulties users experience when engaging with your websites or products. You’d want to conduct usability research to fish them out. They’re tainting your user experience.

To get unbiased and accurate results, it’s best to use a representative group of your target audience. You can gain a variety of perspectives by doing this, and you can find out what your product’s strengths and weaknesses are.

While you collect data about usability issues and prioritize them for fixes, also collect data about the feeling users associate with elements in your product. For example, are you naming your menu items correctly? Is there a difference between “apparel” and “clothing” for them? Or “blog” vs “articles”?

It could be that despite users being able to accomplish what they want on your site, their general feeling towards it might be negative or different from what you had intended.

Here is how John Ostrowski approaches ICP research:

As per the decision-first approach to research, selecting research methods is a prioritization step in itself.

Let’s take a step back and exercise first-principles thinking.

Based on our exploration, what are we trying to do and how will this inform different testing decisions?

For the teams I worked with, we are mostly trying to Identify and Understand problems users have so we can generate specific testing hypotheses.
Using Behzod Sirjani‘s cheat sheet from Reforge User Insights for Product Decisions, interviews seem to be our best call.

I believe it prioritizes information depth instead of breadth.

So you’re telling me that interviewing 8 to 10 people is enough? Really?

As you can see, surveys are your second best option, and this is where quant data will help you refine your findings for an even stronger use case.

Do you have time and resources for running all at once? Is the juice worth the squeeze?

If I can only choose one, I start building my qualitative use case with interviews.

How Can Persona 2.0 Fuel & Power A/B Tests?

Here are some tips from Jon Crowder, Head of Website Experience at Journey Further, to use Persona 2.0 to power A/B testing:

You may already be using the “Jobs to be Done” (JTBD) framework documented and proposed by Clayton Christensen. It’s a logical way to solve problems in product design. It works on the principle that your customers are trying to achieve a specific goal with their visit (and potentially other related, but less-important, goals).

It is a design process that encourages the designer to acknowledge that when a user shops for a vacuum cleaner, their primary reason and driving force for doing so is that they want to clean their floor. It starts with that as your primary design feature and then builds detail on that concept. Some of your users will be pet owners, looking to specifically meet the challenge of removing pet hair. Some of your users will be seeking to make the process of vacuuming easier and may respond positively to cordless/lighter options. Some of your users will need a vacuum cleaner suitable for the car. Some will have more pressing and urgent needs and need a vacuum as soon as possible and so delivery and your supply chain are more important. You can go even deeper and seek to understand the motivation behind the desire for a clean floor, in order to better speak to your user’s driving needs.

This process refers equally to services as it does to products and is a way of directing design thinking towards the user and their immediate needs.

Where it feeds into experimentation is that it’s useful for creating more relevant hypotheses for AB testing. If you understand the different ‘jobs’ your users are trying to do, you can attempt to tackle those jobs with your design.

At Journey Further we start every journey with your data. In order to understand what goals your customer is trying to achieve, understanding that data is essential. We conduct research to understand your users and how they interact with your website and your product, and then use that data to form hypotheses which can be tested and proven, in order to deliver a game-changing experience.

Doing this naturally means understanding and recognizing the JTBD framework, as part of what we’re trying to understand is what brought a user to the product, what motivates them to act, and what differentiates (or does not differentiate) your product from that of your competitors.

We must also acknowledge that your customers are not a homogeneous mass acting with a single mind and motivation, they are composed of many individuals with different needs and motivations. One customer who purchases your product may have entirely different

motivations from another. The more utilitarian the product, the more diverse the motivations can become. Extending this to its natural end, if you are in the business of selling raw materials, your customers’ motivations may be part of a large expanse and your website acts as a static marketplace where those specific motivations are rarely heard. If you are willing to identify those motivations and to speak to them specifically, you will stand head and shoulders above the competition, who are unable to experiment this way.

After the early broad-spectrum research, it is usually possible to identify a loose collection of ‘personas’ and their specific driving motivations. These typically differ from standard marketing personas in that they’re only focused on relevant goals and outcomes, rather than demographic metrics such as age or sex. To extend the vacuum cleaner example, we are able to identify “pet owner” and “hygiene-focused” users. We can see “convenience-driven” and “longevity-focused” users. Each of these personas has individual and overlapping motivations which can be experimented with in messaging and positioning on a webpage. Those experiments can then validate for us… Did we hit the mark or were we off-base? What matters most to your users? We know what their goals are, but what is the best way to speak to that goal?

Experimentation holds the key.

Close the Loop: Add Experimentation Insights Back to your Persona 2.0

Your feedback loop cycles back with experimentation, specifically, the insights you pull from experiments that improve the persona you had at the start.

Customer experience optimization (CXO), which is basically experimentation with the focus on understanding customer behavior, is the core discipline for this purpose. It covers everything we’ve discussed so far, including

• Customer research

Helps validate and/or remove assumptions about your target audience and ideal customers by learning about their motivations and objections through surveys and interviews.

• Qualitative research

Via mouse tracking and heat map analysis gives you a crystal-clear picture of how users engage with your site — much more accurate than personal opinions about what’s happening.

• Social listening

Provides insights that give you a broader sense of what’s happening in your industry, with your brand, product, and your competitors, and how your target audience articulates their problems.

• Usability research

How satisfied are users with your product? What’s the user experience (UX) like? How effectively can your product help them complete the “job” it was “hired” for? Usability research helps you answer these questions, and spot good elements to improve plus bad ones to fix.

CXO also includes cohort analysis, card sorting and tree testing, and A/B testing personas.

The overarching goal here is to understand the initial state of your customer before they search for a solution, how they arrive at that solution, and their motivations or fears along the way.

To measure these and add valuable insights to your persona doc, you’d want to stay aligned with customer metrics (not revenue metrics) such as engagement depth scores, UX quality scores, referral rates, sharing rates, speed, etc.

In the end, a Persona 2.0 document only has value if it’s put to use. To get everyone in your organization on board with the creation process, you have to sell its merits.

This isn’t just a regular semi-fictional description of your ideal customers. It’s a constantly-updated document that focuses on jobs potential buyers wish to complete and thus want to hire your product, tool, or service for. The insights collected from experiments you run based on your JTBD research further flesh out and mature the document.

For it to stay updated, testers have to keep feeding it insights from experiments.

Allow everyone to view the document, but appoint someone to update it. That could be you if you’re the one with the strongest understanding of experimentation and/or user experience.

Then, make it a part of your experimentation learning repository. If you don’t have one, the idea is to keep a centralized real-time document where experimentation teams can jot down data about ICPs of note.

You can use Airtable or Notion as your centralized location for this purpose. Some teams even use Google Slides.

Ensure insights are properly logged so the persona document can be regularly updated and used to inform decisions in your organization.

John Ostrowski says this responsibility also extends to disseminating insights:

Within a cross-functional product team, UX is the voice of the customer — I love this concept.

I’ve seen that working wonders for product development while working at Brainly with a Sr. UX with eight-year tenure in the business.

Depending on how the “testing team” is structured, having JTBD for reference is an input, meaning it’s managed by a UX professional.

Where does that professional sit? That’s an org chart discussion.

In my experience, working close to product teams organized following the Spotify Model (love it or hate it), jobs to be done is maintained by Product Managers.

Now, if there’s space for Product Managers to co-exist with Experimenters, that’s a discussion I’m still digesting.

As of today, I believe that CPOs/COOs benefit from a leaner organization having PMs capable of running their own experiments. When platform technicalities get tricky or statistics get ugly the Center of Excellence is there for the rescue.

If I’m hired tomorrow as Experimentation Director for your business, that’s part of the vision I’d advocate and execute.

Conclusion

Persona 2.0 operates in a feedback loop that gains from experiments and, in turn, feeds more robust experiments. This symbiotic relationship starts with more actionable persona docs boosted by Jobs to be Done insights (thanks to solid ICP research).

You have to rethink personas and how they’re created to create this version that becomes part of your experimentation program.