Subscribe now

P.S. Paid subscribers get access to full archive, all content, a private Slack community, Substack lives, and a hub of templates, scripts, and mini-courses

“But it gives us numbers.”

“No thanks.”

That’s the first thing I said when faced with the suggestion of conducting a quantitative usability test.

I was more than happy with my conversation-filled qualitative usability tests. Asking people about their immediate reactions, how they perceived the screens, and their general thoughts on what was confusing and missing. It was a routine I quickly felt comfortable in and enjoyed.

But there’s always a but, isn’t there?

It came when I was sitting with my team, and we were chatting about a flow that many of our users were having trouble with. I had triangulated some data from previous research where the topic had come up, customer support tickets, and also from account managers.

I told my team that we had enough to understand most of the pain points, especially the important ones, and make changes. I was at a point in my career and at the organization where I was privileged enough to be able to say no to user research requests.

Luckily (at the time, it felt unfortunate, but for my career, it was a good thing), my manager was sitting in the meeting. He’s one of the most fantastic managers I’ve ever had (Hi, John 👋🏻), and he asked me a smoldering question:

“How will we know if the changes we make improve the usability?”

Of course, John already knew the answer to his question, but he directed it to me. I knew what he was going for, but I was terrified of the answer.

All I wanted to do was make the changes and then ask the users if the changes we made were helpful — I was even willing to sift through the customer support tickets for the next few months to see if complaints decreased. Anything to stay away from the numbers.

But my team was beaming. This was exactly what they wanted: a clear and straightforward way to measure usability and progress. There was no backing out. It was finally time for me to conduct quantitative usability tests.

And I am so glad for that push because they have become an absolutely essential part of my user research toolkit and have also helped me become a well-rounded (and promoted!) user researcher.

What is quantitative usability testing?

Usability tests, on a whole, are about having participants attempt to do the most common and important tasks on a product/service. While you conduct the test, you, as a researcher, are looking to find problems the participant runs into during the test. You then take these problems to your team and, together, brainstorm and find ways to fix the usability issues — which are sometimes simple and other times complex.

With qualitative usability tests, you are talking to the participants and describing the different reactions, perceptions, or issues they encounter.

However, with a quantitative usability test, you can still describe the problem, but you measure:

• How many people encountered a problem

• How many people were able to complete the tasks

• The time it took them to complete tasks

• How many errors participants ran into

• What types of errors participants encountered

• Participants’ perceptions of usability

With quantitative usability testing, you can find out a lot of important information that can help you generate the impact of your research. For instance, when I was working at a travel company, we conducted a quantitative usability test on our checkout flow.

We found that people were taking a long time to fill out information that ultimately wasn’t that relevant and, thus, dropping off and abandoning the flow for a competitor that was easier to use.

Based on these results, we made some significant changes and retested the flow after the improvements were made. We reduced the time it took to fill out information by 50% (which was faster than people could do on the competitive product as well), and we reduced abandonment by 35%. This meant that we increased revenue by £75,000 annually.

Big impact.

When it comes to measuring usability, we can break that down into three major areas:

1. Effectiveness: Whether a user can accurately complete tasks and an overarching goal

2. Efficiency: How much effort and time it takes for the user to complete tasks and an overarching goal accurately

3. Satisfaction: How comfortable and satisfied a user is with completing the tasks and goal

Below, I walk you through:

• How exactly to run a quantitative usability test for your team (with examples from my work)

• The most important metrics for you to know

• How to write a quantitative usability tasks/scenarios that give just enough detail to your participants

• How to analyze the results for tangible benefit to your team

Exclusively for paid subscribers

Read more

Subscribe now

P.S. Paid subscribers get access to full archive, all content, a private Slack community, Substack lives, and a hub of templates, scripts, and mini-courses

For me, quantitative usability testing was always super straightforward. I put a high-fidelity design or live product in front of someone and asked them to do certain tasks, which I then measured through metrics like task success, time on task, and surveys like the Single Ease Questionnaire.

There was very little room for asking qualitative-based questions or for introducing bias. We were there to truly understand the effectiveness, efficiency, and satisfaction of what we put in front of the participants. Straightforward. Easy, dare I say. In fact, I could even set up an unmoderated test to get even more participants.

However, I felt uncomfortable when it came to qualitative usability testing. I never seemed to be able to strike the right balance and constantly felt like I was asking leading and biased questions. I hated the standard questions like:

• “What would you expect to see?”

• “What do you think of this?”

• “What would you change?”

• “Explore the interface and tell me what you would do.”

I hated those questions because they were so hypothetical and future-based. I felt like I was asking the participant to develop ideas and design the website or app. The data I got from those questions was skewed and unhelpful.

Very rarely, if ever, as a user, do I sit on a website and think, “What am I expecting to see?” I can’t remember the last time I went to a website or app to explore the interface. And, although sometimes I do have opinions on websites/apps, my opinions likely wouldn’t be helpful or actionable to teams trying to make changes.

“This is dumb” is not a very actionable quote.

Below, I walk you through the full approach to make qualitative usability testing stop producing fluffy, hypothetical feedback and start giving your team clear direction on what to fix, what’s missing, and what’s confusing:

• The “what qualitative usability testing really is” definition (why it’s not true usability testing, and where it sits between concept testing and quant usability testing)

• The goal checklist that tells you when this method fits (the exact types of decisions and uncertainty it’s built for)

• The TEDW-based question builder (how to replace “what do you think?” with prompts that pull real experiences, perceptions, and friction without leading)

• The session structure you can reuse (warm-up, overarching scenario, screen-by-screen flow, and wrap-up questions that don’t turn into opinions-only noise)

• The synthesis + activation workflow (deductive tags, affinity by screen, clustering patterns, and the built-in handoff into an ideation workshop so the work actually moves)

Exclusively for paid subscribers

Read more

I’ve spent the past few months watching synthetic research show up everywhere. Conference talks. LinkedIn posts. Case studies. Every time I saw it, I had the same reaction. This is going to be a disaster.

Not because synthetic research is inherently bad, but because I know what happens when a new tool promises efficiency. Someone uses it to skip the real work. Someone points to synthetic data and says “we don’t need to do user research anymore!”

I’ve cleaned up those messes before. I’ve been in the room when analytics replaced interviews, when surveys replaced conversations, when speed became more important than understanding. Every single time, the promise was “this will make research faster and cheaper.”

Every single time, we lost something that mattered.

So when synthetic research started gaining traction, I braced for the worst.

Then I looked around and realized this isn’t a future problem. This is already happening. 73% of market researchers have already tried synthetic responses. A third are using it regularly. Companies like Booking.com and Dollar Shave Club aren’t experimenting with it, they’re building it into their research practice.

The market moved without me.

And I had to ask myself: am I avoiding this because it’s actually bad, or am I avoiding it because I’m scared of what it means?

From Panic to Openness

Qualtrics asked if I’d be willing to test their synthetic audiences feature and write about the experience. My first reaction was panic. Was I going to try something like this? And how would it go?

Through the anxiety came another thought. How will I ever know if I don’t experience this?

I’ve been that person before. I was deeply suspicious of surveys when I was a qualitative purist. I thought unmoderated testing would never work. I resisted Jobs to be Done because I worried it would replace other frameworks I valued. Every time, my fear was the same, that this new thing will make my skills less valuable. Every time, I pushed these methods away and, ultimately, my career suffered.

I eventually tried the above methodologies and learned that they don’t replace what I do. They just give me a different way to approach certain problems.

Maybe synthetic research was the same.

So I said yes.

What I Needed to Understand First

Before I ran anything, I had to understand what I was actually testing and what synthetic really was because, trust me, there are a lot of different definitions out there.

Not all synthetic is AI randomly generating responses based on what it thinks humans might say. That’s what generic LLMs do when you ask them to “pretend to be a user.” Those fall apart fast. The responses are too agreeable, they lack demographic variation, they miss all the nuance that shows up in real research. It’s like having someone pat you on the back and tell you, “yes, that is the best product idea ever!”

I found that Qualtrics works differently. It’s trained on millions of actual human survey responses collected over 25+ years of research. When you ask it a question, it’s not guessing or fabricating. It’s looking at patterns across massive amounts of real human data and predicting what humans would likely say to that question.

This kind of synthetic data is model-generated data that’s built to mimic how a real audience might respond. Think of it as a way to explore likely patterns without running a full study with live participants. The key idea here is that mimicry isn’t the same thing as replication. In research, replication is tied to real people, real sessions, and repeating a study to see if the outcome holds. Synthetic data isn’t trying to do that. It isn’t claiming to recreate lived context or match what real participants would say word-for-word. It’s built to give you a fast, pattern-based preview of how a defined audience is likely to react, so you can learn, compare options, and move with more confidence before you invest in full fieldwork.

That difference matters. One is AI making stuff up. The other is AI drawing on what humans actually said when asked similar questions.

Source

I still went in skeptical, but at least I understood I wasn’t testing random text generation. I was testing whether a model trained on real research data could show me something useful before I talked to a single person.

The Research Question

I picked a research question that felt like appropriate territory for testing synthetic: how do people make food decisions when they’re tired, stressed, and low on money?

This question type matched what synthetic is actually designed to handle. Synthetic performs well with perceptions, preferences, and intent-based questions.

As of right now, it doesn’t perform as well with past behaviors, detailed recall, and awareness questions. So I avoided anything like:

• “Tell me about the last time you ordered takeout”

• “Walk me through your typical dinner routine”

• “How often do you meal prep?”

Those questions need memory. They need a biography. They’re asking people to reconstruct events and explain what happened. That’s not what synthetic is for. And, for me, that actually allowed a sigh of relief. Those questions above are the bread and butter of my deep, qualitative research and they aren’t what I would be asking a synthetic panel. I could save those for human-based research.

Instead, I focused on the mental state that shapes decisions:

• What mindsets do people bring to food choices when they’re under pressure?

• What kinds of friction stop people from deciding at all?

• What does “good enough” mean when you’re exhausted and broke?

• What would actually help in those moments?

These questions are about recognizing patterns, not recalling specific events. They focus on mental shortcuts and trade-offs.

WIth this, I built a 13-question survey. Every question had clear scenario framing, a specific cognitive task, and distinct answer choices that felt similar to those I’d create in a standard survey.

I kept the screening broad, just “are you involved in household food decisions?” You can screen with synthetic, you just can’t go hyper-niche, just like in real research. Screening for moms, millennials, pet owners, households of 3+, specific income brackets? Fine. Screening for “left-handed surgeons from Omaha?” That’s not happening. It’s the same as human research, the more niche you go, the harder it gets.

I launched the survey on a Wednesday morning.

By noon, I had 500 responses. How’s that for speed?

What The Data Actually Showed Me

I was met with a dashboard of data which, as a qualitative researcher, can sometimes feel overwhelming, but I took a deep breath and scrolled through the results.

Each question was broken down in several ways:

I really wanted to read the results as I would a traditional survey that I sent to real humans to fill out. The force was real with this. But, I took a step back first and really thought about how I would analyze these results. I took the first question: Now imagine it’s the end of a long day. You’re hungry, tired, and don’t feel like cooking, but you also have a limited budget this week. Which option becomes more important in your decision?

When I look at a synthetic-based chart like this, the goal isn’t to pretend I’m staring at “real behavior.” I was looking at a pattern the model thinks is likely when people are tired, hungry, short on time, and trying not to overspend. It’s a stress-test scenario. The value is in seeing how priorities reorder when the pressure dials up.

This is how I read the chart:

“Choosing something filling, even if not healthy” gets the strongest pull. In a depleted moment, people want to feel fed. Satiety beats ideals, beats convenience, beats health. It becomes the anchor. If I were running real sessions, this is where I’d expect the richer stories on what “filling” means, where they go, what they reach for, which shortcuts they trust.

“Sticking to something low-cost” sits right behind fullness. Tired or not, budget pressure stays in the picture. This gave me a directional hint that cost is a constraint people keep even when their energy is shot. In live research, I’d probe how they manage that tension. What counts as “low cost” in their mind? What crosses the line?

“Using what I already have” lands in the middle. It tracks with what real people say, that the intention is there, and it matters, but it’s easier to abandon when they’re drained. Synthetic data is giving me the shape of that trade-off, not the emotional story behind it.

“Getting something fast, even if it costs more” was lower that I expected. With a tight budget in the prompt, synthetic response showed a prioritization of price and fullness over speed. That doesn’t mean speed stops mattering for real people. It just means, in this hypothetical, it’s not the dominant driver.

“Skipping a meal to save money” sits at the bottom, but it’s not zero. That’s a cue for segmentation, not a headline. In real research, this is where I’d ask one of those beautiful, open-ended questions from above, “Walk me through the moment you decide not to eat. What’s happening right before that choice?”

“Not much changes for me” shows up in every study, real or synthetic. It’s the group with strong routines or fixed habits. They stick to their script regardless of energy or budget shifts.

Here are some other ideas I discovered while looking through the results.

The Cost-Versus-Speed Tension

One question asked: “When you’re low on time or energy, which of these factors usually has the biggest impact on your food choice?”

The results came back nearly split: “Cost” at 143 responses (29%) and “Speed” at 135 responses (27%).

Synthetic responses showed me that when people are tired and resource-constrained, cost and speed genuinely compete as decision drivers in human behavior. Neither factor dominates cleanly in the model’s prediction because neither factor dominates cleanly in real human decision-making. Results from the synthetic data said that humans are split on this because it’s a genuine dilemma they navigate.

I might not have seen that tension as clearly if I’d just written the question and jumped straight to interviews. I might have assumed one factor would always win. I might have designed my interview guide assuming cost would be the primary driver and missed the whole speed dimension.

This became a key area to explore in follow-up human research. When does cost win? When does speed win? What makes someone prioritize one over the other? The synthetic data had flagged where genuine decision-making complexity lives before I’d talked to anyone.

The Emotional Overwhelm Discovery

Another question asked: “What would make you completely give up on choosing food and go to bed without eating?”

“Feeling overwhelmed or anxious” came back at 222 responses (44%), which was significantly higher than practical barriers like “not wanting to clean up after” (112 responses, 22%) or “realizing everything costs too much” (101 responses, 20%).

I’d designed that question thinking about practical constraints like money, effort, time. But the model highlighted emotional overwhelm as a much bigger driver of food decision abandonment than practical barriers.

I’d been thinking about this as a resource constraint problem. How do people choose food when money and time are limited? But the model was indicating that this is primarily an emotional regulation problem. People don’t give up because the barriers are too high, but they give up because making one more decision feels unbearable.

When I redesigned my interview guide for follow-up human research, I centered the emotional state instead of practical constraints. I made that the primary lens for exploration rather than treating it as a secondary factor.

The “Good Enough” Complexity

I asked: “Which of these feels most like a ‘good enough’ food decision when you’re tired, busy, and low on money?”

The distribution came back with no clear winner: “something I had already planned” (153 responses, 31%), “anything hot, filling, and settles my hunger” (138 responses, 28%), “something that is comforting or emotionally satisfying” (107 responses, 21%), “something that meets a low-cost threshold” (99 responses, 20%).

“Good enough” doesn’t mean one thing to humans. The model demonstrated that all of these definitions of “good enough” show up with roughly equal frequency, likely because they’re drawing on genuinely different frameworks depending on which resource (physical energy, emotional capacity, money, cognitive bandwidth) is most depleted in that specific moment.

When you’re physically depleted, “good enough” means something that fills you up and settles your hunger. When you’re emotionally exhausted, “good enough” means something that brings comfort. When you’re financially stressed, “good enough” means hitting a cost threshold. When you’re cognitively overloaded, “good enough” means not having to deviate from the plan you already made.

The flat distribution was the model telling me that I’d asked a question that collapses multiple real human patterns into one. When I designed follow-up human research, I split this into separate questions focused on physical needs, emotional needs, financial constraints, and cognitive load. That separation allowed for much clearer exploration of how different resource depletion states drive different definitions of “good enough.”

The synthetic data hadn’t just caught a poorly-worded question. It showed me I was conflating multiple human decision frameworks that needed to be understood separately.

While reviewing the results, something shifted. Not in a “this changes everything” kind of way, but in a “huh…this is actually interesting” kind of way. I started to see where this could fit into my workflow.

What This Actually Gave Me

After two days with the data, I had to ask: was this worth it?

I didn’t get finished, complete insights. I didn’t learn everything about how people make food decisions under stress. I wouldn’t feel comfortable walking into a meeting and presenting findings that would drive our next biggest product decision.

But I also didn’t waste anyone’s time. I didn’t recruit participants for a survey with broken logic. I didn’t schedule interviews based on assumptions that turned out to be wrong. I didn’t build a research plan around questions that collapsed multiple concepts.

Instead, I learned where the complexity was before talking to anyone. The cost-versus-speed split showed me there’s genuine tension in how people prioritize. The overwhelm spike showed me I’d underweighted emotional factors. The flat “good enough” distribution showed me I’d conflated multiple decision frameworks.

I redesigned my research approach with better questions. Instead of asking broadly about “good enough” decisions, I split it into questions about physical needs, emotional needs, and financial constraints. Instead of treating food decision abandonment as a practical problem, I centered on emotional regulation. Instead of assuming one factor would dominate the cost-versus-speed trade-off, I designed my interview guide to explore the tension.

I saved weeks of iteration time. If I’d skipped synthetic and gone straight to human research, I would have recruited eight people for pilot interviews, spent three sessions realizing my questions were off, spent three more sessions trying to figure out what I should have asked, and then recruited twelve more people for a second round with better questions. That’s 20 participants and three to four weeks of calendar time minimum. Synthetic compressed that learning cycle and helped to narrow the scope of a very broad study.

By the time I sat down with real participants, I wasn’t wondering if my questions made sense anymore. I was listening for the nuance synthetic can’t capture and the reasons why people make specific choices, with the emotional context around decisions, and the moments where behavior contradicts stated preferences.

I could focus on listening because I wasn’t worried about structure.

So, yes, to me this felt worth it.

A Note on Data Richness vs. Data Poorness

Data-rich organizations have massive amounts of customer data, years of research, thousands of completed studies. They’re not using synthetic because they have no data. They’re using it to drill deeper into the data they already have.

Data-poor organizations are the opposite. They are startups, small teams, companies entering new markets, researchers exploring completely new problem spaces. They don’t have years of existing research to draw from. They can’t afford to run massive studies. They need directional clarity fast, and they need it cheap.

That’s where synthetic becomes really interesting.

If you’re exploring a new market and you have zero data about how people in that space think, synthetic can give you initial pattern recognition based on what the model has seen across millions of responses in adjacent spaces. It’s not definitive and it’s not finished insight, but it’s enough to help you figure out which questions are worth asking humans and where the complexity might live.

Or take the “water cooler ask” scenario. Your VP drops by and asks, “Hey, do you think our customers would care about this feature?” You don’t have time to spin up a full study. You need directional sense in the next day or two. Synthetic can help you triage whether this is worth a real study, or can we deprioritize this?

Or feature prioritization when you have ten concepts and you need to narrow to three before you invest in validation research. Synthetic can help you screen fast without burning participant goodwill or budget on ideas that won’t make the cut.

Nobody’s replacing human research. They’re using synthetic to figure out where to focus their human research or to move faster with human research when they don’t have the luxury of time.

It’s triage. It’s prioritization. It’s the pre-work that helps you not waste resources on the wrong questions.

The Dance

For me, the most interesting use isn’t synthetic OR human. It’s the dance between them.

I could start with synthetic to map the territory quickly, like to see where tensions live, which concepts need exploration, what questions I’m not asking yet. Then bring in humans to understand the why behind those patterns. But it doesn’t stop there.

Sometimes human research surfaces something unexpected. Instead of just noting it and moving on, I could go back to synthetic to test whether that’s a broad pattern or an edge case specific to the five people I talked to. Then back to humans to dig into the nuance if it looks like a real pattern.

It’s not linear. It’s iterative.

With my food decision study, synthetic showed me the cost-versus-speed tension. I’d bring that to human interviews and discover, let’s say, that the tension resolves differently on weekdays versus weekends. I could go back to synthetic and test that hypothesis across a larger set of responses to see if it holds. Then return to humans with a more refined question about what drives the weekday-weekend difference.

Or the emotional overwhelm finding that synthetic flagged. I’d explore it with humans and learn that it’s specifically decision fatigue, not general stress. Back to synthetic to see if that distinction shows up in how people describe the overwhelm. Then back to humans to understand what actually helps with decision fatigue versus other types of stress.

The dance is what makes it work. Synthetic gives you speed and breadth to test hunches. Humans give you depth and context to understand what the patterns mean. You need both, and you need to know when to use which.

I didn’t see that coming. I thought I’d use synthetic once, pressure-test my questions, and move to humans. What I actually discovered is that the back-and-forth between them is where the real value sits.

What I’d Use This For

After this experience, I know exactly where synthetic fits in my practice.

Pre-research testing: Before I invest time and participant goodwill on a real survey, I can use synthetic to rehearse. Test if my questions make sense. See if different demographic groups might interpret things differently. Catch confusing wording or weak spots before anyone real has to deal with them. It’s like a dress rehearsal before opening night.

Follow-up exploration: After I collect real data and find something unexpected, I can use synthetic to explore “what if” scenarios without having to recruit a whole new round of participants. Test additional questions I forgot to include. Stretch my research budget by running quick follow-ups to probe areas where the human data pointed to something interesting but didn’t give me enough depth.

Concept screening and feature prioritization: When I have ten concepts and need to narrow to three before doing validation research, synthetic can help me screen fast. I’m not using it to make the final decision, but using it to figure out which three are worth the investment of real human research. Same with feature prioritization, I can quickly assess which features seem most valuable before committing resources to full validation.

The water cooler ask: When someone drops by with a question and I need directional sense-making fast, synthetic can help me triage. Is this worth a real study? Should we deprioritize this? Can I give them enough direction to make a next step without spinning up a whole research project?

Data-poor exploration: When I’m exploring a completely new problem space or market where I have zero existing research, synthetic can give me initial pattern recognition to help me figure out which questions are worth asking humans and where complexity might live.

What I absolutely wouldn’t use it for:

• Answering research questions on its own without human validation

• Understanding personal motivation, emotion, or lived context

• Making high-stakes product decisions without any human input

• Past behaviors, memory, or detailed recall questions

• Anything where getting it wrong would hurt people

• Replacing interviews, usability testing, or actual human research

The Fear I Started With

The fear of “is this going to replace me?” was never really about synthetic research. It was about value.

We worry that someone will decide our work isn’t necessary. That someone will find a faster, cheaper way to get “insights” and we’ll be left explaining why depth matters, why context matters, why you can’t replace human understanding with pattern recognition.

Using synthetic didn’t make that fear go away.

But it did clarify something for me. The work that matters, the listening, the probing, the synthesis of human stories into understanding, can’t be automated. A model trained on millions of survey responses can tell me where patterns exist. It can’t tell me why those patterns matter or what they mean for any individual person.

But the prep work like the question testing, the concept narrowing, the structural validation, can be faster. And when you speed up the parts that don’t require human voices, you create more space for the parts that do.

What I’d Tell Another Skeptical Researcher

If you’re reading this because someone asked you to try synthetic and you’re suspicious, I get it. I was there three months ago.

Keep your skepticism. Don’t let anyone tell you synthetic can replace human research. Don’t let anyone pressure you into treating AI-generated responses like they came from real people. But you can try it once with clear boundaries.

Pick a research question that’s attitudinal and forward-looking. Write 8-10 questions with clear scenarios and distinct options. Run synthetic. Look at the data for pattern flags and complexity signals, not finished insights.

Then ask yourself:

• Did this help me design better questions?

• Did it show me where complexity lives?

• Did it save me from wasting participant time on bad design?

• Did it help me figure out where to focus when I bring in humans?

If the answer is yes, you’ve found a tool that can make your human research stronger.

If the answer is no, you’ve confirmed your skepticism with evidence instead of assumptions.

Either way, you’ll know.

I’m still a qualitative researcher. I still need human voices. I still believe depth and context and lived experience are irreplaceable.

I just have one more tool for figuring out the right questions to ask before I involve the people who matter most.

Subscribe now

P.S. Paid subscribers get access to full archive, all content, a private Slack community, Substack lives, and a hub of templates, scripts, and mini-courses

I remember a time when stakeholders started to get excited about user research. It was an interesting switch for me — I went from constantly checking in to identify user research projects within teams to colleagues coming to me with research project ideas in hand.

It👏🏻was👏🏻awesome👏🏻

I felt like research was exploding. I felt like I finally had a say. I felt like I finally had power.

But “with great power comes great responsibility” (Source: Uncle Ben + others).

And I quickly realized that, with these research projects, as exciting as they were, I started to feel extremely overwhelmed by them. It wasn’t necessarily the amount of projects (that would come later) but rather the number of questions people had within each research project.

The first time I encountered this was at my job at a social media management company. One of my stakeholders had an idea in mind for a concept they wanted to test. We had heard several times within previous research that the analytics on our platform were not aligning with users’ expectations and needs. In fact, they fell short in several key areas.

Some of the key pain points highlighted from previous research included:

• We did not provide sufficient engagement analytics for our clients, inhibiting them from making data-driven decisions

• Many clients were asking for manual reports from account managers as our platform isn’t providing sufficient metrics/data that allows them to compare data

• Our current metrics had little context and weren’t very useful/reliable for our clients to make decisions

These were some pretty big flaws in our platform, rendering it an underutilized feature and, ultimately, creating more work for customers and our account managers.

So, with that in mind, my stakeholder came to me with a concept based on this previous research. I was thrilled. Not only had they listened to previous research, but they had used it as a jumping-off point for a concept! Hurrah!

And then I looked at the list of questions this stakeholder had that they wanted answered within the research project:

• Do people understand the concept?

• Do people like the concept?

• Do people perceive our recommendations as trustworthy?

• What types of comparison timelines do people prefer when it comes to analytics?

• What kind of engagement metrics are most important to see?

• How do people perceive the difference between engagement and interaction metrics?

• Can people use the concept?

• Would they like to use the concept to try it out?

• Are people annoyed when they have to open a new window to compare data?

• Is it clear how people navigate through the concept to get a monthly report?

😱😱😱😱😱😱😱

Not only were these a whole lotta questions, but they were also a lot of unideal-for-qualitative-user-research kinds of questions. There was no way we could answer so many questions in a 60-minute concept test, let alone get answers to the majority of these questions.

I went back to the stakeholder, terrified that I would disappoint them. I had just started the research ball rolling, and the last thing I wanted to do was say no to a research project or tell them that I couldn’t answer these types of questions.

Since I was still early on in my career, I had a tough time rewriting and narrowing down the scope of the questions. We went into the concept test with way too many yes/no and preference questions to answer.

This was one of the first projects that had landed on my desk from a stakeholder, and the results were a bit disappointing. Because the small sample sizes within qualitative user research aren’t ideal for answering yes/no questions (all the “do” and “are” questions), I didn’t have much impact.

Saying, “8 out of 12 people understood the concept,” was not powerful.

Similarly, saying “7 out of 12 people liked the concept” did not tell us anything. Many of the stares I got during my report said, “So what?” or “What now?”

I was gutted (my new British slang). That wasn’t the first or the last time I received a research project with a slew of questions that were either impossible to answer with user research or were way too broad in scope.

If you’ve ever been handed a 20-question laundry list and told “can you just test this,” the next part is your escape. Paid subscribers get the full system I use to:

• Turn “do/can/which/are” questions into interview questions that actually produce stories

• Separate what research can answer vs what needs A/B testing, analytics, or a survey

• Cut scope fast without disappointing the stakeholder

• Run a clean prioritization meeting that ends with a short, usable question set

• Use the upgraded spreadsheet (columns, definitions, scoring) to make decisions in real time

• Spot repeated questions across teams and turn them into strategic research themes

• Walk through the full example (your analytics concept test) with a finished sheet you can copy

Exclusively for paid subscribers

Read more

Subscribe now

P.S. Paid subscribers get access to full archive, all content, a private Slack community, Substack lives, and a hub of templates, scripts, and mini-courses

Concept testing can sometimes feel like a mystery. There were quite a few times in my career when teams put an idea in front of me and asked me to test it with users (well, to validate or ask for preference, which we don’t do, of course 😁). And for some time, I felt incredibly stuck when my teams made these requests.

The ideas weren’t solid enough to conduct a usability test.

They weren’t basic enough for me to conduct generative research.

These concepts were in the in-between (or the upside-down, if you will 😂).

Whenever a team came to me with these concepts, I cringed. I had no idea how to get them the information they needed without leading the participants and asking biased questions. I knew I shouldn’t be asking things like:

• “Do you like this idea?”

• “Would you use something like this?”

• “How would you make this better?”

• “Is this going in the right direction?”

But I wasn’t sure how else to engage with participants.

So, I tried to conduct usability tests on concepts. That ended as a major failure. The ideas were too early to test. Participants got lost and confused because there wasn’t a flow. And, to be honest, I had no idea what I was trying to test. My goals for those research projects were vague, and the findings were unhelpful.

We understood that people had a hard time navigating a loosely defined concept, but we still had no idea whether or not we were heading in the right direction. Each of those reports ended up disappointing not only my teams but also me.

It was by chance that I heard about the concept of concept testing (very meta, I know). At first, I wasn’t sold. How were we meant to evaluate concepts in an unbiased way? And how were we meant to investigate their reactions without relying too heavily on future-based data?

But, after some more disappointing results and failed usability tests on ideas, I finally decided to give concept testing a whirl. Admittedly, I wasn’t very skilled at conducting those tests, but with some practice and guidance, I finally understood the importance of concept testing. And, from there, I never looked back.

What is Concept Testing, Anyway?

Concept testing is one of those elusive methods that, I believe, we don’t discuss nearly enough, as it can be an extremely powerful tool to use early in the discovery process. Because it can be such an “in-between” method, we often skip it, going straight from generative research to usability testing.

However, concept testing definitely has its place in our process. The way I define it is:

Concept testing is a way to engage with participants to more deeply understand a specific problem and their current process through a stimuli (concept). Through concept testing, we gather feedback that allows us to gauge how aligned we are (or not) with participants’ mental models regarding an idea.

Within the scope of this definition, we are looking for immediate reactions and perceptions from our participants. We are looking to see how participants respond to the idea and where there are gaps or confusion about what we’ve put in front of them.

This is the crux of the definition and often where I can see concept testing go wrong (and where I’ve done it incorrectly before).

Where Concept Testing Goes Wrong

As I mentioned, it took me a good amount of practice to hone my concept testing skills. Because it is a less-discussed methodology, I struggled with finding the proper resources on how to conduct a concept test and what exactly I was looking for as an outcome.

Unfortunately, I see concept testing used a lot for things like:

• Product/idea validation

• Preference testing

• A/B testing

• Asking about future-based behavior

When I first started concept testing, I made these mistakes. I wanted the concept test to tell me whether or not participants liked the concept, if they would use it or not, and if I tested multiple concepts, which one they preferred.

The problem with all of the above is that concept testing is still a qualitative method. And with qualitative methods, we can’t answer these types of questions. Qualitative user research isn’t set up for success to answer “whether or not” or “if” or “preference” questions.

Qualitative research involves uncovering reactions, perceptions, feelings, and mental models. Concept testing should be no different.

When I first started conducting concept tests, I asked many of those questions, and the results were more disappointing than the usability tests I had attempted to run on the concepts.

What did it mean if people liked or disliked a concept? What did preference mean when it came to the concepts? How would we know people would actually use the product in the future? Not only that, but usually, during these tests, there can be social desirability bias present, where participants will tell you what you want to hear.

So, when I delivered my results, my teams weren’t always sure what actions to take. We knew that people liked the concept and which they preferred, but there was so little depth to the answers and so little action within the data that the teams ended up feeling just as stuck as before.

From there, I changed how I thought and approached concept tests, ensuring that I got my teams the data they needed without asking participants questions that could skew our decision-making.

If concept testing has ever made you freeze, this next part is the fix.

Paid subscribers get the full concept testing playbook:

• A quick “Should we even run a concept test?” decision checklist

• A goal-setting script to pull the real decision your team is trying to make

• A concept setup guide (what to show, how low-fi to go, what to avoid)

• A question bank built for concept testing (TEDW prompts + follow-up ladders)

• Run-of-show templates for 60 and 90 minutes (1 concept, 2 concepts, 3 concepts)

• Sample screener prompts to recruit people with the real problem, not hypothetical interest

• A synthesis workflow tailored to concept tests (deductive tags + clustering pattern map)

• An ideation workshop plan that turns findings into prototype-ready directions

Exclusively for paid subscribers

Read more

Paid subscribers get the power tools: the UXR Tools Bundle with a full year of four top platforms free, plus all my Substack content, and a bangin’ Slack community where you can ask questions 24/7. Subscribe if you want your work to create change people can feel.

Subscribe now

Hello curious human,

The number of times I’ve had someone come to me and say “let’s just run a survey on this” is almost as many times as I’ve run a survey without enough thought or intention.

Surveys are powerful little tools but, unfortunately, they can be seen as a extremely simple method to get answers quickly to something. And the reason I say unfortunately is because, a lot of the time, we are expecting way too much from surveys (kinda like personas).

Because they appear so simple on the surface, they can get very misused and yield very misguided results. I remember running surveys without clearly understanding what I was doing and interpreted them very incorrectly, leading to some poor decisions being mad.

And it’s not the fault of surveys but how we use them. So, in this article, I will talk through the subjectivity of surveys and how we can use them more appropriately (and also why I’ve started to send fewer surveys in my research projects).

PS. If you want to have a laugh…

The trap of simplicity

Surveys look easy. That’s the problem.

When a stakeholder says, “Let’s just throw it in a survey,” what they really mean is, “I don’t want to slow down and think too hard about this.” And sometimes, if we’re honest, we mean the same thing.

Surveys feel like free money. You spin up a form, blast it to a panel, and within 48 hours you’ve got hundreds of responses neatly laid out in rows and columns. It feels efficient. It feels like research. It feels like progress.

But surveys are the UX equivalent of instant noodles: quick, cheap, and not nearly as nourishing as you pretend they are.

I’ve been guilty of this. I once ran a survey with 20 questions on feature adoption. I asked things like “How often do you use X?” and “How satisfied are you with Y?” I thought I was being thorough. When the responses came in, I did what many of us do: I sorted by the biggest numbers and slapped them into a slide deck.

It looked impressive. Until someone asked me, “So what does this mean for what we build next?”

I froze. I didn’t know. The data was shallow, the questions were vague, and the only real conclusion I could draw was that people clicked boxes when asked.

That’s the trap of simplicity: because surveys are easy to spin up, we stop being critical about whether they’re the right method. We start to believe that any survey = insight. That’s not how this works.

Stop surveys from becoming your lazy default:

1. Start with the question behind the question. Don’t ask, “What should go in the survey?” Ask, “What do we actually need to know, and can a survey realistically tell us that?” If what you need is why, a survey is already the wrong tool.

2. Set a higher bar. Pretend you have to defend the survey in front of your toughest stakeholder. If all you can say is, “Well, it’s fast,” you’re not ready. You should be able to say, “We’re asking these three questions because they will help us decide between option A and option B.”

3. Make surveys the last resort, not the first instinct. Treat surveys like antibiotics: useful when needed, harmful if overprescribed. Ask yourself: could a quick interview, usability test, or even an analytics check answer this faster and better?

A quick self-check template:

Before you send out a single survey link, run it through this filter:

• What is the decision we’re trying to make?

• Can a survey actually give us the evidence to make it?

• What’s the risk if we’re wrong?

• Have we considered another method that might be stronger?

If you can’t answer these, don’t send the survey.

Expecting Surveys to Do the Wrong Job

Surveys often get hired to do jobs they’re simply not qualified for.

I’ve seen teams use them to try to explain churn, validate a new feature idea, or predict future behavior. On the surface, it makes sense: you have a question, you send it out to hundreds of people, and you get a spreadsheet of answers. Fast, neat, and clean.

Except it rarely gives you what you’re actually looking for.

At one company, we were struggling with churn. Customers were leaving at a higher rate than expected, and leadership wanted answers. The first reaction was, “Let’s run a survey asking people why they left.”

So, we did.

Thousands of churned users received the survey, and the top two responses came back as:

• “Too expensive”

• “Didn’t need it anymore”

Leadership latched onto those answers immediately. The conversation turned to pricing experiments. Maybe we should lower costs, try tiering, or roll out discounts.

But then we did follow-up interviews with some of those same customers. And the story changed. Pricing wasn’t really the problem. The real issue was that people didn’t see enough value in the product to justify the price at any level. “Too expensive” was an easy checkbox to click when you didn’t want to type out, “Your product never became essential in my workflow.”

If we had stopped at the survey, the company would have made sweeping pricing changes that wouldn’t have fixed the underlying problem and would probably have hurt revenue even more.

That’s the trap of overpromising: expecting surveys to reveal motivations, predict behavior, or validate desirability. These are jobs surveys cannot do well. They’ll give you surface-level answers, but not the depth you need to make confident decisions.

The mismatch between questions and methods

Some of the most common misuses I see:

• Validating desirability. Asking “Would you use this feature?” is nearly guaranteed to give you a false positive. People like the idea of a feature in theory. Their actual behavior is another story.

• Predicting behavior. “How often will you use this in the future?” No one can predict this accurately. People overestimate their future selves and underestimate friction.

• Explaining motivation. A five-point satisfaction scale tells you nothing about why someone is dissatisfied. It only confirms they are.

A simple way to check if a survey is the right tool

When you’re tempted to launch a survey, pause and ask:

• Do I need counts? (How many, how often, what proportion?)

• Do I need context? (Why, how, under what conditions?)

• Do I need behavior? (What actually happened?)

Here’s the shorthand I use with teams:

• Surveys → Counts

• Interviews → Context

• Analytics → Behavior

It’s not that surveys are “bad.” They just have a very narrow set of jobs they’re good at. The problem comes when we stretch them outside that boundary.

Try this with your own surveys

Think about the last survey your team ran. Open it up, look at the first three questions, and classify them into one of the three buckets above: counts, context, or behavior.

• If the question belongs in counts → good, keep it.

• If it belongs in context → wrong tool. That’s an interview.

• If it belongs in behavior → wrong tool. That’s analytics or logs.

This one exercise helps you spot overpromising quickly. It also gives you a way to push back when someone says, “Let’s just send out a survey.” You’re not saying no to the survey but instead showing the mismatch between the question and the method.

Subjectivity in Design

One of the biggest lies about surveys is that they’re “objective.” Put the same questions in front of hundreds of people, tally up the results, and voilà, you’ve got hard numbers you can trust.

Except every single piece of a survey is subjective.

• The way the question is phrased

• The scale you choose

• The assumptions baked into the wording

• The order you ask things

All of that nudges people toward certain answers. And those nudges add up fast.

Loaded wording

I once reviewed a survey that asked:

“How helpful was Feature X for you?”

Notice the trap? The question assumes Feature X was helpful in the first place. Someone who didn’t find it helpful still has to pick from a scale that presumes it did something good. They’ll either click the lowest option or skip it, but the framing has already tilted the results.

A stronger, less biased version would be:

“What was your experience with Feature X?”

That leaves the door open for positive, negative, or neutral answers.

Scale design traps

Scales look objective, but they carry a lot of hidden bias.

• A five-point scale pushes people to the middle.

• A seven-point scale spreads responses but can be overwhelming.

• A scale that labels only the end points (“Not at all useful” → “Extremely useful”) leaves the middle up to interpretation.

I’ve seen teams argue for hours about whether to use five, seven, or ten points. The truth is: none of them are neutral. You’re shaping the outcome no matter which one you pick. The key is to choose a scale, stick with it over time, and be clear about how you’ll interpret it.

Leading the witness

It’s surprisingly easy to write a survey that accidentally tells people what you want to hear.

Take this question:

“How much do you agree that Feature Y saves you time?”

That’s a leading question. You’ve already told participants that the feature saves time. Even the “disagree” option is reacting to your statement.

A better version:

“If you use Feature Y, how does it impact the time it takes to complete your task?”

Now people can say it saves time, wastes time, or makes no difference. The data is richer, and you haven’t pushed them in a direction.

How to keep yourself honest

Before you send a survey, do a quick pilot run. Ask three to five people (inside or outside your company) to read the questions and then tell you, in their own words, what they think you’re asking.

If what they say back doesn’t match what you intended, your survey is carrying bias. Rewrite until the intention and the interpretation line up.

I use a quick checklist before finalizing any survey:

• Does this question assume a positive or negative experience?

• Is this scale forcing people toward a certain answer?

• Could two different people interpret this question in completely different ways?

• Is there any jargon or insider language that someone outside the company wouldn’t get?

Surveys may look like hard numbers, but every decision we make in design turns the dial a little bit. Pretending surveys are neutral only makes the problem worse. A better approach is to admit that subjectivity exists and manage it intentionally.

Subjectivity in Sampling

Even if you write flawless survey questions, the answers still come from people. And who those people are matters more than anything else.

Surveys often get treated like magic spells: send them out, get hundreds of responses, trust the numbers. But those numbers only reflect the people who happened to answer. And if those people don’t represent the group you’re actually trying to understand, the data doesn’t mean much.

The “easy list” problem

At one company, we ran a survey about feature adoption. We wanted to know how new customers were experiencing onboarding. Sounds simple enough.

Except the survey didn’t go to new customers. It went to the email list we already had set up, the same list that included mostly long-term, power users.

Guess what happened? The responses were glowing. People loved the onboarding. They said it was clear and simple.

But those responses weren’t from the people struggling with onboarding. They were from the people who had already survived it. By surveying the “easy list,” we ended up patting ourselves on the back instead of fixing real problems.

Convenience samples sneak in everywhere

It’s not just email lists. Panels, social media links, and even intercept surveys on your product tend to pull in whoever is easiest to reach. And “easy to reach” often means a very specific type of user: the vocal ones, the loyal ones, the ones with time on their hands.

If you’re not careful, those groups become your de facto research audience. You start designing for the squeaky wheels instead of the quiet majority.

A quick gut check for representativeness

When I look at a survey, I ask myself three questions before I trust the numbers:

1. Who did we want to hear from? Be clear on the audience before you send anything. “All customers” is rarely specific enough. Do you need churned users, new users, enterprise accounts, first-time buyers? Define it.

2. Who actually responded? Look at the demographics, tenure, roles, or usage patterns of your sample. Does it match what you need? Or are you just hearing from the same 20% who always respond?

3. What’s the gap, and how risky is it? If your target was new customers but 80% of respondents were long-time power users, that’s a big gap. If you were aiming for a mix of roles but only designers responded, that’s another red flag.

Turning subjectivity into clarity

You can’t always get the perfect sample. That’s the reality of survey research. But you can at least be explicit about who you actually heard from.

Instead of saying, “Customers told us…” in a readout, frame it as:

“We heard from 250 long-term users who have been with the product for 12+ months. Their perspective is valuable, but we’re still missing newer users.”

That simple shift in framing makes the subjectivity visible. It also makes it harder for stakeholders to run off and overgeneralize the results.

Surveys will always be shaped by who shows up. You have to be honest about the lens you’re looking through, and making sure your team understands the limitations.

Subjectivity in Interpretation

You’ve crafted careful questions. You’ve checked your sample. You’ve got a tidy spreadsheet of results.

Now comes the most subjective part of all: interpreting the data.

Numbers don’t actually speak for themselves. We speak for them. And that’s where things can go sideways fast.

The story we want to hear

At one company, we ran a customer satisfaction survey after a redesign. The scores were decent with most people hovering around neutral with a few leaning positive.

When the PM presented the results, she said:

“Look, 60% of people rated the new design positively.”

Technically true. But she left out the fact that nearly 40% were neutral or negative. By spotlighting only the positive slice, the survey became a pat on the back instead of a warning sign.

This isn’t always intentional. We all fall into confirmation bias: noticing the numbers that support our hopes and quietly ignoring the rest. But in stakeholder-heavy environments, that bias can shape entire roadmaps.

The “average” trap

Another common pitfall: reporting averages.

Let’s say your survey asked people to rate ease of use on a 1–7 scale. The average score comes back as 4.8.

On paper, that looks fine. Not stellar, not terrible. Middle of the road.

But if you dig into the distribution, you might find two very different groups:

• Half of users rated it a 7 (very easy).

• The other half rated it a 2 (very hard).

Averaging those together hides the fact that you’ve got a split audience: one group sailing through, the other drowning. If you report just the average, you’ll completely miss that divide.

Numbers without context are dangerous

The temptation is to grab the highest percentage and headline it. “80% of users agree Feature Z is useful.”

But that number only means something if you put it next to context:

• Who said it?

• How many skipped the question?

• What does “useful” actually mean in their workflow?

Without that framing, you’re just throwing big numbers around. And big numbers are seductive to stakeholders who want quick validation.

How to keep interpretation honest

Before you share survey results, run through these steps:

1. Show the full distribution. Don’t just report averages. Show the range of responses so splits and outliers are visible.

2. Pair numbers with open comments. If you asked an open-ended follow-up, bring those voices in. They keep the numbers grounded in reality.

3. Acknowledge uncertainty. Frame your insights as “This suggests…” instead of “This proves…” Surveys are directional, not definitive.

4. Triangulate with other data. Always check: does this align with what we’re hearing in interviews or seeing in analytics? If not, call out the tension instead of smoothing it over.

Surveys give you patterns. It’s our job to interpret those patterns responsibly and admit where the limits are. Otherwise, we risk turning numbers into a comforting story instead of a useful one.

When Surveys Are the Wrong Tool

Read more

Paid subscribers get the power tools: the UXR Tools Bundle wi…

Read more

Subscribe now

P.S. Paid subscribers get access to full archive, all content, a private Slack community, Substack lives, and a hub of templates, scripts, and mini-courses

Read more

If you want to see everything I post, subscribe below!

Subscribe now

Read more

If you want to see everything I post, subscribe below!

Subscribe now

Read more

Subscribe now

Hi there, you amazing, curious person!

The most common answer in user research, product, tech, and what feels like the entire world is:

“It depends.”

There are very few times I will say, “Do X, and you will be better at Y.”

Meet one of those few exceptions:

Yes. Ted gets its own big image. And also two fun references:

Or:

Okay. I’m done with gifs. On to the gold.

There are very few copy-and-paste formulas that lead to better outcomes and fewer quick wins. But I’m happy to say this is a quick-win formula to help you ask better, unbiased, non-leading, open-ended questions that lead to depth, rich insights, and data gold.

Most “bad questions” aren’t evil. They’re just lazy structures that trap your participant in yes/no, opinions, and future fantasy. TEDW is the simplest fix I know that works across almost any generative session. Paid subscribers get the version you can actually use mid-interview, under pressure:

• the TEDW framework (with examples for each letter)

• a “horrible questions” checklist so you can spot bad prompts fast

• before/after rewrites you can copy into your next guide

• practice prompts to turn TEDW into muscle memory

• real-life TEDW phrasing you can use in meetings, 1:1s, and stakeholder chats

Exclusively for paid subscribers

Read more

👋🏻Hi, this is Nikki with a 🔒subscriber-only 🔒 article from User Research Academy. In every article, I cover in-depth topics on how to conduct user research, grow in your career, and fall in love with the craft of user research again.

Subscribe now

If you know me, you know that I absolutely hate asking people for their preferences.

Understanding user preferences can be tricky because they are so subjective and influenced by many factors. Imagine asking someone if they prefer chocolate or vanilla ice cream. They might say chocolate because they love the taste, while another person might choose vanilla because it reminds them of childhood memories.

The same complexity arises in design choices. If we ask users whether they prefer a blue or purple website background, it’s a decision they usually don’t make. Or, consider asking if they prefer a large image banner or a smaller one with text. These are not typical choices users face, and their responses can be inconsistent and hard to interpret.

I actually tried this with my mom while she was trying to purchase furniture online for their new house. I showed her two different furniture websites and asked her which she preferred. She told me she preferred the second option more because it had fewer sponsored/ad-based search results. I then proceeded to watch her browse for a little bit, pick a sponsored search result and buy that product. Huzzah.

Plus, preferences might not align with best design practices. Someone might like a large image banner, but it could pose accessibility issues. This makes designing solely based on user preferences challenging, as it’s hard to understand the reasons behind their choices and ensure those choices work well for everyone.

When it comes to comparing different versions of ideas or products, I tend to go with usability testing. The version that is more effective and efficient will generally be the one that people end up using more…

However, as we’ve been asked hundreds of times as researchers, sometimes we have to compare certain aspects without the aid of usability testing.

Sometimes we have to understand a degree of preference — as much as I hate writing those words.

But that doesn’t mean that we have to ask people whether or not they prefer something. Instead, introducing: Pairwise comparison.

What’s Pairwise Comparison?

Imagine I gave you a list of ten songs you’ve listened to recently and asked you to rank them in order of preference. You, like most people, might really struggle to fill out this survey accurately.

I love Taylor Swift’s “Love Story,” but do I like it more than Queen’s “Bohemian Rhapsody” or The Beatles’ “Hey Jude?” And then where does Adele’s “Rolling in the Deep” sit within all of this?

You might struggle, trying to decide if you like Taylor Swift’s “Love Story” more than Queen’s “Bohemian Rhapsody” or if The Beatles’ “Hey Jude” tops Adele’s “Rolling in the Deep.” But if I give you two songs at a time and ask which one you like more, it becomes much easier.

This is pairwise comparison in a nutshell. This method simplifies decision-making by breaking down complex choices into manageable comparisons, which then become a series of head-to-head votes. It’s rooted in the idea that humans find it easier to make decisions when faced with two options rather than many.

Pairwise comparison is a fantastic technique for ranking, prioritizing, and comparing options efficiently and effectively.

Let’s look at an example:

Food for a Party

Imagine you are planning a party and trying to figure out the best food to serve your guests. One way you could figure it out is by having people force-rank all the different options. However, as mentioned above, force ranking gives people a lot of different options to choose from, whereas pairwise comparison makes these rankings easier for users.

Let’s look at the options for food:

1. Pizza

2. Chips

3. Chocolate cake

4. Sushi

5. Fruit Salad

6. Chicken wings

We would then go on to create survey questions to compare each option:

1. Pizza vs. Chips

2. Pizza vs. Chocolate cake

3. Pizza vs. Sushi

4. Pizza vs. Fruit salad

5. Pizza vs. Chicken wings

6. Chips vs. Chocolate cake

7. Chips vs. Sushi

8. Chips vs. Fruit salad

9. Chips vs. Chicken wings

10. Chocolate cake vs. Sushi

11. Chocolate cake vs. Fruit salad

12. Chocolate cake vs. Chicken wings

13. Sushi vs. Fruit salad

14. Sushi vs. Chicken wings

15. Fruit salad vs. Chicken wings

Here’s what one of those questions would look like:

The total number of possible pairwise comparisons from a list of options is n(n-1)/2, where “n” stands for the number of options in the set. In the above example, we have six examples, so the formula would look like 6(6-1)/2. Therefore, there would be 15 combinations to ask about, so there would be 15 total survey questions.

In a pairwise comparison survey, one participant can be tasked with voting on every possible pair combination or a bunch of people can each be given a sample of pairs to complete that can later be used to calculate the group’s overall combined ranking.

Different Types of Pairwise Comparison Tests

Pairwise comparison fundamentally involves head-to-head voting, but there are several ways to customize your survey to suit different needs. These formats can be used independently or combined for more flexibility.

Complete Pairwise Comparison

In a Complete Pairwise Comparison, each respondent is shown every possible pair from the list of options. This method provides a detailed and accurate representation of an individual’s preferences. The total number of possible pairwise comparisons is calculated using the formula n(n-1)/2. For example, if there are 10 options, the total number of pairs is 10 \times 9 / 2 = 45. This then means the respondent is going through 45 survey questions.

45 survey questions?! That is wayyyy too much.”

That’s what crossed my mind the first time I heard about this approach. I try to limit my surveys to 10 questions, or at least surveys that take less than seven minutes to account for increased attrition if you hit the seven-minute mark. 45 questions felt like the participant would be there all day taking the survey, or, worse, that no one would take it.

However, pairwise comparison is pretty fast — I recommend you try taking a pariwise comparison survey to see just how quickly questions can go — and since the participant is only choosing between two options at one time, the amount of time and cognitive load for each question is fairly low.

Example:

Imagine a small team of designers choosing their favorite colors for a new logo. With only 10 colors to choose from, each team member compares every pair to express their preferences clearly.

Complete Pairwise Comparison is ideal for small groups, individual preference ranking, or surveys with a limited number of options (usually up to 15-20).

Partial Pairwise Comparison

Read more

If you want to see everything I post, subscribe below!

Subscribe now

Understanding how people feel about your brand—brand sentiment—is incredibly important. It's all about the vibes people get from your brand, ranging from enthusiastic praise to serious gripes. These emotions and perceptions are like a direct line to how consumers see your brand, influencing everything from acquisition to loyalty over time. If people love your brand, they’ll rave about it; if they don’t, you’ll hear about that too.

For user researchers, weaving brand sentiment into their work is super important. While traditional research methods are great at figuring out how users interact with your product or service, adding brand sentiment to the mix helps you understand what users are doing and why they're doing it. For example, knowing that users love a particular feature is good, but understanding that it makes them feel secure or happy? That’s gold—it shows you're hitting the right emotional notes, which can be a big deal for customer loyalty.

Integrating brand sentiment can also make your research more relevant and actionable. It ties the emotional reactions of your users to broader business goals, helping everyone from product designers to marketing teams make smarter, more informed decisions. For instance, imagine a tech company noticing that users feel frustrated with the security features of their app. By addressing these concerns directly and enhancing these features, the company improves user satisfaction and boosts its reputation for reliability. Ignoring brand sentiment issues can lead to immediate loss of customer trust and long-term damage to brand reputation, affecting sales and profitability.

By tapping into how users feel, user researchers can offer a fuller, richer view of the user experience. This approach goes beyond basic usability to include emotional responses, making the insights gathered a powerful tool for nurturing stronger connections between brands and their customers. It's about making research informative and deeply resonant, ensuring it has a real impact on the brand’s journey.

The Importance of Brand Sentiment

Brand sentiment analysis is all about figuring out how people feel about your brand by checking out what they’re saying in reviews, on social media, or any place where they're dropping comments about you. The goal? To sift through all that chatter and find out what's working, what's not, and how your brand comes across to the outside world.

Emotional Components: It's All About the Feels

When it comes to emotions, brand sentiment can run the gamut from happiness and trust to irritation and disappointment. These feelings can pop up instantly based on a customer's experience with your brand. For instance, imagine a customer using a new app feature that makes shopping a breeze—they're likely to feel joy and satisfaction.

Conversely, have you ever had a website crash during checkout? Or, when you’ve found a restaurant to order from only to learn that it isn’t in your delivery area? These moments can be frustrating and irritating for anyone. It’s these raw, spontaneous emotions that give you direct insight into how people are reacting to your brand in real-time.

Cognitive Components: What People Think

This part gets into what customers believe about your brand—their perceptions of your quality, value, and reliability, shaped by their experiences and what you tell them through your marketing. Take a brand like Tesla; people might see it as cutting-edge and innovative because it’s tied up with high-tech vibes and green energy. These beliefs don't just pop up overnight; they build over time and heavily influence how your brand is viewed in the long haul.

Behavioral Components: Seeing Feelings in Action

Emotions and beliefs then translate into people’s behavior. This could mean anything from how often customers buy your products to whether they recommend your brand to friends or even bad-mouth it online. Positive behaviors might look like glowing reviews or repeat purchases, while negative actions could be as serious as a boycott or nasty comments across social platforms.

Pulling It All Together

Mixing these three elements—emotional, cognitive, and behavioral—gives you a full spectrum view of your brand sentiment. Here’s how it breaks down:

• Emotions give you an instant reaction to your brand.

• Thoughts shape how your brand is perceived over time.

• Actions show how these feelings and thoughts play out in real life.

To nail down these insights, you can use tools and approchaes like natural language processing (NLP) to dig through text, sentiment scoring to measure how intense the feelings are, and behavioral analytics to track what consumers do based on their sentiments.

Why It Matters

Diving into brand sentiment analysis helps companies get a clear picture of where they stand now and where they might be heading. It’s about more than just fixing problems; it’s about proactively tweaking your marketing and product strategies to meet customer expectations better. By staying tuned into the sentiment around your brand, you can keep your business moving forward and keep your customers happy.

Define Your Business and Research Goals

When you consider including brand sentiment in your research, the first step is to nail down your objectives. What are you hoping to find out? Whether you're trying to improve your organization's image or just want to keep your finger on the pulse of what your customers are thinking and feeling, defining clear, focused objectives is crucial. Here's how to do it, step by step:

Pinpoint Your Purpose

Before diving into the how, it’s important to clear up why you are doing the project — as always, starting with your goals sets your research up for success. One of the ways I recommend doing this is by asking yourself and your stakeholders:

• What exactly do we want to understand about our brand sentiment?

• How will this research contribute to our larger business goals?

• Are there specific aspects of our brand perception we're concerned about?

• What are we trying to do with this information?

• What decisions will this information help us make at the end of the project?

For example, if you're launching a new product line aimed at millennials, you might want to understand how this demographic perceives your brand to tailor your marketing strategy effectively.

Collaborating Across the Organization

With brand sentiment research, you can work across many different departments since this type of research can support many different teams — from sales to marketing to UX writing to product marketing, and beyond. You can research how people feel about your brand under each of these umbrellas, from when people find your brand (marketing) to when they engage with your solution (sales) so when people use your product (product/tech/UX). Each of these stages is incredibly valuable to understand better.

With this in mind, make sure that you are talking to the different departments about how you can support them within their particular phase. You can use those specific questions from above and apply them to any team to brainstorm ways you can help them with brand sentiment research. I recommend reaching out to different teams to experiment with how you can use brand research to positively impact the organization cross-functionally.

Set Success Metrics through SMART Goals

Read more

Subscribe now

I remember when one of my stakeholders came to me and asked, “how can user research help support the growth of the company?”

At first, it seemed obvious to me that user research was fully capable of helping a company, but when I went to answer, I wasn’t entirely certain what to say.

While the research I had done greatly helped improve the current experience and understanding our current users, I found this much easier to tie to things like retention rather than growth and acquisition.

When I returned to the stakeholder, instead of saying “yes,” which I so badly wanted to say, I asked him what he meant by growth to understand what he was asking for fully.

With his response and context, I learned about a new type of user research I could conduct — growth user research.

How I Define Growth User Research

In the past, I have used pirate metrics to help me connect user research to business impact. This model, coined by Dave McClure* in 2007, highlights the five most critical metrics for businesses to track for success.

I’ve found pirate metrics to be incredibly helpful in this because they hit on the entire funnel of getting customers through to revenue, which is extremely applicable to most businesses. They give a very straightforward way to think about really critical parts of a successful business.

When it comes to growth user research and pirate metrics, you focus on the acquisition and activation phases. These phases essentially cover how people come to you and find your product/service and then how you convert those visitors into actual customers. By focusing on acquisition and activation, you can directly impact the company's growth.

The primary goal of growth research is identifying opportunities and barriers for business growth within user interactions and market trends. This type of research not only scrutinizes how existing users interact with an app but also why potential users might choose to download or ignore it. It seeks to understand the user journey from initial awareness to loyal usage, pinpointing specific interventions that can transform casual users into committed ones.

*Dave McClure is not a stunning human, having been accused of sexual harassment. I’m not a fan of him, but I wanted to cite the original source of the pirate metrics. While he sucks, hopefully, we can leverage this model for the greater good.

Why is Growth Research Important?

There are so many products and services out there that aim to solve pain points and help users achieve particular goals. And while I am a huge believer in the fact that there are plenty of users to go around, companies still want to understand how to scale and grow.

And, with all the noise out there with all these products and services, many companies face stagnation in daily active users (DAU) and monthly active users (MAU), as well as potentially high Customer Acquisition Costs (CAC). Experiencing stagnancy in these metrics usually signals problems with growth within acquisition, activation, and market positioning.

Companies with flatlining user growth find it difficult to attract venture capital, sustain revenue generation, and scale, which can lead to companies laying off many employees (which we’ve seen a lot of recently) or completely going under (something else I’ve also seen and experienced). Furthermore, in social and network-based apps, where user value often increases with the size of the user base, stagnant metrics can lead to a downward spiral in user engagement and revenue. And (fortunately or unfortunately), revenue is essential to organizations.

Conducting growth user research can help positively impact these metrics by understanding these key areas of acquisition and activation. Through this approach, you can understand why people are engaging with or ignoring your product/service and why they are (or not!) converting from visitors to actual customers.

Through this type of research, you can impact some of the most important metrics for companies—every company needs to grow and to grow, they need to understand how to get customers. With this information, you can help the company innovate, create more effective user acquisition tactics, enhance activation strategies, and ultimately deliver a product that resonates more deeply with its target audience.

So, how do you conduct this powerful type of user research?

The Fundamentals of Growth User Research

When conducting growth user research, I typically focus on the top two areas of the pirate metrics funnel: acquisition and activation.

Acquisition - how people find and are introduced to your product/service

Acquisition is all about getting new customers into your product/service so that they know it exists as an option to solve one of their pain points or support an unmet need in one of their goals. There are many ways a company can do this, such as:

• SEO

• Marketing (including email and social media)

• Sales

• Paid advertising

Acquisition is the top of the funnel. Without properly acquiring users, sustaining a product or service over time is difficult, which is why it is incredibly important to apply user research to this part of the process.

Activation - how people begin to use your product/service

It’s good if people find you, but that first interaction is key. After running my business for almost two years full-time, I know how important it is to activate users and get them to take that first step with you.

There are many ways to activate users, which hugely depends on your product/service/organization. However, when you think about activation, think about the primary conversion metrics that determine the success of certain channels and campaigns rather than high-level or micro-conversions. This could look like a funnel:

1. Someone comes to your website

2. They see value in your work

3. They try a free trial, book a demo, or sign up for a newsletter

Activation is the beginning of your relationship with the customer. Before this, they’re anonymously researching your business and competitors before they take any specific action that allows you to begin directly engaging with them.

For me, these two areas are growth research because they allow you to understand and grow your user base. Of course, retention is extremely important, but you need people to retain before turning to retention, and that is where growth user research comes in.

Goals of Growth User Research

Research goals are the most important part of setting your study up for success, and it is no different when it comes to growth user research. By understanding what you are trying to accomplish from the get-go, you ensure you choose the right methodology to get the exact information your stakeholders need to make whatever decisions they face that need to move forward.

I tend to bucket goals into two different areas:

1. Business goals, which focus on the goals of the actual business and tend not to be user-focused. This is what the business is hoping for out of the study.

2. Study goals, which focus on the user and on what information you are trying to understand from participants

I split these goals up because they serve completely separate needs. Typically, from stakeholders, I receive very business-oriented goals. For example, they want to increase the conversion rate of a product or service. The thing is, I can’t go and ask users, “Why didn’t you convert?” Not only is that a hugely leading and biased question, but it is also impossible for someone to answer (try answering it yourself about a product!).

However, these business goals are incredibly important for us to incorporate as user researchers because one of the best ways we can impact an organization is to tie our research project to business-related goals and metrics.

By separating these goals, your study can simultaneously hold a business and user focus, ensuring you are doing the most impactful research for the business and focusing on the users’ needs, goals, and pain points.

Business-Related Goals

When it comes to growth user research, here are some business-related goals you can expect:

• Increase App Downloads: Companies can optimize their marketing channels and app store presence by understanding the pathways through which potential users discover apps. For example, if research indicates that most downloads come from social media ads, focusing research on understanding current campaigns could be beneficial.

• Increase App Store Ranking: Your app’s position in search results and category rankings influence visibility and download rates. Companies often want to optimize where they fall within this ranking, which is great for social proof.

• Increase Conversion Rate: Conversion rate doesn’t always involve money. This can also represent the number of people who take action after visiting a product or service, which indicates a movement from acquisition toward activation.

• Decrease Time to Activation: This looks at the average time it takes for a new user to reach the activation point after first engaging with a product or service, which is incredibly important in converting customers from casual to more committed.

• Increase Activation Rate: Activation rate looks at the percentage of users who take a specific desired action within the product/service, such as completing a profile setup or making a first booking.

• Increase Retention Rates at Day 1, 7, and 30: Retention rates measure the percentage of users who return to the product or service one day, seven days, and thirty days after the initial conversion. I know retention is a separate study, but looking at early retention can still help understand acquisition.

As you can see, these are very business-focused, and nowhere in these goals is the user mentioned in terms of more deeply understanding them. This is why it is so important to include user-focused study goals, so we ensure we aren’t asking people why they haven’t yet activated or converted.

Study-Related Goals

Now that we understand our business counterpart, it is time to look at user-centric growth research study goals that will get us the information to help answer those business goals. Later on in this article, I will highlight the different methods you can use to act on these goals, but let’s first take a look at some examples:

1. Identify channels that potential users frequent and understand their current experience with our product/service on these channels

2. Discover pain points during the activation and onboarding process

3. Uncover the current journey people go through when trying to find a product/service to help them accomplish [x goal]

4. Learn about unmet needs (both inside and outside our product/service) people are experiencing, from awareness to onboarding

5. Discover users’ goals when searching for and engaging with similar products/services

6. Evaluate the current process from awareness to onboarding to locate key problems and confusion

These goals are completely focused on people's experiences and the feelings associated with them. This type of information is critical for answering our business goals. Without understanding the journey people are going through, the friction they experience, and what is missing, we can’t meaningfully impact the business goals.

When we empathize and deeply resonate with our users’ experiences, we can create products/services appropriate to their needs and goals, which will naturally have a positive influence on the organization’s business goals.

Key Metrics to Monitor (aka Success Metrics)

In addition to business goals, I highly recommend looking into growth-related metrics, which I always include in the success metrics section of my research plan.

Ideally, your success criteria should be measurable and related to your business and study goals. A great question is, “What would determine if this research project succeeds?” You can look at this from both a business and user perspective.

Regarding growth user research, it is crucial to compare certain metrics before and after the research is conducted and changes are made. Comparing these metrics before and after the changes helps identify how successful the project was and also helps us demonstrate concrete impact as researchers.

When it comes to these, I recommend measuring the following success metrics (not all might apply to your organization, so pick and choose what makes the most sense):

• DAU and MAU: Daily Active Users and Monthly Active Users provide a clear picture of the app’s usage frequency and can indicate user engagement over time.

• Download Rate: Total number of downloads over a specific period. This metric helps measure the immediate impact of acquisition efforts.

• Cost per Acquisition (CPA): The cost associated with acquiring a new customer, typically calculated by dividing marketing expenses by the number of new users acquired. It’s essential for evaluating the financial efficiency of your acquisition strategies.

• Activation Rate: The activation rate looks at the percentage of users who take a specific desired action within the product/service, such as completing a profile setup or making a first booking.

• Time to Activation: The average time it takes for a new user to reach the activation point after first engaging with the product. Faster activations can indicate a more intuitive user experience.

• User Retention Rates (first visit): Measures the percentage of users who return to the app after their first visit. Low retention rates can indicate problems with app functionality, value proposition, or user experience.

• Conversion Rates from Visitor to Registered User: This tracks how many visitors become registered users, which can help identify barriers in the registration process or initial user experience that could be optimized for better conversion.

• Bounce Rate: Understanding where people bounce off your product/service is extremely helpful in seeing where there might be problems. With a lower bounce rate, you can assume the content is better resonating with users/customers.

Some of these metrics are directly tied to the business goals, which makes complete sense. Writing these may feel repetitive, but it is always important to ensure we have measurable metrics written somewhere to explore after the research project and changes are made.

Preparing for Growth User Research

Before diving straight into conducting growth user research at your organization, there are some steps to take to ensure the project's success.

Understand the Point of the Project

I know this might seem like a silly point, but my experience has sometimes involved taking a “full steam ahead” approach without thinking more deeply about the project I am trying to complete.

Especially when faced with something new or less familiar, I can forge ahead without being as intentional or thoughtful about what I’m trying to accomplish. This mindset has led me to say yes to projects only to figure out later that I either couldn’t get the data or got data that, ultimately, wasn’t helpful.

Take the time to understand the project's point/ultimate goal and if user research can help. For instance, if your organization wants to know your potential customers' different preferences, user research isn’t suited for that type of project because we can’t practically measure preference.

At this stage, deeply understand your organization’s and stakeholders’ needs for the project and assess if it is the best fit for research or if another department, such as marketing, can better help.

If your stakeholders struggle to articulate their needs, I recommend doing stakeholder interviews or having them fill out an intake document.

Set Research Goals

Once you have identified that user research can help answer your stakeholders' questions, it is time to set research goals immediately. These goals can help you further establish if user research is the best approach for the project because, if you struggle to create goals, it can indicate that user research might not be right to answer stakeholders’ questions and needs.

As mentioned above, goals help you establish what you need to learn from your research project and the most relevant information you need to gather from your participants. Establishing these goals not only sets the focus for your project but also helps tremendously with recruiting the correct participants and forming the best interview questions.

One way to start formulating research goals is to ask yourself (and stakeholders, of course) a few different questions, such as:

• What do we want to learn about [research topic]?

• What type of experiences do we want to learn about?

• What information do we need by the end of the study?

• What decisions are we trying to make by the end of the study, and what can help us make those decisions more confidently?

You can also fill out the following prompt:

I need [information] to understand [proposed research goal] to make [decision] that will impact [team/organizational goal]. Then, by the end of the study/workshop, I need [ideal outcome].

By writing down all this information, you can create concrete research goals that ensure you get the exact information you need from your study. As a refresher, here are some potential goals you can use as a jumping-off point:

1. Identify channels that potential users frequent and understand their current experience with our product/service on these channels

2. Discover pain points during the activation and onboarding process

3. Uncover people's current journeys when they are looking for a product or service to help them accomplish [x goal]

4. Learn about unmet needs (both inside and outside our product/service) people are experiencing, from awareness to onboarding

5. Discover users’ goals when searching for and engaging with similar products/services

6. Evaluate the current process from awareness to onboarding to locate key problems and confusion

I recommend having no more than three goals for any study. Going over three goals increases the scope and makes it hard to get in-depth information on each goal.

Identify Your Target Audience

With a clear focus on the information you need, identifying who you need to talk to becomes much easier because you can create screener questions based on the goals you need to accomplish.

When it comes to segmenting your users, there are several ways to do it:

• Demographic Segmentation: Break down your audience by age, gender, income, etc., to understand who uses your product or service.

• Behavioral Segmentation: Analyze how different users interact with your product. Look at usage patterns, purchase behaviors, and loyalty.

• Psychographic Segmentation: Consider the attitudes, goals, and other psychological criteria influencing how different audience subsets engage with your product.

Personally, I am a fan of a combination of all three of these different segmentation types, depending on the product/service. For example, when I was working on understanding how Gen-Z gets fashion inspiration, demographic information was key in understanding age and income. Still, I also had to look at psychographic segmentation, including their goals and attitudes toward fashion.

When I sit down to brainstorm recruitment criteria, I usually bucket them into the above types of segmentation. For example, when I was working with a real estate company trying to understand intent to purchase a new home, we brainstormed the following criteria:

Behavioral data:

• Have searched for a house in the past three weeks

• Have never bought a home before

• The person "in charge" of the home-buying experience

• Checking for homes every day

Product usage:

• Have used the product in the past month

• Have used a competitor's product in the past month

Psychographic data:

• Would like to move in the next two months

• Are looking for specific criteria in a house (ex: yard, kid-friendly, location)

Demographic data:

• Age

• Income level

You can then use these criteria to help you form a screener survey. Because your screener questions are based directly on the criteria you need, it will help ensure that you get the best participants for your study—the ones who can give you all the juicy information you need.

Now, with growth user research, it is important to go one step further because you want to ensure you are picking a segment that shows growth potential. For instance, if younger users are more engaged but represent a smaller portion of your base, they might be a key target for your research to understand how to grow that particular segment.

Look into your data to understand the potential for growth from your current user base, and take the time to hone into those segments as well.

Choose the Best Research Methods

Understanding the goals and the participants you need to speak to allows you to choose the best approach to the study intentionally. When it comes to growth user research—or, really, all research studies—I always recommend a mixed methods approach whenever possible and applicable.

With a mix of quantitative and qualitative data, you give your team a holistic picture of users and the experience, armoring them with all the data they need to make the best decisions moving forward. I know incorporating mixed methods isn’t always possible, so don't worry too much if you can't. However, if you can, I recommend trying it.

Below are some potential methods you can use in growth user research.

Potential Quantitative Methods:

Read more

Subscribe now

As a qualitative user researcher, I often felt overwhelmed by quantitative data and was unsure how to fit it into my user research process. I did my best to circle around the problem as much as possible, sometimes even ignoring the need my teams had for quantitative data to back up the qualitative. And for some time, I even said that quantitative data wasn’t necessary and all you really needed was some qualitative research because it told a better story.

Eeek. 🙈

I did this not because I didn’t find quantitative data useful, but because I had no idea how to use it in user research. I was so unfamiliar with product analytics and using data this way that I tried to avoid it at all costs.

There is no other way to put this: avoiding quantitative data negatively impacted my career. Data can help in many ways, including how we track our impact and show value as user researchers.

Finally, there came a point in my career where I couldn’t avoid using quantitative data in my work anymore and I had to learn how to incorporate this critical approach into my day-to-day (well, really, it didn’t start as day-to-day, but rather the occasional experiment). I made using quantitative data my top professional development goal so I had plenty of motivation to make positive progress (I was also terrified at the same time).

Once I learned to work better with data, I didn’t look back. It changed how I approached user research. As a disclaimer, I am not a quantitative user researcher and have some mixed methods experience, but I would still call myself a qualitative user researcher. So, this is a guide for those in that space looking to get more comfortable incorporating quantitative data into their process.

Importance of Quantitative Data

There are three main types of research design:

1. Exploratory sequential design involves initial qualitative research to explore a concept, followed by quantitative research to test these explorations broadly. For example, interviews to understand user needs are followed by surveys of a larger audience to quantify needs.

2. Explanatory sequential design reverses this order, beginning with quantitative research to identify trends and then using qualitative methods to delve deeper into those findings, like surveying for top pain points and conducting interviews to get detailed insights on why/how those pain points occur.

3. Parallel convergent design occurs when qualitative and quantitative research are conducted independently and simultaneously, and then results are merged to reach a conclusion. This approach allows for comprehensive insights by combining the depth of qualitative data with the breadth of quantitative data, enhancing the understanding of research findings. An example is simultaneously conducting interviews to explore users’ needs and goals while deploying surveys to measure user satisfaction levels and then integrating these insights to guide product development decisions.

For the majority of my career, I focused on exploratory design. I sometimes threw in a survey at the end, but mostly as a way to quell stakeholders’ needs for quantifying data or getting numbers rather than as a way to gather more data.

However, there were many times when I could have benefitted from explanatory sequential design, starting with data analytics.

Data analytics can give you an extremely clear picture of what is happening. It can show user behavior, trends, and potential problems and greatly enhance qualitative data. Additionally, data can help you with narrowing the scope of your research project.

There were many projects I started that were way too broad and could have used narrowing down. One of the best ways to do this is using quantitative data to better understand the landscape before diving into a broader topic or problem space. I remember once having to tackle the question of churn. At that point, I decided to try to tackle churn through interviews.

It was rough. I barely spoke to anyone because, well, churned users don’t care about helping the company they no longer use, and the interviews felt all over the place. There was little structure, and I would have benefitted from incorporating quantitative data into the project.

By making the shift toward using quantitative data, you can:

• Use more objective decision-making based on data as it will show you what is happening, such as the concrete user behavior.

• Enhance deliverables, such as journey maps and personas, by including concrete behavior data and quantifying qualitative information, such as needs, goals, and pain points.

• Narrow the scope of your research projects to be more effective and impactful so you focus on a tangible scope.

• Through product analytics, such as bounce rates, drop-offs, session durations, and other metrics, you can see potential pain points or friction in your product/service.

Quantitative data can help in various ways, and learning these skills is very important to move your career forward as a user researcher. This doesn’t mean you have to become a quantitative or mixed-methods user researcher, but knowing how to utilize quantitative data can up-level your skills and help you advance in your career. It also makes you feel more confident in the studies you run!

So, how do we go about bringing quantitative data into the fold?

Defining Research Goals

The first and most important thing to do as a user researcher starting a new project, especially when starting a project type different from the one you are used to, is to write out the project's goals. This stage was critical when incorporating more quantitative data into my research process. It helped me better understand and define what I was trying to accomplish through the project and by using quantitative data.

Research goals help you give focus to your study and enable you to understand what approaches to take and methods to use to get you the information you need most effectively and efficiently. When I first started using quantitative data in research and experimented with explanatory sequential design, my goals looked like this:

• Look at data to understand what is happening

• Explore data analytics to understand behavior

While it was great that I looked at this data, I still had no clear goals in mind, so I got lost in the product analytics, uncertain of what I was looking for. This approach wasted a lot of time and, ultimately, ended up with me feeling frustrated and defeated in the process.

I wouldn’t have written qualitative research goals like this, so I set out to revise how I approached quantitative research goals.

Writing Quantitative Research Goals

Research goals directly relate to your research statement because they are the more in-depth areas you want to explore in your research statement that will help you answer what you are trying to learn. Your research goals should address what you want to learn and how you will study the research statement.

You want to gather information about these goals by the end of the study. They aren’t posed as questions, but you want to be able to “answer” them by getting enough data to feel comfortable making decisions.

To define these goals, I ask myself the following questions:

• What do we want to learn about [research topic]?

• What type of experiences do we want to learn about?

• What information do we want at the end of the study?

• What decisions are we trying to make by the end of the study, and what can help us make those decisions more confidently?

When it came to qualitative research, this felt easy to me because I was familiar with writing goals about deeply understanding peoples’ processes and mental models. However, writing goals for quantitative research projects confused me. I wasn’t exactly sure how to write goals about understanding what was happening or about behavior since I had avoided these types of goals in my qualitative studies.

So, I went back to basics and asked myself the above questions, but with a quantitative lens. When it came to updating my research project goals, they looked more like this:

• Understand the trends of behavior patterns within [x flow]/[y topic]

• Identify potential pain points or friction within [x flow]

• Measure user satisfaction when it comes to [x product]/[y topic]

• Evaluate the top behaviors within [x product]/[y flow]

With these types of goals, it became clearer what I wanted to achieve when looking at the quantitative data rather than simply “exploring data.”

When writing quantitative research goals, we are looking at what is happening, so try to align your goals to identify behaviors and patterns within the quantitative data you are looking at.

Tying into Business Objectives

Conducting impactful user research means engaging in research aligned with business objectives. By understanding what the business is trying to achieve, we can utilize that information to help give us clarity and direction on what we need to understand better and ensure we are researching the most important information for the business.

I struggled a lot with integrating research into the business, and for a while, I thought it was simply impossible. I believed research lived just outside the scope of business and that I couldn’t bring the two together. However, over time, I was able to better understand business needs and see how research could help a business achieve its objectives.

Regardless if you are conducting qualitative or quantitative research (or both!), it is essential for us to align our research with greater business objectives. I’ve found this easier to do regarding quantitative user research.

For example, imagine you work at a company that streams TV shows and movies, and your business goal is to enhance user engagement; you can use that to guide your research study. Once you define user engagement (let’s say for this we can say viewing time and subscription renewals), you can dive right in with quantitative data to better understand what is happening, such as:

• What is the current average viewing time, and how does this differ from our ideal viewing time?

• What is the current subscription renewal rate, and how does this differ from our ideal?

• Where are the potential frictions/pain points in the current viewing experience?

• What are people searching for that currently might not be on our platform?

Straight off the bat, you can dig into analytics to understand what people are interacting with and start to form some impactful research goals like these:

1. Understand what users are currently interacting with and where there are gaps in our platform

2. Identify potential pain points in the viewing experience

3. Evaluate any trends behind when people decide not to resubscribe (e.g., after a certain period of time)

With these goals in mind, you can use the following approaches and methods to gather the information to answer these goals:

Read more

Subscribe now

Surveys can feel like a simple methodology — write a few questions, send the survey to a bunch of people, and, voila, we have insights. If surveys were that easy, we’d all be swimming in deep and rich insights, and, likely, surveys would be the only method we need.

But that isn’t necessarily the case because surveys, while a great research method, can be difficult to assemble to get appropriate information. There have been many times when I’ve used a survey when I should have used another method, such as 1x1 interviews, diary studies, or looking into quantitative data.

I remember desperately trying to ask five open-ended questions in a survey to get qualitative information. That was back when I thought surveys could solve pretty much anything. I got my heart broken several times over trying to use surveys incorrectly (and for everything).

Now, I am very intentional and thoughtful when I choose surveys as a method

What are User Research Surveys (and How are they Different)

Imagine you are throwing a party for your friends, and you want to understand what your friends need to make the party enjoyable for them. You might need to understand the types of food to get for the party, the kind of music people listen to, the games they play, and a bit about different parties they’ve enjoyed in the past and what made them enjoyable. You are throwing out feelers to make the night successful. To make it a good experience for your guests.

A user research survey is similar. You are asking questions to understand, at scale, the pain points, habits, behaviors, and needs of your users. With a survey, you are looking to broaden your understanding of an audience, specifically around what they are doing and feeling.

Because of this “what” caveat, a great way to use surveys is to supplement with other “why” methodologies, such as 1x1 interviews. For example, surveys are fantastic at telling you, on a larger scale, what people are doing or what tools people are using, but they don’t go a great job telling you why a person is acting in a certain way or using a particular tool. Or how the experience is for someone beyond a rating.

it is super important to keep this in mind as you decide whether or not surveys are the right approach for your study.

How are they Different?

There are a lot of different surveys out there, so it can sometimes be confusing to delineate which one is best for what you need. Here are ways you can differentiate them:

• Market Research Surveys: These surveys look at the big picture of your market or audience. They tend to have a much broader question or scope, such as looking at the entire audience to understand what type of party to throw, rather than getting into the nitty-gritty of people's experience.

• Customer Satisfaction Surveys: These are quick check-ins post-purchase or interaction, seeing how the experience was for the user, typically surrounding their satisfaction. These focus a lot on post-experience, so people can tell you what actually went wrong and recall from a particular usage. This would be like asking people how satisfied or dissatisfied they were with the snacks at your party.

• Feedback Forms: These are even quicker, often just a few questions or a rating scale about a specific concept or thing, like the new dip you tried out at your party. They're super focused and don't dive deep into the whole experience but rather look at a small slice or moment.

Where can they Fail?

Many companies rely solely on survey data to make data-driven decisions. They include comment boxes with the hope users will leave a message explaining their survey response— but most users don’t.

The truth is surveys don’t drive better decisions on their own. Too many companies sometimes use surveys to convince themselves they are customer-centric. But when you only use one method of user research or use only quantitative or qualitative data, you are missing other pieces of the puzzle that give you a more robust and empathetic understanding of your users.

It is super important to weigh surveys' pros and cons, such as:

• Lack of Depth: Surveys may not provide the depth of insights that can be gained from qualitative methods like interviews or ethnographic studies. The predefined answers can limit the range of responses, potentially missing nuanced feedback.

• Low Response Rates: Depending on the survey's design and distribution method, achieving a high response rate can be challenging, leading to sample bias if only certain users are more likely to respond.

• Misinterpretation of Questions: Users might misunderstand survey questions, leading to inaccurate responses. The lack of interaction with respondents means there's no opportunity to clarify questions or probe deeper as you can in interviews.

• Over-reliance on Self-reported Data: Surveys rely on users' perceptions and self-reported data, which can be biased or inaccurate due to memory recall issues or the desire to present oneself in a favorable light.

Surveys can be a really great tool when used properly. Let’s dive into how to know a survey is the right fit and, once you determined that, how to write an effective survey.

Where to Start with Your User Research Survey

Starting with Understanding

I often see a problem when people try to create a survey, and they don’t know the meaningful answers to put in as choices for participants. For example, when working at a hospitality start-up, I sent a survey to understand people’s top pain points with our software.

The issue? I had no idea what to list as the multiple-choice or ranking options. For half of those answers, I guessed what might be the pain points they were experiencing. This is obviously not ideal because, when it comes to surveys, people might just choose an easy answer that isn’t a one hundred percent fit or the one that is closest to what they are experiencing rather than the actual experience they are having.

What are Your Goals?

Defining and setting your goals (like with every other project) are so incredibly important for the success of your survey. They give you a super clear focus on what exact information you need from your survey and can help you define whether or not a survey is the right fit for your study. These goals will also help you develop questions that get you the information you need by the end of the study.

Creating goals is always the first step that I take because it honestly sets the entire project up for success, no matter what approach, method, or problem you are trying to solve.

I usually define goals for surveys into three distinct buckets:

1. To look at usability

2. To better understand your user base

3. To prioritize information

However, this is only the surface of creating goals and, sometimes, they can be tricky to create, especially if you aren’t used to writing them. So let’s go step-by-step into how you can determine your survey goals.

1. Define the overarching why. The best way to start this process is to understand what you are trying to achieve through the survey. This will help you determine if a survey is, in fact, the right method to use. In this step, you can ask yourself:

   1. What are you trying to achieve with this survey?

   2. How will this survey help you/the team?

   3. What do you expect from the results (assumptions)?

   4. What do you expect to do with the results?

   5. How does this survey fit into any business/organizational goals?

2. Specify your needs. Next, defining what exactly you are trying to get from your survey is critical. I recommend drilling down on these questions to know whether a survey is the right approach for what you are trying to get. These questions will also help you develop screener questions.

   1. What are you trying to learn about your users?

   2. What information are you trying to get from your users? Does a survey make sense, given that information?

   3. What kind of information do your users have to be able to give you?

   4. What gaps does the information from this survey aim to fill?

3. List the criteria of your ideal participant. Once you better understand your needs and if a survey is the right fit, you can start thinking about who you want to recruit. I tend to do this before finalizing survey goals because it gives me another opportunity to ensure a survey is the best way to get the information I need from users. Read an in-depth guide on screener surveys here!

   1. What are the questions your users have to answer to get you meaningful information?

   2. What gaps in knowledge do you have that you need your participants to fill in?

   3. What behaviors do you need to understand more?

   4. What habits are you trying to target?

4. Write your goals. Now that you’ve gathered all this information, it is time to write your goals for the survey. As I mentioned above, I use certain models for writing survey goals that go into one of those three buckets:

   1. Usability goals. These are all about understanding the usability of a product or service and often include UX metrics such as the SEQ, SUS, and UMUX-lite. You can read more about this process here! Here are some example goals:

      1. Evaluate people’s perception of the ease of use of [product/service]

      2. Identify the task- or product-based satisfaction of [product/service]

   2. Understanding goals. These goals focus on understanding different behaviors, actions, tasks, or tools people use and the frequency of that information.

      1. Identify behaviors and actions people are taking on [product/service]

      2. Uncover the different tools participants are currently using to achieve their goals and what their experience is like with them

      3. Uncover people’s perceptions/feelings regarding certain actions/tasks on [product/service]

   3. Prioritization goals. Whenever you have a large amount of qualitative data, it is important to prioritize the data to allow your team to take action. Prioritization surveys, such as the opportunity gap survey, are perfect for this situation.

      1. Prioritize qualitative insights based on the current level of importance and satisfaction

      2. Understand the most impactful unmet needs/pain points to solve for

I highly recommend working with stakeholders on defining your goals to ensure everyone is aligned on the expected outcomes and information from the study!

Formulating Effective Survey Questions

Read more

Subscribe now

Read more

Subscribe now

After my article on writing a generative research interview guide, I received several questions on synthesizing generative research data, which has led to this article! Thank you to everyone who submitted the question — I love knowing I am writing on helpful topics!

Analysis and synthesis of qualitative data have always felt shrouded in a bit of a mystery. Whenever I started as a user researcher, I was confused and overwhelmed by the seemingly magical occurrences between getting raw data and creating actionable insights.

I used to google how others did analysis and synthesis all the time to get concrete tips or information on navigating this puzzling process. However, there wasn’t much information on how to apply the concepts behind synthesis to my work. And, rightly so. A lot of analysis and synthesis is incredibly confidential — we use it to make decisions about customers, create products, and innovate. It was no wonder people weren’t sharing exactly how they synthesized raw data into insights.

But, it certainly made it difficult for me to understand how exactly to turn notes into codes, codes into patterns, and patterns into insights, especially when it came to generative research. I felt a bit lost when heading into this part of my research process, which didn’t help with my confidence levels. I dreaded synthesis because I was terrified that I couldn’t pull effective insights from the data.

After years of analysis and synthesis, I have become much more comfortable and confident with the process, and, now, it is my favorite part of research — I love watching data from different humans come together in a cohesive and impactful insight.

In this article, I’ll run you through my step-by-step process of analyzing and synthesizing generative research data by using a past example of my work to make the concepts/learnings as applicable and concrete as possible.

Let’s Define Synthesis

Synthesis is about conjoining information across different participants to see the overlap. When you find this overlap, it could indicate a pattern in the data.

From there, patterns mean that you might have to create, improve, or change something in your product. So, synthesis means assembling a load of qualitative data to find similarities across people to improve your product or create something innovative.

When you get enough data and identify these patterns, you can create insights to help your team make decisions and act on the research. Without synthesis, looking at a bunch of qualitative data and pulling meaningful conclusions from it is incredibly difficult. Synthesis is the glue that brings the different participants together into a cohesive story.

A Synthesis Process

I quickly learned that synthesis processes can look different depending on the organization and resources you have at your disposal. As a team of one who worked at low-budget start-ups and typically did research in a vacuum, my synthesis process looked like:

• Step 0: Record each research session (audio or video is fine). If you cannot record, beg someone to come in and take notes for you. If no one can, apologize profusely to the participant and say that you must take notes if you can’t audio record the session.

• Step 1: Review each interview within 24 hours of the session. During this time, I essentially write a transcript of the interview in Excel.

• Step 2: Highlight important notes or quotes. I will then timestamp these to make video/audio clips for presentations or reports easily later on.

• Step 3: Create a research summary based on that individual participant

• Step 4. Define the codes for the project, either through inductive or deductive methods.

• Step 5: Go through each line, noting any relevant tags — not every transcript line will have a tag. Do this across all the different transcripts.

• Step 6: Combine the codes/tags across the participants using an affinity diagram.

• Step 7: Find the patterns that come up the most frequently across participants.

• Step 8: Write insights based on the most common patterns that come up during synthesis to send and present to the team and follow up with an activation/ideation workshop.

I know there are a lot of steps in this process, and sometimes, it can feel like synthesis might take forever, but once you get more comfortable and confident with it, you will get into a good routine. Plus, there are some ways to save time, such as during code creation or by running debrief sessions after each interview.

To illustrate each step better, I will use an example from my previous experience working as a user researcher at a travel company called fromatob. This was a ticketing company where you could input your starting location and destination, and we would show you a combination of methods (ex, car, bus, train, plane) to get to your destination.

I don’t have the research plan anymore from this particular study, but I will outline what we were trying to achieve through this project:

Background: 

With generative research, we are looking to more deeply understand how our users think about making travel decisions (from inspiration to planning to booking) and how they interact, at a high level, with the fromatob product.

Goals:

• Understand how people make decisions for leisure travel, from inspiration to planning to booking, and their mental models during this process

• Discover peoples’ pain points while planning leisure travel

• Uncover peoples’ needs and goals that emerge when planning leisure travel

• Identify how people are currently interacting with the fromatob website/app based on their last booking experience for leisure travel

Methodology:

• 25 one-on-one 90-minute interviews — for the first 60 minutes, focusing on goals one, two and three, and for the last 30 minutes, diving into the last booking experience on fromatob

Step 0: Record the Session

My first step, beyond any other, is to record the interview in any way possible — whether that be video or audio, on your computer or through your phone (with permission, of course). Recording the session enables you to focus on the participant fully and helps reduce “busy bias,” which can happen when you are trying to split your focus and write down an interpretation rather than what a participant really said.

If, for whatever reason, you can’t record, ask someone to come with you so that they can take down a transcript of the interview. Notice I said transcript and not notes. Taking truly unbiased notes is difficult as we naturally put our spin and perspective on what people say/do, so simply writing a transcript is much easier.

Resources:

• Consent form

Step 1: Review Each Interview

After each session, and ideally within 24 hours, I review the interview recording and type up a transcript. I typically do this for two reasons:

1. No one else will do it for me (ex, I don’t have a transcriber or access to a transcription service)

2. It helps me remember the smaller parts of the interview that I might have missed or misinterpreted

So, what does this look like? I listen to the entire interview and type up the transcript using Excel. I use Excel because it is what I learned, but you can also use Word if you feel more comfortable typing in that. The whole point is writing a transcript.

I wish I could explain when exactly I make line breaks in Excel, but it isn’t an exact science — usually, it is when I feel there is a new idea or sometimes a new thought, but that’s not always consistent. What matters more here is that I am writing a transcript of the session and not interpreting what the person is saying but writing what they said. I usually use the first person (“I did this”), but you can sometimes see when I flip to the third person (“She said she books the best class).

There is nothing strategic about this. It is merely the fact that I am typing quickly and can sometimes make mistakes 😁. Again, what matters is that I am relaying exactly what the person said or did rather than interpreting it.

I recommend one of two things:

1. Hire someone or get access to a tool that does this for you if you have the resources available

2. Try out a few different ways to do this until you find one that feels good for you. My preferred method is Excel, but yours might be Word, Evernote, or something else. Just see what fits best

And, if you are really tight on time, or you find this part particularly painful, you can opt to hold a debrief session instead. I typically do a debrief in addition to writing the transcript because I find it refreshes my memory very well, but I know it isn’t always realistic.

A research debrief is the time you take after a session to reflect on it and encourage deep learning and complex connections.

Think of it as downloading all the information you just learned without writing the transcript. The debrief is a perfect time to reflect on what just happened during the session and bring many minds together. You can do this by holding a 30-minute debrief after each interview session.

Resources:

• How to hold a user research debrief

• Debrief Miro template

Step 2: Highlight Important Notes or Quotes

I tend to do this in parallel with step one because the two work together very well. If I find a particularly interesting quote or clip of the session while I am transcribing the interview, I usually highlight it and put the timestamp in another column in Excel. This makes it super easy whenever I try to find impactful quotes or when creating video clips.

Read more

Subscribe now

Read more

Subscribe now

Have you ever received this question from stakeholders:

“Why are people leaving/churning?”

The first time I heard this question, I was immediately invested. I wanted to find out all the reasons people had left our app. I wanted to unlock this magical realm of information for our teams that would solve all of our churn problems.

And then, I tried to recruit churned users.

I tried to recruit people who were annoyed with us, didn’t want to hear from us, were pissed about something we did, removed themselves from our product universe, and, quite frankly, disliked us.

Spoiler: it didn’t work well.

I spent months trying to find people to talk to. Most of my recruitment efforts were rebuffed. No one wanted to talk to me about why they left. No one wanted to take my churn survey, even for a pretty solid incentive.

I repeated this process a few times with little success and, when I was finally accepting defeat, I decided to go about it in a slightly different way. What if we focused on retention instead of churn?

What’s retention?

Retention is all about keeping your customer base. It’s great to acquire new customers, but it is often (for most companies) really expensive. And, if you are acquiring new customers, but they are leaving, then you are burning through quite a lot of money.

Retaining customers not only ensures a stable revenue stream but can also lead to brand advocacy and long-term growth. You can measure retention using metrics like:

• Customer lifetime value

• Churn rate

• Returning to your website

• Opening emails repeatedly

• Checking your product repeatedly in a given timeframe

For this particular article, we’re going to focus on calculating and tracking retention rate:

Retention rate: (Number of active users over a period of time / Total Number of active users in the previous period) x 100

Since I hate numbers and math, let’s give a concrete example of calculating a retention rate:

Assume that 2,000 people downloaded your app. After one day, 400 people used the app. After a week, only 200 people used it. After one month, there were just 100 people who continued to use your app. In this case, the retention rates are as follows:

• (400/2000)*100 = 20% retention after one day

• (200/2000)*100 = 10% retention after a week

• (100/2000)*100 = 5% retention after one month

So, when it comes down to it, retention looks at how many people continue to use your product over a period of time.

Where does user research come into this?

Retention research can have a huge impact on an organization. By understanding why your customers stay and the flip side of why they leave, you can positively influence important metrics like revenue, customer lifetime value, and customer acquisition cost.

However, one of the key factors when it comes to user research and retention is the fact that user research allows us to think about outcomes rather than just the metric.

If we go into the project only thinking, “We want to retain more users” or “We want our retention rate to be 70% over three months,” we are missing a key component here. With this approach, we are hugely focused on the business outcome but are missing the user outcome. Unfortunately, users won’t stay if we tell them we aim to retain 50% more of them.

Users stay with products because of the positive experience the product has had in their lives. This positive influence can come in the form of:

• Helping them achieve a goal

• Making a task more efficient

• Alleviating a recurring pain point

• Making their day-to-day life easier

• Enabling them to do a difficult task more effectively

• Meeting an unmet need that they can’t get met elsewhere

A product serves a purpose for people. By getting a clear understand of the purpose, goals, needs, and pain points, we can start to. understand how we might help people and encourage them to stay. User research is the key to developing that understanding.

Read more