The first time I handed AI a study, I gave it the transcripts and asked for themes back. What I got was a list of things the team and I could have told ourselves before we ever talked to a participant. “Users value transparency.” “Onboarding should be more intuitive.” Nothing surprising, nothing that sounded like a real person had said it, just our own hypotheses with a participant count stuck on the end.

And, folks, that’s not user research! (Thank God or we’d all have left the building a long time ago)

User research is about finding the unknown unknowns, the stuff nobody on the team even thought to ask about. It’s finding a key learning, figuring out why it’s happening based on how people think, and then working out what that means for the team.

“Users value transparency”? I could have told you that. Anybody could have told you that. It read like I’d googled it, or like the headline on some website selling something vague — synergy! Business! Circling back!

Also, who doesn’t value transparency?!

With that AI-generated synthesis, we might as well not have done the study. I could have walked over to the team, told them their onboarding sucked and they weren’t being transparent enough, and pissed people off but saved everyone a few weeks.

So I did what I do when I'm pissed off, and I went back to doing everything manually. Of course, I recorded sessions and got the transcripts, but I was rewriting, reformatting plans, creating reports and presentations completely by hand, which is just the worst thing in the whole entire world for researchers who don't have an appetite for design.

Yeah, just telling myself, obviously AI isn't going to work out. User researchers don't cut corners because look at what happened. I cut corners, and all I got was stupid, obvious themes that would make me look dumb in front of stakeholders if I presented them.

And that worked okay because I can be efficient. I've been doing this for a long time so I know where I can speed up my process. I can write research plans and screener surveys in my sleep. It was okay until stakeholders started coming to me and asking why everything was taking so long when they could just post the transcripts into Claude and get answers back in a matter of hours.

You know what sucked? I had absolutely nothing to say.

Yes, I could’ve said, "Oh well, it's going to give you back shitty themes that are absolutely flat and don't really mean anything. They don't tell you anything that's actually helpful. They kind of actually just give you a pat on the back of, like, 'Well, I kind of knew our onboarding sucked, and we had to change it.'"

I could have said that to the stakeholders, and I guarantee that they would have said, "Well, does that matter?"

So I went looking for why AI kept handing me the most obvious version of every theme.

Let’s say a pet insurance company wants to know whether their claims process is too slow. You talk to 12 pet owners who’ve filed a claim this year, hand the transcripts to AI, and get this back:

Users want a faster, simpler claims process (9 of 12 participants)

True? Yup.

Useful? Not even a little.

I could tell you, when it comes to insurance, every single person wants something that's faster and more simple. I could tell you, every single time it pretty much comes to anything that people have to do, they want things to be faster and more simple. People want faster horses.

(Also a quick aside — if you ask people what they want, they will give super generic answers. Additionally, if you ask AI what people want, it will give you super generic answers because it’s trained on what people are responding to when they are asked what they want which is, drumroll please, SUPER GENERIC. We need user research to pull out the real stories, mental models, the things people don’t even know they need until we’re talking about it in an interview).

Anyways, if we would look deeper into that pet insurance research, with a human eye, we might find:

What’s going on underneath has very little to do with speed. These owners are scared, and every time they file a claim, they’re scared of being told no at the moment they can least afford it, of being treated like they’re lying, or of losing their claim to a form that eats their work. So they protect themselves. One pays the small bills out of her own pocket so she never has to feel disbelieved again, and another keeps a folder of evidence like he’s preparing for a dispute.

And once someone has stopped trusting you to show up when their pet is sick, they start asking why they’re paying you every month, and they answer that question at renewal. A faster form doesn’t touch any of that. The team would have spent a quarter making it quicker for people to file claims they no longer want to file.

AI gave us the obvious version since it writes by predicting the most likely next words.

So when it summarizes 12 interviews, it reaches for the most likely sentence about them, and the most likely sentence about any claims process ever is “people want it faster and simpler.” It counted “fast” and “easy” nine times. It didn’t hear her laugh when she said she’d rather eat the cost, or that the screenshot guy sounded exhausted rather than annoyed. And since the brief asked whether claims were too slow, it went looking for slow (and, surprise, it found slow!).

Now I run what I call the swap test on every AI theme. I swap the company for something totally different, like a bank, a gym app or a food delivery service, and if the theme still sounds true, it’s the obvious version and I go back to the transcripts. “Users want a faster, simpler process” works for all three. Yikes.

Okay, now I’m going to slightly contradict myself (bear with me).

The obvious version isn’t always a disaster. If the pet insurance team is deciding whether the button says “Start a claim” or “Get reimbursed,” that’s a small, low-risk change they can test and roll back next sprint, so a slightly generic theme isn’t going to send them far off course. But if they’re deciding whether owners have to send an itemized vet invoice before any claim gets paid, a generic theme can send them off building the wrong thing for people who are already scared.

So I stopped asking “which tasks can I hand to AI?” and started asking “how big is the decision this study is feeding?” That’s what tells me how much AI I can get away with, and how much checking I need to do.

At kickoff, before anyone even says the words “method” or “tool,” I ask the stakeholder two questions and score each one from 1 to 3:

Then I multiply them together (I promise this is the only math in this newsletter).

Decision size = how hard to undo × who is worse off

The button? 1 × 1 = 1. We can change it back next sprint, and nobody’s crying in a car park over it.

The invoice rule? 3 × 3 = 9. The people it hits are the ones waiting on money for a $2,000 surgery, and nobody, and I mean nobody, renews with the company that held up their claim during the worst week of their pet’s life.

Then the size tells me how much to hand over:

So the button study gets done in a few days with AI taking the first pass (huzzah!), and I take all that saved time and sit with every single claims session myself, listening for the anxiety and fear, for the unknown unknown, the mental model that pushes people to make decisions.

And stakeholders love this because I am able to push smaller things for them to change, leading to the feeling of productivity while the bigger more high-risk information gets the time it deserves.

Once I know the size, it goes in the research plan as one sentence, so the stakeholder sees my reasoning before we start instead of asking me at the readout why it took so long. Here’s my fill-in-the-blank version.

This decision is [easy / hard] to undo, and if we’re wrong, [who is affected and how]. So [tool] will [what it does], I’ll [what I do], and the readout will be [timeline].

And here’s what it looks like for the invoice rule.

This decision changes how claims get paid, so it’s hard to undo, and if we’re wrong, owners wait longer for money they need during their pet’s treatment, and some won’t renew. So our research tool will tag the transcripts and pull quotes, I’ll do the interpretation myself and link every insight to the sessions behind it, and the readout will be in three weeks.

I do the same thing in the readout. Under each finding, I add a little label so everyone knows exactly how it was made:

Then I add one slide on what we didn’t hear, like “nobody in this sample had a pet over ten years old, so none of this covers senior pets.” Stakeholders trust the findings so much more when they can see what I checked, and nobody has to awkwardly ask, “did AI write this?”

Remember how I had absolutely nothing to say? Now I do (finally!). Having the size written down means I’m pointing at something we agreed on together, instead of getting defensive about my process. These are the answers I keep coming back to:

And if someone still isn’t buying it, show them the swap test on an old study they already know. Put the AI themes next to what you reported, swap in another company’s name, and let them see for themselves which version still sounds true. It’s a pretty fun moment when they realize the AI version works for a gym app too.

Overall, try the two questions at your next kickoff, and run the swap test on the next AI theme you get back. If it still sounds true for a completely different company, you know exactly where to go digging!

This week you get three downloads, all markdown files you can read on their own or upload to Claude, ChatGPT or whichever tool you use.

I recommend running the Swap Test on the last AI synthesis you got back. It’s a pretty eye-opening way to see what got flattened!

Exclusively for paid subscribers: