A Guide to Pairwise Comparison

When you need preferences without asking for people's preferences

👋🏻Hi, this is Nikki with a 🔒subscriber-only 🔒 article from User Research Academy. In every article, I cover in-depth topics on how to conduct user research, grow in your career, and fall in love with the craft of user research again.

---

If you know me, you know that I absolutely hate asking people for their preferences.

Understanding user preferences can be tricky because they are so subjective and influenced by many factors. Imagine asking someone if they prefer chocolate or vanilla ice cream. They might say chocolate because they love the taste, while another person might choose vanilla because it reminds them of childhood memories.

The same complexity arises in design choices. If we ask users whether they prefer a blue or purple website background, it’s a decision they usually don’t make. Or, consider asking if they prefer a large image banner or a smaller one with text. These are not typical choices users face, and their responses can be inconsistent and hard to interpret.

I actually tried this with my mom while she was trying to purchase furniture online for their new house. I showed her two different furniture websites and asked her which she preferred. She told me she preferred the second option more because it had fewer sponsored/ad-based search results. I then proceeded to watch her browse for a little bit, pick a sponsored search result and buy that product. Huzzah.

Plus, preferences might not align with best design practices. Someone might like a large image banner, but it could pose accessibility issues. This makes designing solely based on user preferences challenging, as it’s hard to understand the reasons behind their choices and ensure those choices work well for everyone.

When it comes to comparing different versions of ideas or products, I tend to go with usability testing. The version that is more effective and efficient will generally be the one that people end up using more…

However, as we’ve been asked hundreds of times as researchers, sometimes we have to compare certain aspects without the aid of usability testing.

Sometimes we have to understand a degree of preference — as much as I hate writing those words.

But that doesn’t mean that we have to ask people whether or not they prefer something. Instead, introducing: Pairwise comparison.

What’s Pairwise Comparison?

Imagine I gave you a list of ten songs you’ve listened to recently and asked you to rank them in order of preference. You, like most people, might really struggle to fill out this survey accurately.

I love Taylor Swift’s “Love Story,” but do I like it more than Queen’s “Bohemian Rhapsody” or The Beatles’ “Hey Jude?” And then where does Adele’s “Rolling in the Deep” sit within all of this?

You might struggle, trying to decide if you like Taylor Swift’s “Love Story” more than Queen’s “Bohemian Rhapsody” or if The Beatles’ “Hey Jude” tops Adele’s “Rolling in the Deep.” But if I give you two songs at a time and ask which one you like more, it becomes much easier.

This is pairwise comparison in a nutshell. This method simplifies decision-making by breaking down complex choices into manageable comparisons, which then become a series of head-to-head votes. It’s rooted in the idea that humans find it easier to make decisions when faced with two options rather than many.

Pairwise comparison is a fantastic technique for ranking, prioritizing, and comparing options efficiently and effectively.

Let’s look at an example:

Food for a Party

Imagine you are planning a party and trying to figure out the best food to serve your guests. One way you could figure it out is by having people force-rank all the different options. However, as mentioned above, force ranking gives people a lot of different options to choose from, whereas pairwise comparison makes these rankings easier for users.

Let’s look at the options for food:

1. Pizza

2. Chips

3. Chocolate cake

4. Sushi

5. Fruit Salad

6. Chicken wings

We would then go on to create survey questions to compare each option:

1. Pizza vs. Chips

2. Pizza vs. Chocolate cake

3. Pizza vs. Sushi

4. Pizza vs. Fruit salad

5. Pizza vs. Chicken wings

6. Chips vs. Chocolate cake

7. Chips vs. Sushi

8. Chips vs. Fruit salad

9. Chips vs. Chicken wings

10. Chocolate cake vs. Sushi

11. Chocolate cake vs. Fruit salad

12. Chocolate cake vs. Chicken wings

13. Sushi vs. Fruit salad

14. Sushi vs. Chicken wings

15. Fruit salad vs. Chicken wings

Here’s what one of those questions would look like:

The total number of possible pairwise comparisons from a list of options is n(n-1)/2, where “n” stands for the number of options in the set. In the above example, we have six examples, so the formula would look like 6(6-1)/2. Therefore, there would be 15 combinations to ask about, so there would be 15 total survey questions.

In a pairwise comparison survey, one participant can be tasked with voting on every possible pair combination or a bunch of people can each be given a sample of pairs to complete that can later be used to calculate the group’s overall combined ranking.

Different Types of Pairwise Comparison Tests

Pairwise comparison fundamentally involves head-to-head voting, but there are several ways to customize your survey to suit different needs. These formats can be used independently or combined for more flexibility.

Complete Pairwise Comparison

In a Complete Pairwise Comparison, each respondent is shown every possible pair from the list of options. This method provides a detailed and accurate representation of an individual’s preferences. The total number of possible pairwise comparisons is calculated using the formula n(n-1)/2. For example, if there are 10 options, the total number of pairs is 10 \times 9 / 2 = 45. This then means the respondent is going through 45 survey questions.

45 survey questions?! That is wayyyy too much.”

That’s what crossed my mind the first time I heard about this approach. I try to limit my surveys to 10 questions, or at least surveys that take less than seven minutes to account for increased attrition if you hit the seven-minute mark. 45 questions felt like the participant would be there all day taking the survey, or, worse, that no one would take it.

However, pairwise comparison is pretty fast — I recommend you try taking a pariwise comparison survey to see just how quickly questions can go — and since the participant is only choosing between two options at one time, the amount of time and cognitive load for each question is fairly low.

Example:

Imagine a small team of designers choosing their favorite colors for a new logo. With only 10 colors to choose from, each team member compares every pair to express their preferences clearly.

Complete Pairwise Comparison is ideal for small groups, individual preference ranking, or surveys with a limited number of options (usually up to 15-20).

Partial Pairwise Comparison