Qualitative vs Quantitative research: An inexplicable eternal misunderstanding in UX and not only!

If you Google “Qual vs Quant UX research” you will come up with millions of results. Well-known institutions, service providers and experts in the field of UX among others still publish articles comparing the merits of qualitative and quantitative research methods. Even worse, there is a controversy still taking place in academia among researchers about the value of each approach. And so the question inevitably emerges “why is this still happening? Why do people still not get the difference?”.

But things are really simple. The handsaw and the hammer: as simple as that

Both tools have been designed to perform a completely different job. It would be a logical mistake to mention that the hammer is better than the handsaw. A handsaw is perfect for cutting hard materials such as wood and the hammer is awesome for driving in nails. The hammer can’t do the job of the handsaw and vice versa. These tools creatively combined by someone who has been trained to use them without hurting him/herself or the materials, can give us a table.

The same applies to qualitative and quantitative research methods. They are both tools that perform different tasks. When used creatively by people who have been trained to do so without “hurting” the data and so the quality of the study, they can answer research questions and inform business decisions based on data, not assumptions!

Qualitative research methods

To put it in a nutshell, qualitative research methods analyze any kind of non-numerical data such as words spoken in the format of an interview script or observations such as facial expressions, body language, breathing rate, gestures, emoticons, and symbols, not to mention any kind of human activity that can’t be captured and so expressed through numbers i.e., art pieces, rituals, hierarchies etc.

Qualitative data are typically collected by interviews or observations. The goal of a qualitative researcher is to collect and analyze behavioral (what subjects of study do) and attitudinal data (what subjects under study say they do) aiming at uncovering behavioral patterns. This goal is achieved by collecting data through open-ended “why”, “how” and “what” questions from small samples of interviewees in search of meaning; the Interviewee’s meaning, not the researcher’s …

The most common qualitative UX research methods are:

• Interviews
• Moderated usability testings, and
• Field studies such as
  • usability testing,
  • Guerilla usability testing
  • contextual inquiry

For example, qualitative research methods could be used by a UX research team to answer questions such as: “How do our older users experience their interaction with our brand new touch screen ATMs?”

In that case, a good research method would be usability testing sessions combined with follow-up user interviews. Interviews could feed the research team with data about their users’ beliefs, perceptions, and experiences while user testing could unravel behaviors i.e., what users actually do vs what they say they do. Be always aware of that gap!

Examples of research questions you can approach with qualitative methods

• How do users experience their interaction with our vendor machine?
• What do our users believe about the usability of our new product feature?
• What are our users' first impressions of the homepage?

Quantitative research methods

On the other hand, quantitative research methods strictly analyze numerical data such as the total number of users, the total number of errors  (discrete data), or time on task (continuous data). However, in quantitative research, it is also possible to handle qualitative data that could take a numerical form such as gender, marital status, region, etc. (nominal data) or NPS scores, level of education, level of happiness, etc. (ordinal data). Quantitative researchers aim at answering the “how many'' and “how possible is it ''.

In other words, quantitative research methods help researchers measure, and describe human activity plus predict human behavior across the total population making inferences from smaller groups of people (samples).

To sum up, their major goals are to (a) describe through statistics such as means, medians, mode, etc. the characteristics of a sample (descriptive statistics) and (b) to make inferences from these samples about the general population (inferential statistics).

The most common quantitative UX research methods are:

• Surveys
• Quantitative usability testing (> 30 participants)
• A/B testing
• Tree-testing (can be both quant and qual)
• Card-sorting (can be both quant and qual)
• Web analytics, eye tracking etc. 

Quantitative UX research methods could be used by a UX research team to answer questions such as:

• How much time does it take on average for our users to check out?
• What is the average number of errors that our users make during a sign-up process?
• Females or males are more likely to use that product feature?

Mixed methods

Last but not least, both methods can be used creatively in combination with one another to give a deeper understanding of our users’ behaviors and attitudes. A common application of mixed methods is the methodology used to define our users’ personas based on both qualitative and qualitative data. The first step in such research projects is to conduct interviews and observations with our users to identify key behaviors, beliefs, and pain points that differentiate one cluster of users from another. Workshops with knowledgeable stakeholders could further validate or even elaborate that step.

The second step is to survey a large population of our existing or prospective users making use of questions that represent findings that emerged from user interviews, observations, and workshops. Data collected from a large number of respondents capable of fueling the study with enough statistical power should be further processed by any statistical tool making use of the appropriate cluster analysis technique.

Finally, the research team will have the opportunity to validate through statistics the user personas that emerged from user interviews and observations.

To sum up

It is a shame that there is still a controversy or even a lack of understanding among professional researchers about the value and usefulness of each research approach. Often quantitative research methods are perceived to be more sophisticated. Both methods are two completely different tools specially designed to serve different research purposes. Comparing qualitative to quantitative research methods is a logical mistake similar to comparing hand-saws and hammers. They perform different jobs but use appropriate and sometimes, creatively combined, they can give us more insights than each separately.

Happy researching!