5 or more participants when doing user interviews? Keep calm and look for theoretical saturation

Sidebar

by Vasileios Kalyvis, PhD. follow at Linkedin

31/08/2022 • 8' read

I am sure you have all come across that avalanche of articles about the question “how many users?” in usability tests and user interviews over and over again. Why is this still happening? Why are all these articles still coming into our inbox or Linkedin feed? I hold the view that one of the reasons behind that phenomenon is that we, UX researchers, haven’t really understood how sampling works when it comes to qualitative research. Instead, we are often looking for quick fixes on Google without even going through the articles most of the time. Business and technology move fast, right? Too little time to study? This time you will have some tips from science on a single page.

In quantitative research, it is crystal clear when we should stop our A/B testing sessions or what is the minimum number of users we need for our quantitative remote unmoderated usability tests. But this is not the case in exploratory research.

It is impossible to know in advance the exact number of users we need to interview.

Any other answer to “how many participants?” is welcomed and respected especially when it comes after years of experience in the field of UX research or product management and development. However, it would still be vulnerable from a strictly scientific point of view. Since we apply science to solve real-world problems, let’s put aside the rule of thumb of 5 users and see how sampling really works in qualitative research.

- "Oh no! … What have I been doing all these years? … and what should I do?... What’s the rule?”

In exploratory research, we stop interviewing and, thus, recruiting new participants once we have achieved what we call theoretical saturation.

Theoretical saturation is a state where your themes are saturated with data to such an extent that it would be difficult to get some new insights from any additional interview.

That’s why it is not possible to know in advance how many participants we need to interview. If you stop your interviews before theoretical saturation is achieved you put at risk the quality of your study and so your product’s success. If you do more than needed you spend precious time that could be invested in the analysis of your data.

- “How can I know if I have achieved the so-called theoretical saturation? Is there a metric or something?”

The only one who knows is you and your research team, in case you work with others. There will be a stage in your research journey where you will have, let’s say, 10 interviews done, and realize that you haven’t actually learned something new from interviews number 9 and 10. That’s a strong indication that you have probably achieved theoretical saturation. In that stage, you have the confidence to say “I have adequately answered my research question! I feel I have really been in my users’ shoes!”.

- “Yes, but what should I do if it is very difficult to recruit new participants?”

Sometimes due to the very special characteristics of the group under investigation, it is difficult to recruit or even find new participants. For instance, consider the case where you study people with weight-gain, not weight-loss, issues. People with that kind of health issue are just a few and so difficult to recruit in order to achieve theoretical saturation. To your surprise, your very last participant gave you a perspective that none of the previous ones did and you desperately want to recruit new participants to find out if this applies to other people too. In that case, you don’t necessarily have to find additional participants. Go back to your existing participants and ask them 2-3 questions about that new perspective given by your very last participant. That iterative process will help you make your themes concrete and so achieve theoretical saturation.

Last but not least, there are a number of factors beyond theoretical saturation that affect the number of users interviewed and the time spent until theoretical saturation is achieved.

The depth of knowledge you want to gain. The greater the depth the larger the number of interviewees you want to interview. In case you explore a very specific group of people that it is difficult to gain access to, you can come back to your existing participants and probe even more!
The level of interviewer’s experience. The more experienced the interviewer is in interviewing and qualitative data analysis, the earlier the theoretical saturation is achieved and so the smaller the number of participants interviewed.
The extent of the literature review (aka desktop research) conducted. Before you start any research about the needs, expectations, and experiences of your users make sure that there is no same or relevant research already conducted out there! Before you design your interview guide, go to Google Scholar and spend 2 days reviewing the existing literature. Your answers may already be out there! That strategy will save you a tremendous amount of time and it will probably help you reach your research goals with fewer participants.
Desktop research can also help you build a semi-structured interview guide rather than start your research with a totally unstructured interview. That happens because some themes and categories will be already there as a result of your literature review. The more structured the interviews the less the number of participants required until theoretical saturation is achieved.
The level of participants' experience over the topic under research. For example, if you are working on a digital product that aims at supporting people with dementia, go straight to people such as nursing staff, carers and relatives who closely interact with them on a daily basis and for a number of hours. You’d better avoid people who don’t interact closely or do so on a rare basis with dementia patients or people who only possess theoretical knowledge. Such a strategy will help you minimize the number of participants needed until you achieve theoretical saturation.