AI in UX Research (UXR): A Junior Messiah

AI is everywhere as if it were the new omnipresent God. AI is the latest buzzword capable of making more people buy a product just because it is “AI-assisted!” Hundreds of courses can be found out there promising to train people to welcome the new Messiah so that people can speak its language and not be left out of the new Noah’s Ark because of their AI illiteracy. But none of these realities hurts more than the fact that some user-centered Gods and Goddesses are flooding my inbox trying to convince me that this beta stage technology of negotiable usability and trustworthiness, is my new UXR buddy. Negotiable if not false.

UXR work AI does not perform today based on my experience

To the best of my knowledge (by mid-September 2024):

1. There is no tool that can conduct any sort of evaluative UX research. No AI wizard can analyze a video and identify usability issues after a UX researcher uploads a video from a usability testing session.
2. No AI-assisted wizard can capture and analyze user silences, gestures, facial expressions, or even read between the lines of what people say. I don’t believe all people, even UX researchers, are empathetic by default or even trained to be empathetic. However, I haven’t witnessed any greater empathy than that from any AI language model.
3. No AI-based tool can break down transcripts into codes, categories, and themes to identify patterns and generate insights or theories in social sciences. If it’s possible to happen and we haven’t made it yet, then we just came up with a major usability issue.
4. Not to mention how disappointing it is to apply AI-based clustering features for sorting post-it notes in some of the most broadly used collaboration tools. Most of the time I found myself trying to fix the clustering mess that a well-promising feature created.
5. No UXR tool can conduct a full heuristic evaluation or card-sort analysis that goes beyond the surface to provide insights. But even if I am missing something and there is still such a tool out there, I don’t think that you would use it without going back and double-checking yourself.
6. Last but not least, I am sure that we have all witnessed user research tools or even expensive survey platforms attaching inaccurate tags (likes, dislikes, suggestions, expectations, etc.) on parts of our transcripts or open-ended survey questions.

I am not going to say even a word about the idea of interviewing synthetic users, i.e., AI faking real users…

 UXR work AI performs today (always under human supervision…)

1. Needless to say how helpful it is having AI-based tools doing the transcription work for us. Although mistyping is common, these tools are executing a heavy-lifting work for us saving huge amounts of time that could be invested in other more meaningful activities.
2. As a non-native English speaker, AI-based products have helped me improve my writing and communication skills while reducing the time spent on reporting. We know that critical communication skills are essential in our field. Also, picking the right tone of voice based on who your audience is is “a nice to have,” too.
3. Broadly used AI language models such as ChatGPT have also been proven to be legit UXR assistants when it comes to executing lightweight high-level qualitative data analysis, such as summarising likes/dislikes or key points of an interview.
4. The same applies to creating interview guides. Well-written prompts can generate good interview questions and serve as a starting point for further refining your interview guide.

Best practices

1. Don’t treat AI tools as if they were the trustworthy UX colleague or mentor you have been missing! Instead, treat them as a junior researcher who is capable of executing some lightweight tasks under supervision. Don’t worry; AI will not complain about micromanaging it!
2. Always double-check the output of any AI tool even when it has to do with light-weight tasks such as summarizing a user interview or identifying users’ likes/dislikes. Ask the system to accompany each like/dislike with the relevant quote and you will have them!
3. When you give it search-based prompts such as “What is the current state of video resumes in Europe?” always ask the system to reference its resources!

Account

Trust in technology means not going back to double-checking the machine's output. Have you ever caught yourself double-checking the output of your pocket calculator with pen and paper? Have you ever caught yourself double-checking Excel’s or SPSS's calculation accuracy? That said, I would not put my faith in AI, but I would trust it at that point when there was no need to go back and double-check its output. To sum up, the Messiah is not still here. And if it is here, it is a junior and still requires some training before he walks over the sea of UXR.

Happy researching!