Usability Testing with users with disabilities

Usability testing is the cornerstone of UX research methodologies. Running such a test is always a demanding process with plenty of challenges. As we are trying to design and develop truly usable digital products, in other words, products that are accessible to as many people as possible, it is imperative to do testing with users with disabilities. Moreover, these tests come with their own particularities, issues, methodological and technical details that require special attention.

Having performed many usability testing studies with users with disabilities I wanted to share a simple list of issues (certainly not an exhaustive one) that need extra consideration.

Before the test

First, never skip this step (it is not "nice to do", is an absolute requirement):

• Before conducting the usability test, an expert-based manual test must have taken place along with the required remediation efforts so that we can be sure there is an acceptable level of accessibility already in place. In other words, make sure that most of the critical and major accessibility issues have been resolved. It is a matter of respect to the users that you are calling to participate in the test. If you cannot do it or do not have the resources, call external accessibility experts to do this for you.  And remember: Α quick scan with an automated tool is NOT an alternative for this.

Timing and user tasks

• While planning the test, beware the time given to users with disabilities to complete each task should be at least double the time we would give users without disabilities.
• We have to make sure that user tasks and scenarios "match" the type of disability of each user
• It is better that you use specific testing scenarios with a clear beginning and end. Even if your research goal requires that you test open-ended user scenarios, try to make them look as if they had a clear endpoint.
• We must test beforehand all the user tasks and scenarios that will be given to the users and we must be sure that they can be performed with the appropriate assistive technology.
• No matter the nature and complexity of the product under evaluation and the tasks, it is always insightful to put special emphasis on navigation and functional components such as input forms (if exist), links and buttons.

Communication and preparation

• Make sure to send meeting invitations with the proper instructions. First, be 100% transparent on the process and the method of usability testing. This is very important before delving into other technical details. The instructions need to be clear and precise so that participants can follow them easily and without stress.
• Do not forget that the file with users’ consent must be accessible and comprehensive.
• Do your preparatory work with the assistive technologies that users will use during the test. Get familiar with them, and use them as much as you can.

During the test

It may seem obvious but during testing, users should use the machine, operating system, browser, and assistive technology that suits them, just as they would outside the testing context.

Moderating

• [Thinking aloud protocol] This is not something new, we all know the importance of thinking aloud protocol, i.e. having users express verbally their thoughts while they are interacting with the digital product under evaluation. However, we want to further stress the importance of it when testing with users with disabilities. Try your best to make participants feel comfortable with it.
• The moderator must give feedback for specific actions. For instance, (in the case of a remote test using a video call platform) a screen reader user will not know if you can see his/her screen or not. In order to keep them focused and calm, give them feedback to let them know whether they have performed the screen sharing correctly.
• When starting the test session, allow the user a couple of minutes or more to «scan» the digital product they have to test. They can take the time to orient themselves, to have a basic understanding of the complexity and depth of the product and the related user tasks that will be performed during the test. As a result, users feel more calm and confident about the process and this increases the chance for researchers to get valuable feedback from them.
• Be careful with the wording (as moderator) and communication of the user tasks. It is important so that the users can realize whether they completed a task or not. 

After the test

• Be extra careful with the analysis.  You have to analyze both accessibility issues and usability issues. They are not identical.
• When the analysis is done, it is a good practice to send a short memo of the main findings (not a detailed report or deliverable) to the participants.
• When remediation is done send them a message to notify them about the changes and ask them for additional feedback (if any).

All of the above takes time and practice. It is ok to do mistakes. Just make sure that you respect the time and effort of the participants by being well-prepared. And show them that you really care about their experience.
Happy testing!