Accessibility in User-Centered Design: Reporting Usability Testing
The Evaluating for Accessibility page provides guidance on incorporating accessibility into common evaluation methods, including standards review, heuristic evaluation, design walkthroughs, and informal evaluation with users with disabilities. The Usability Testing section is an overview of usability testing with participants with disabilities.
Analyzing data from and reporting on a usability study that includes participants with disabilities involves the following considerations:
- Distinguishing Between Accessibility and Usability Issues
- Including Relevant Study Parameters
- Being Careful about Categorizations and Comparisons
- Clarifying Conclusions
- Writing about People with Disabilities
In addition to finding accessibility problems, usability testing with participants with disabilities will find general usability problems that impact all users, that is, including users without disabilities.
Understand the difference between accessibility and general usability. The "Accessibility related to usability" section in the Background chapter discusses the relationship between accessibility and usability. There is not a clear distinction between accessibility for people with disabilities and general usability for all. Some things are clearly accessibility; some are clearly usability; and many things are in a gray area where accessibility and usability overlap.
One way to start looking at the distinction between the two is to categorize interface problems:
- Usability problems impact all users equally, regardless of ability; that is, a person with a disability is not disadvantaged to a greater extent by usability issues than a person without a disability.
- Accessibility problems decrease access to a product by people with disabilities. When a person with a disability is at a disadvantage relative to a person without a disability, it is an accessibility issue.
The distinction between usability and accessibility is especially difficult to define when considering cognitive and language disabilities. Many of the accessibility guidelines to improve accessibility for people with cognitive disabilities are the same as general usability guidelines. The distinction is further blurred by the fact that features for people with disabilities benefit people without disabilities because of situational limitations (that is, limitations from circumstance, environment, or device—such as using a mobile phone in bright sunlight with one hand because you're holding a sleeping baby with the other), and accessibility increases general usability.
Another point to cloud the distinction is “usable accessibility”—how usable are accessibility solutions. For example, if a website uses images for navigation and there's no alt text, the site is clearly not accessible. If the site has frustratingly verbose alt text (such as "This image is a line art drawing of a dark green magnifying glass. If you click on it, it will take you to the Search page for this Acme Company website" instead of just "Search"), one might say that the site is technically accessible because there is alt text; however, the alt text is so bad that the usability of the site is awful for anyone who relies on alt text.
Distinguish between usability and accessibility issues, as appropriate. When designing products, it's rarely useful to differentiate between usability and accessibility. However, there are times when such a distinction is important, such as when looking at discrimination against people with disabilities and when defining specific accessibility standards. In some usability test reporting it may be important to distinguish between accessibility and usability problems.
When usability test reports are used internally to improve the usability of the product for all users, it is usually not necessary to distinguish between usability and accessibility issues. However, when usability test reports make statements about accessibility, it can be vital to distinguish between usability and accessibility issues.
There can be problems when people don't understand the issues around the distinction between usability and accessibility.
A research study reported results on website accessibility without clearly separating general usability issues not related to accessibility. Because usability issues were mixed with accessibility issues, the study reported incorrect conclusions about web accessibility guidelines.
Academic discussions of accessibility and usability can actually harm the cause of accessibility if they are not presented carefully for people who don't understand the complexity of the issue.
Include relevant study details in published reports of usability testing with people with disabilities. In order for readers to understand and use study data, you may need to include more details than usual on the scope of the study and the evaluation parameters, such as testing methods and user characteristics. For example, in some study reports you may only generally list the software used; however, in a study that includes screen readers, it is usually important to also list version numbers and even customization settings because these impact study results and their applicability to other situations.
Include relevant participant characteristics in the report, and don't include irrelevant participant information. Generally internal reports for product design need little participant information; whereas a public research report might need more.
A participant's specific medical diagnosis and personal history are usually not relevant. Information that usually is relevant includes the adaptive strategies and assistive technologies used and the participant's experience with similar products. For example, that the participant has been blind from birth and has been using a mobile phone for 5 years is relevant; whereas, the medical condition that caused blindness is probably not relevant.
Confirm that any categorizations are appropriate and useful. See "Careful with categorization" in the Planning Usability Testing section.
Confirm that any comparisons are appropriate and useful. Be careful about reporting comparisons, especially with small sample sizes.
In one case, a published usability test report compared data between participants with disabilities and participants without disabilities. The products evaluated had poor accessibility and thus the results showed poor performance by participants with disabilities. Had the study included accessible products, it could have shown that it was the inaccessibility of the products that caused the poor performance, instead of leaving some readers to conclude that participants with disabilities always have poorer performance.
Clearly indicate what the report asserts and what it does not assert, especially when people who are not familiar with accessibility evaluation will likely read the report. For example, if a study included only participants who are blind, its report should clarify that it did not evaluate conformance to accessibility guidelines and that it does not apply to all people with visual disabilities, nor people with other disabilities. Thus the report can help readers draw appropriate conclusions.
Avoid the pitfall that a few participants represent all people with disabilities, as explained in the "Individual Differences" section of Analysis Phase.
Successful usability test results do not guarantee that the product is accessible to all people with disabilities, nor that it conforms to accessibility standards. Even large-scale usability studies cannot cover the diversity of disabilities, adaptive strategies, and assistive technologies. Use other evaluation techniques described in Evaluating for Accessibility to help cover the diversity, and standards review to determine if a product meets standards.
Use appropriate language to refer to people with and without disabilities. See "Use 'people first' language" and "Avoid potentially offensive terms or euphemisms" in the Interacting with People with Disabilities chapter for more guidance related to writing about people with disabilities.
When referring to participants who are not disabled, use "people without disabilities" or "nondisabled." Avoid "normal," "able-bodied," and "healthy," in contrast with people with disabilities. 
In writing usability reports, don't portray a successful participant with a disability as superhuman or a hero. Report about participants with disabilities in the same tone that you would use in writing about any usability test participant. 
The next section provides a reference for this chapter on usability testing with participants with disabilities: a checklist.
- Guidelines for Reporting and Writing about People with Disabilities. The Arc of the United States.