Key requirements for a medical device summative usability test
A summative usability test provides final evidence that a medical device can be used safely and effectively by the intended users in the intended use environment(s). Specifically, a summative usability test is intended to validate that all implemented mitigations in response to identified high-severity use-related risks (i.e., critical tasks) are reasonably safe and effective.
The list below includes essential requirements every medical device company should adhere to when preparing for a summative test.
1. Ensure the evaluated device is production-equivalent
The product samples evaluated during the summative test should be production-equivalent. This includes all evaluated materials such as the medical device, its labeling, packaging, IFU, and training materials (if applicable). The use of production-equivalent samples is essential because potential use errors linked to a non-commercial version of the product might not be representative for the validation of the device.
2. Ensure the use-related risk analysis is robust
Before writing the summative test protocol, ensure the use-related risk analysis is robust:
- Identify all use-related hazards.
- Define all critical tasks (i.e., user tasks that could cause serious harm if not performed correctly or not performed at all).
- Ensure the summative test directly evaluates the identified critical tasks during representative naturalistic evaluation activities.
- If a critical task cannot be identified in a naturalistic approach, consider implementing knowledge tasks.
3. Define user groups, sample size, and test location
The following table provides an abbreviated summary of the intended user groups, sample size, and test location requirements for a summative usability test for either an FDA, MDR, or NMPA submission.
| Market | Participants | Test location |
|---|---|---|
| United States (FDA) | A minimum of 15 participants per distinct user group | United States |
| Europe (MDR) | No strict participant requirement | No strict location requirement |
| China (NMPA) | A minimum of 15 participants per distinct user group | China |
4. Include representative training
If a medical device company intends to provide formal training to end users during real-world use (e.g., onboarding sessions, certified training, in-service demonstrations, instructional videos), then the summative usability test should include a training session that reflects this intended training. Training is considered part of the user interface, and the validation must evaluate whether trained users can safely and effectively complete all critical tasks.
5. Focus on critical tasks
Every identified critical task from the use-related risk analysis should be be evaluated. These tasks must be clearly identified during the use-related risk analysis and evaluated thoroughly during the summative usability test. Each critical task should have predefined success criteria, and any observed use error must be analyzed for its root cause and residual risk. The goal is to demonstrate that all design and labeling mitigations introduced during the medical device development effectively reduce the likelihood of use-related harm.
Conclusion
A well-designed summative usability test provides the final confirmation that a medical device can be used safely and effectively by its intended users. To meet global regulatory expectations, manufacturers must evaluate a production-equivalent device, complete a thorough use-related risk analysis, and ensure the summative usability test evaluates all identified critical tasks. The test must include representative users, appropriate sample sizes, and meet test location requirements. If applicable, representative training that mirrors real-world use should be included. By aligning these elements, the summative evaluation accurately reflects actual conditions of use and provides a framework to demonstrate that the implemented design and labeling mitigations successfully reduce use-related risks.
At Human Factors Insight, we are passionate about all things human factors and usability related.
Reach out by using the chat button or contact us via e-mail if you have any questions!