---
title: Analyzing Usability Test Results: From Observations to Design Changes
description: How to turn raw usability observations into root-caused use errors, design changes, and risk file updates under EN 62366-1 and EN ISO 14971.
authors: Tibor Zechmeister, Felix Lenhard
category: Electrical Safety & Systems Engineering
primary_keyword: usability test analysis design changes
canonical_url: https://zechmeister-solutions.com/en/blog/analyzing-usability-test-results
source: zechmeister-solutions.com
license: All rights reserved. Content may be cited with attribution and a link to the canonical URL.
---

# Analyzing Usability Test Results: From Observations to Design Changes

*By Tibor Zechmeister (EU MDR Expert, Notified Body Lead Auditor) and Felix Lenhard.*

> **Analyzing usability test results under EN 62366-1:2015+A1:2020 means classifying each observed event as a use error, a close call, or successful use, root-causing every use error, deciding on a risk control action, and feeding the result back into the risk management file under EN ISO 14971:2019+A11:2021. The analysis is where the usability engineering file earns its value. Raw video and scribbled notes are not evidence until this step is done.**

**By Tibor Zechmeister and Felix Lenhard.**

## TL;DR
- Clauses 5.7 to 5.9 of EN 62366-1:2015+A1:2020 require that summative evaluation results be analyzed against the pre-defined success criteria from the test protocol.
- Every observed event is classified as a use error, a close call, or successful use. Every use error is root-caused. Every root cause points to a risk control action.
- Risk control follows the hierarchy from EN ISO 14971:2019+A11:2021: inherent safety by design first, protective measures second, safety by information last.
- Design changes driven by usability analysis require change control under the QMS. Significant changes may trigger re-testing of the affected tasks.
- The output of analysis is an updated risk management file, an updated user interface, and a usability engineering file that a notified body reviewer can follow end to end.
- Treating use errors as training problems is the classic auditor finding. Training is the last resort, not the first response.

## Why this matters

Tibor has reviewed usability reports where the analysis section was a single sentence: "All observed issues were discussed with the development team and will be addressed in a future release". That is not analysis. That is a promise to do the analysis later, and later usually never arrives because the team moves on to the next sprint.

The consequence shows up at the notified body review. The reviewer asks which use errors were observed, how they were classified, what the root causes were, what risk controls were applied, and how the residual risk was re-evaluated. If the answers are not in the file, the usability engineering evidence fails. The fix is rarely a single document edit. It usually requires re-opening sessions that closed months earlier, re-convening the risk management team, and sometimes re-testing specific tasks.

There is also a real-world consequence beyond the audit. A use error that is observed in summative testing and dismissed without analysis can surface in post-market feedback as a harm event. Tibor has seen this pattern with a device that had prolonged skin contact. A usability observation about users wearing the device longer than intended was dismissed during development. Post-market, skin irritation complaints triggered a change procedure and notified body engagement at the next surveillance audit. The analysis step is where that failure is supposed to be caught.

## What MDR actually says

Annex I §5 of Regulation (EU) 2017/745 requires that risks linked to ergonomic features and the environment of intended use be reduced as far as possible. Annex I §22 requires that the design account for the intended user population. The MDR does not say how to analyze usability data, but it does require that residual risks be reduced as far as possible, and the standard route to demonstrating that is EN 62366-1:2015+A1:2020 combined with EN ISO 14971:2019+A11:2021.

Clause 5.9 of EN 62366-1 requires that summative evaluation results be analyzed against the pre-defined success criteria from the usability evaluation plan. The analysis must identify use errors, determine whether each use error is hazard-related, and evaluate whether residual use-related risks are acceptable. Clause 5.9 is explicit that this analysis feeds into the risk management process.

EN ISO 14971:2019+A11:2021 then takes over. The risk management file must be updated with the new or refined hazard-related use scenarios, the probability and severity estimates must be re-evaluated in light of the empirical evidence from summative testing, and any residual risk that remains unacceptable must trigger further risk control action. The risk control action must follow the hierarchy: inherent safety by design first, protective measures second, safety by information last. Training is part of safety by information and is therefore the last resort.

This is the point Tibor stresses with every coached startup. The MDR requires "as low as possible" risk reduction, which is stricter than the "as low as reasonably practicable" framing sometimes found in ISO 14971 discussions. The usability analysis is where the MDR ratchet bites. If a design change would further reduce a residual use error and is technically feasible, it is required, even if the team would prefer to solve the problem with a warning label.

## A worked example

A startup was developing a benchtop point-of-care blood analyzer for general practitioner offices. Summative testing with 15 general practitioners and 15 medical assistants covered six critical tasks. Five tasks passed the pre-defined success criteria. One task did not.

The failing task was loading a sample cartridge into the analyzer. Four of 30 participants inserted the cartridge in the wrong orientation. Two of those four closed the lid and started the run, producing an invalid result that the software flagged. The other two caught their mistake before starting the run.

The analysis team met the week after testing. They classified all four events as use errors. Two were hazard-related because a repeated invalid result in a real clinic could delay a diagnostic decision. The root cause analysis identified two contributing factors: the cartridge shape was nearly symmetric and the orientation marking was a small printed arrow that was hard to see under typical office lighting.

The risk control decision followed the EN ISO 14971 hierarchy. Option one was inherent safety by design: change the cartridge shape to be mechanically asymmetric so it physically cannot be inserted wrong. Option two was a protective measure: add a mechanical keying feature to the analyzer slot. Option three was safety by information: enlarge the arrow and add a printed instruction on the cartridge.

The team chose option one. The tooling change cost about 18,000 euros and delayed production by six weeks. The revised cartridge was re-tested with eight additional participants against the same task protocol. Zero use errors. The risk file was updated, the design history file recorded the change, and the notified body accepted the updated usability engineering file without further questions.

Felix notes that the startup considered option three as a cheaper path. Tibor supported the decision to pick option one. The MDR ratchet does not care about tooling cost. If a design change is feasible and would reduce the residual risk further, it is required.

## The Subtract to Ship playbook

Analysis is where most usability files either come together or fall apart. The steps below produce an analysis that a notified body can follow and that actually improves the device.

**Step 1. Transcribe and time-stamp every session within a week.** Memory fades fast. The moderator, the observer, and the human factors lead should sit down together within seven days of each session and produce a time-stamped event log. Every deviation from expected user behavior becomes an entry. This is raw data, not interpretation.

**Step 2. Classify every event.** Each logged event is classified as a use error, a close call, or successful use. A use error is a user action or inaction that produced a result different from the intended one. A close call is a user action that did not produce a use error but would have if the user had not caught it. Successful use is what it sounds like. The classification rules must be written down before analysis starts.

**Step 3. Decide hazard-related status.** For each use error, the risk manager and the human factors lead decide whether the use error maps to a hazard-related use scenario in the risk file. If yes, the event becomes part of the risk management analysis. If no, it is still recorded and reported but does not trigger a risk control action. The decision and its rationale must be documented.

**Step 4. Root-cause every hazard-related use error.** Root cause analysis is a structured exercise, not a hallway conversation. The team identifies the contributing factors for each use error. Was it the user interface layout, the labeling, the workflow, the environment, the training of the participant, or a combination? The analysis must identify controllable factors, not just assign blame to the user.

**Step 5. Apply the risk control hierarchy.** For each root cause, the team identifies candidate risk control actions and ranks them against the EN ISO 14971 hierarchy. Inherent safety by design is the first option considered. Protective measures by functionality are the second option. Safety by information is the last option. The rationale for choosing one option over another must be documented. If the team picks a lower-hierarchy option, the rationale must explain why the higher options are not feasible.

**Step 6. Update the risk management file.** Every accepted risk control action triggers an update to the risk management file under EN ISO 14971:2019+A11:2021. New hazard-related use scenarios are added. Existing probability and severity estimates are refined with the empirical data from testing. Residual risks are re-evaluated. The risk manager signs off.

**Step 7. Trigger design change control.** Design changes driven by usability analysis go through the standard design change control process in the QMS. The change is documented in the design history file. The risk impact is assessed. Verification and validation requirements are identified. Significant changes may require re-testing of the affected tasks, either as a focused summative round or as an extension of the original testing.

**Step 8. Write the usability engineering file sections.** The final usability engineering file includes the test protocol, the raw data, the event log, the classification, the root cause analysis, the risk control decisions, and the link to the updated risk management file. A notified body reviewer should be able to walk from any single observation to its final resolution without needing help from the team.

**Step 9. Close the loop back to the design team.** Tibor has seen startups complete all the regulatory steps and then forget to actually brief the design team on what changed and why. Close the loop. Hold a short debrief. Make sure the next iteration of the device carries the lessons forward.

## Reality Check

Walk through these questions against your most recent summative evaluation. If you cannot answer yes to all of them, the analysis is not complete.

1. Is there a time-stamped event log for every summative session, produced within a week of the session?
2. Has every logged event been classified as use error, close call, or successful use against pre-defined rules?
3. Has every use error been evaluated for hazard-related status against the risk management file?
4. Has every hazard-related use error been root-caused with a documented, structured analysis?
5. Does every accepted risk control action sit at the highest feasible level of the EN ISO 14971 hierarchy, with documented rationale if a lower level was chosen?
6. Has the risk management file been updated with refined probability and severity estimates based on the empirical data?
7. Have design changes driven by the analysis gone through design change control in the QMS?
8. Have re-testing requirements been identified and planned for any significantly changed user interface elements?
9. Can a notified body reviewer walk from any single observation to its final resolution without needing help?

## Frequently Asked Questions

**What counts as a use error versus a close call?**
A use error is an action or inaction that produced an outcome different from the intended one. A close call is an action or inaction that would have produced a use error if the user had not caught it. The distinction matters because close calls indicate that the design is relying on user vigilance, which is a weaker defense than inherent safety by design.

**Can we dismiss a use error as participant-specific?**
Only if the analysis shows the participant did not match the intended user profile, which would mean the recruitment was flawed and the summative result is partially invalid. Otherwise, a use error observed in a representative participant is a design signal, not a participant problem.

**How many design iterations are normal after summative testing?**
Well-prepared teams usually need zero to two design iterations after summative testing, because the design was refined through formative work beforehand. Teams that skipped formative evaluation often need three or more, and the total cost ends up higher than if formative work had been done properly.

**Do we have to re-test every task after a design change?**
Not every task. Tasks whose critical elements were not affected by the change usually do not need re-testing. Tasks whose user interface changed materially do need re-testing, at least with a small focused sample. The judgment should be documented and can be discussed with the notified body in advance.

**Is training an acceptable risk control?**
Training is safety by information and sits at the bottom of the EN ISO 14971 risk control hierarchy. It is legitimate but only after inherent safety by design and protective measures have been considered and documented as infeasible. Tibor and Felix both see training-as-first-response as a near-automatic notified body finding.

**What if a use error was observed but the residual risk is still acceptable?**
Document the analysis, document the acceptability decision, and document the rationale. The risk control decision to accept must follow the same process as the decision to mitigate. Acceptance without analysis is not acceptance. It is silence, and silence is what notified bodies flag.

## Related reading
- [Writing usability test protocols](/blog/writing-usability-test-protocols). The protocol is the contract the analysis must measure against.
- [Usability test participants recruitment](/blog/usability-test-participants-recruitment). Representative participants are a precondition for valid analysis.
- [Risk management and usability engineering link](/blog/risk-management-usability-engineering-link). How usability findings feed the EN ISO 14971 risk file.
- [MDR design changes and iterations](/blog/mdr-design-changes-iterations). The change control process that closes the loop from usability analysis back to the device.

## Sources
1. Regulation (EU) 2017/745 on medical devices, consolidated text. Annex I §5, Annex I §22, Annex II.
2. EN 62366-1:2015+A1:2020, Medical devices, Part 1: Application of usability engineering to medical devices. Clauses 5.7, 5.8, 5.9.
3. EN ISO 14971:2019+A11:2021, Medical devices, Application of risk management to medical devices.

---

*This post is part of the [Electrical Safety & Systems Engineering](https://zechmeister-solutions.com/en/blog/category/electrical-safety) cluster in the [Subtract to Ship: MDR Blog](https://zechmeister-solutions.com/en/blog). For EU MDR certification consulting, see [zechmeister-solutions.com](https://zechmeister-solutions.com).*