There is no MDR clause that says "your AI must be explainable." What MDR does require — in Annex I §1, §8 (benefit-risk), §17.1 (software lifecycle) and Article 61 (clinical evaluation) — is that risks are reduced as far as possible, that benefit outweighs risk, and that your clinical evaluation demonstrates safe use. Explainability shows up as an emergent notified body expectation because it is often the only credible way to meet those obligations for a model a user must trust.

By Tibor Zechmeister and Felix Lenhard.

TL;DR

  • MDR contains no specific explainability article. Auditor expectations derive from the risk, benefit-risk, and clinical evaluation obligations.
  • Notified bodies treat explainability as risk-control evidence: how does the user know when to trust an output?
  • Model cards, decision logs, confidence indicators, and worked examples in the IFU are the practical artefacts.
  • Framing explainability as ethics or marketing will fail. Framing it as risk control under EN ISO 14971:2019+A11:2021 will not.
  • Black-box models are not banned. The burden of proof for safe use is just higher.
  • The expectation is rising. A file that met the bar two years ago probably does not meet it today.

Why this matters

We will be honest about something many regulatory blog posts are not: there is no MDR article you can cite that imposes a specific explainability requirement. Anyone who tells you "MDR Article X requires explainability" is paraphrasing beyond the text. The regulation is more indirect than that.

But sit in a notified body meeting for an AI/ML device today and explainability will come up — not as a philosophical question, but as a practical one. How does a radiologist know when to override your segmentation? How does a triage nurse know when the priority score is unreliable? What does the user see when the model is near a decision boundary? Auditors are asking these questions because they cannot confirm risk reduction and clinical benefit without answers.

This post frames explainability the way auditors actually frame it: as evidence that you have controlled the risks of using a model in clinical practice. That framing lets you respond honestly, build the right artefacts, and pass review without pretending MDR says something it does not.

What MDR actually says

The relevant MDR text, pulled together honestly:

Annex I §1 (general safety and performance requirement). Devices shall achieve the performance intended by the manufacturer and shall be designed and manufactured in such a way that, during normal conditions of use, they are suitable for their intended purpose. They shall be safe and effective and shall not compromise the clinical condition or safety of patients. Any risks shall be acceptable when weighed against the benefits and shall be compatible with a high level of protection of health and safety.

Annex I §8 — benefit-risk. All known and foreseeable risks and any undesirable side-effects shall be minimised and be acceptable when weighed against the evaluated benefits of the intended performance.

Annex I §17.1. Devices that incorporate electronic programmable systems or software that is a device in itself shall be designed to ensure repeatability, reliability and performance in line with their intended use. In the event of a single fault condition, appropriate means shall be adopted to eliminate or reduce as far as possible consequent risks.

Annex I §17.2. Software shall be developed and manufactured in accordance with the state of the art taking into account the principles of development life cycle, risk management, verification and validation. Here EN 62304:2006+A1:2015 provides the lifecycle reference.

Article 61 — clinical evaluation. The manufacturer shall specify and justify the level of clinical evidence necessary to demonstrate conformity with the relevant general safety and performance requirements. Clinical evaluation shall be based on a continuous update of the clinical evaluation throughout the lifecycle of the device.

EN ISO 14971:2019+A11:2021 — the risk management standard referenced under Annex I GSPR 1–9. This is where explainability arguments live in your file. Risk controls are evaluated on a hierarchy: inherently safe design, protective measures, information for safety. Explainability features typically sit in the protective measures and information-for-safety layers.

None of these says "explainable AI." What they say, taken together, is: you must demonstrate that a user can use this device safely and that risks are controlled. For an AI/ML device where a wrong output can lead to a wrong clinical decision, the auditor's question is: how does the user know? Everything we call "explainability" is an attempt to answer that question in evidence form.

A worked example

A startup builds an AI scoring tool that predicts deterioration in hospitalised patients from vital signs. Intended users are ward nurses and attending physicians. The model is a recurrent neural network the team considers non-interpretable in its weights.

The founder's first instinct is to write in the technical file: "The model is a black-box neural network; explainability is not feasible." We stopped that sentence before it reached the notified body.

Here is what we built instead, as risk-control evidence under EN ISO 14971:2019+A11:2021:

Hazard identified: user acts on a wrong high score (false positive) or ignores a wrong low score (false negative) leading to harm.

Risk controls implemented:

  1. Confidence / uncertainty indicator. The UI displays not a single number but a score plus a qualitative uncertainty band. Users are trained to treat low-confidence outputs differently.

  2. Input-driver display. The UI shows which recent vital sign changes drove the score most heavily in the last hour — a simple feature attribution, not a full interpretability method, but enough for the user to sanity-check the output against what they see at the bedside.

  3. Decision log. Every score the user acts on (or overrides) is logged with timestamp, user, score, uncertainty band, and action taken. This feeds post-market surveillance and supports trend analysis.

  4. Out-of-distribution detection. If incoming data is outside the training distribution, the score is suppressed and a warning is shown. This is an inherently-safe-design layer.

  5. Training and IFU content. Users are trained on conditions where the model underperforms — specific patient populations, specific data-quality situations — and these are documented in the IFU.

  6. Model card. An internal model card documents training data composition, validation performance including subgroups, known limitations, and intended operating envelope. This is an internal artefact used to support the risk management file and clinical evaluation, not a public disclosure.

The file did not argue the model was interpretable. It argued the system was safe to use despite the model being non-interpretable, and it built a chain of evidence connecting each risk control to specific MDR GSPRs. The notified body accepted that framing. They would not have accepted "explainability is not feasible."

That is the core move. Explainability is not a feature of the model — it is a property of the system around the model.

The Subtract to Ship playbook

1. Read Annex I §1, §8, §17 and Article 61 once, together. That cluster is where auditor expectations originate. Do not look for a magic explainability clause — it is not there.

2. Map explainability artefacts to risk controls under EN ISO 14971:2019+A11:2021. Every artefact (model card, confidence indicator, decision log) must appear in the risk management file as a control mitigating a specific hazard. No orphaned artefacts.

3. Build a model card — internal, not public. Training data composition, validation methodology, subgroup performance, intended operating envelope, known failure modes. This document anchors your risk management and clinical evaluation. Update it with every significant model change.

4. Build a decision log into the product. Not for marketing. For post-market surveillance under MDR Articles 83–86. You need evidence of real-world behaviour. Logs are the foundation.

5. Design the UI for skeptical trust. Users should see enough to decide whether to trust a specific output. That may be an uncertainty band, a feature attribution, a flag for out-of-distribution input, or a recommendation to defer. The right form depends on the user and the decision. EN 62366-1:2015+A1:2020 (usability engineering) is the standard to cite.

6. Tie explainability to clinical evaluation. Under Article 61, you argue that the device is safe and effective for its intended purpose. Your clinical evaluation should describe how users integrate the output into decisions and what evidence you have that the integration is safe. Explainability features feature heavily here.

7. Treat the bar as rising. What passed audit in 2023 may not pass in 2026. Notified body expectations on AI/ML have tightened every year. Re-read your technical file against current expectations at each annual surveillance.

8. Do not overpromise. If you write "our AI is fully explainable" in the IFU, a curious auditor will test the claim. Claim what you can defend: "the system provides feature-level indicators of which input changes drove the score" is defensible. "Explainable AI" is not.

Reality Check

  1. Can you name the specific hazards in your risk management file that explainability features control?
  2. Does your file argue system-level safety or model-level interpretability? (System-level is the defensible path.)
  3. Do you have an internal model card that is maintained with every significant change?
  4. Does your product generate decision logs that feed post-market surveillance?
  5. Can a user tell, from the UI, when an output should be treated with additional caution?
  6. Does your clinical evaluation under Article 61 describe how users integrate outputs into decisions?
  7. Have you reviewed notified body feedback on AI/ML from the last twelve months and updated your file accordingly?
  8. If your notified body asked "how do you know users can recognise a wrong output?", do you have an evidence-backed answer?

Frequently Asked Questions

Is there a specific MDR article requiring explainability? No. Expectations derive from Annex I §1, §8, §17.1, Article 61 and the risk management obligations under EN ISO 14971:2019+A11:2021. Explainability is an emergent practice, not a specific clause.

Can I use a black-box model? Yes. But the burden of demonstrating that users can use it safely — through system-level controls, training, UI design, and PMS — is higher. Non-interpretability of the model does not exempt you from Annex I §1 and §8.

Do I need to publish a model card? Not externally, for MDR purposes. An internal model card is good practice and supports your risk management and clinical evaluation. The IFU is the external disclosure vehicle under Annex I §23.

What does the EU AI Act add? The AI Act adds specific transparency and human oversight obligations for high-risk AI systems, including many medical AI devices. Its Article 13 transparency requirements and Article 14 human oversight requirements reinforce the same system-level controls notified bodies already expect under MDR.

How do I document explainability for a Class III device? Same framing as lower classes — as risk control evidence under EN ISO 14971:2019+A11:2021, tied to Annex I GSPRs and Article 61 clinical evaluation — plus public disclosure of relevant elements through the Article 32 SSCP.

Will an NB reject us because our model is not inherently interpretable? In our experience, no. They will ask how the system controls the risks of using a non-interpretable model. A strong answer passes review. A weak answer triggers findings.

Are there harmonised standards on AI explainability yet? Not as of early 2026 in a form that provides presumption of conformity for explainability specifically. Developments continue under CEN-CENELEC and ISO. Use EN ISO 14971 and EN 62304 as the anchoring harmonised standards, and track state-of-the-art literature.

Sources

  1. Regulation (EU) 2017/745 on medical devices, consolidated text. Annex I §1, §8, §17.1, §17.2, Article 61.
  2. EN ISO 14971:2019+A11:2021 — Medical devices — Application of risk management to medical devices.
  3. EN 62304:2006+A1:2015 — Medical device software — Software life cycle processes.
  4. EN 62366-1:2015+A1:2020 — Medical devices — Application of usability engineering to medical devices.
  5. MDCG 2019-11 Rev.1 — Guidance on Qualification and Classification of Software (October 2019, Rev.1 June 2025).