The need for action by evaluators and decision makers in Europe to ensure safe use of medical software
Mattias Kyhlstedt- Synergus RWE, Åkersberga, Sweden
Digital Health Solutions (DHS) approved under the Medical Device Directive (MDD) in the European Union may be used until May 27, 2025. The regulation provides appropriate requirements for the products but lack the evaluation by an external independent organization. For many DHS, the company can make a self-certification that the requirements have been fulfilled. As demonstrated in the evaluation of smartphone-based apps for skin cancer risk assessment such products may expose the public to undue risks. The new Medical Device Regulation provides adequate control of DHS through evaluation of independent organization prior to allowing the product on the market. HTA-evaluators and those who make decisions regarding the use of DHS need to understand the associated risks with the use of products approved according to the MDD and ensure appropriate risk mitigations to ensure that the public is not exposed to undue risk. This perspective aims to inform decisionmakers about the risks associated with the delayed requirement to transition to the new MDR regulation. There is a gap in the current guidance regarding the evolving use of machine learning and artificial intelligence. With the evolving use of DHS, it is important that industry, regulators and HTA evaluators work jointly to establish the safe and effective use of DHS.
Introduction
The proliferation of digital health solutions (DHS) entering the market has led to increased concern about both quality control and user safety. A recent systematic review of diagnostic accuracy studies of smartphone applications for skin cancer risk assessment by Freeman et al. concluded: “The current regulatory process for awarding the CE marking for algorithm based apps does not provide adequate protection to the public.” (1). The limitations of the MDD for class I devices in ensuring the safety of the public were thus made clear.
Originally released in 1993, limitations in the MDD have previously been pointed out, including outdated regulations, lack of consistency, focus on approval and not post-market performance, insufficient scrutiny of notified bodies (2), and the lack of a Software as Medical Device (SaMD) (3) classification, a class of products arguably not foreseen when the regulation was developed. These criticisms led to the establishment of a European Commission medical device working group in 2002 that included several stakeholders (4). The group's work resulted in the revised Medical Device Regulation (MDR), which was enacted in 2017. The MDR addressed several important aspects, in particular the classification of SaMD and the improved selection and oversight of the notified bodies. Whilst the new MDR legislation was initially set to replace MDD on May 26, 2020, a subsequent corrigendum (5) now permits products to remain on the market in their existing form until May 27, 2025. Analysis of the impact of this delayed implementation on how SaMD may operate in the meantime is required, not least from a safety perspective.
The transition from MDD to MDR is of particular importance for software-only products, as there are significant changes in MDR relating to the risk classification. In a study by the National Institute for Public Health and the Environment in the Netherlands (RIVM) a clear majority of products in class I were upgraded to a higher risk class. Indeed, many DHS may currently lead to detrimental health impacts (e.g., by not correctly identifying a skin cancer) through inadequate or incomplete control of the product. As such, any decision to approve use of such technology should include a careful evaluation of the quality, reliability, and validation of health risk assessment products, which are increasingly smartphone-based and self-administered.
Objective
The objective of this study is to assess if the risks identified in the review of algorithm-driven smartphone-based applications (1) would obtain adequate regulatory control under the new MDR regulation. This study furthermore provides a summary of the main differences between the MDD and MDR and suggests ways to mitigate potential safety issues until the MDR is in full force.
Methodology
A comparative review of the MDD and MDR legislative frameworks, associated guidance, and standards for the SaMD approval process in the EU was conducted. The review evaluates the difference in regulatory classification and how this impacts the measures to ensure a safe placement of the product on the market.
Eventual limitations in the legislative requirements or the external evaluator's role and/or function were also assessed.
Results
MDD vs. MDR regulation
Understanding the legislative framework
The MDD requires that DHS products satisfy its Essential Requirements (ER); the corresponding stipulation in the MDR is that they satisfy its General Safety and Performance Requirements (GSPR). The ER include a high-level requirement of all aspects required to ensure a safe medical device or SaMD, including usability software development, across all risk classes of medical devices or SaMD.
For devices including software, the sole requirement stated in the MDD is (6):
12.1 For devices which incorporate software or which are medical software in themselves, the software must be validated according to the state of the art taking into account the principles of development lifecycle, risk management, validation and verification.
This requirement arguably needs to be put in context to be clarified. The MDD is based on the availability of harmonized standards that define how the ER best are fulfilled (6):
Member States shall presume compliance with the essential requirements referred to in Article 3 in respect of devices which are in conformity with the relevant national standards adopted pursuant to the harmonized standards the references of which have been publishes in the Official Journal of the European Communities; Member States shall publish the references of such national standards.
In the case of software development, the IEC-62304 harmonized standard is applied, providing a well-designed methodological framework for how software should be developed and validated to ensure a safe product. The international standards are developed with methodological experts from industry and legislators in a comprehensive manner. The MDD does not require that this standard be used, but it requires that state of the art be used. If a company can justify another method that represents the state of the art, then they may use that approach instead; it is difficult to pursue such attempt in practice, however. As outlined in Table 2, the documentation for the software development will not be reviewed by anyone if the product belongs to class I in the MDD, whereas both the process and the result will be reviewed for products in class IIa, IIb and III MDR.
Risk classification of products leads to better safety control measures
The class of the SaMD determines the measures applied by Notified Bodies to ensure the safety of the products being placed on the market. Some low-risk-class SaMD will remain class I in the MDR as well, such as prevention-based applications e.g., cardio training apps offering workout recommendations (7). How class I SaMD under the MDD might be re-classified as IIa, IIb or III under the MDR requires a more complex analysis of the application of measures, although Table 2 provides a simplified explanation.
Comparing the risk classification for DHS
The classification of medical devices, including SaMD, is based on the potential risk associated with the use of the device in relation to the vulnerability of the human body (8). Based on the present MDD (and corresponding guidance) for classification of DHS (9) SaMD according to the MDD is classified as class I in most scenarios, whereas according to the MDR and corresponding guidance most current devices will be classified as class IIa, IIb or even III (10). The 2018 RIVM report (11) compared the classification between the MDD and MDR across 56 categories of SaMD, and found that 73%(24 of 33) of MDD class I devices will be re-classified as Class IIa or higher in the MDR; an additionally, 12% (2 of 16) of MDD Class IIa SaMD devices would be re-classified upwards in the MDR. These re-classifications will lead to different conformity assessment routes, representing different level of safety assurance controls.
The role of the notified body
The MDD and MDR rely on commercial notified bodies designated by the competent authorities. This system is not without its flaws however. The BMJ and other publications (12) (13–18) used covert methods involving a fake hip implant to illustrate how the intended control process of the Notified Bodies could be circumvented to get a sub-standard product approved. The UK health secretary, Jeremy Hunt, subsequently pledged to stop this “worrying and completely unacceptable weakness in the regulatory system” (19). This promised scrutiny led to a reduction in MDD-related Notified Bodies from 75 to 58 over several years.
Recognizing the limitations in designating Notified Bodies under the MDD, the accreditation process has been reworked in the MDR. As of February 2020, there were only 11 notified bodies who had been authorized according to the new regulation, the first of which was accredited in January 2019 (BSI). Even though it is not possible to fully understand the effect of the revision, the requirements have increased significantly.
Clinical data and literature reviews in the approval process of SaMD
The MDD and MDR require companies to make systematic evaluation of the evidence supporting the utility, performance, and safety of their product, both during initial release of the product and as part of their continued surveillance of the safe use of their product.
Systematic reviews of categories of products, outcomes of relevance and outcomes in standard of care are essential to both developers and evaluators in identifying benefits and risks, and if these are appropriate in relation to the current standard of care. The review of smartphone apps for skin cancer (1) provides an excellent example of how such research can constitute a baseline for future evaluation of technologies in a given product category.
In addition to the review of clinical evidence, §23 in the MDR also outlines the opportunity to develop common standards and specifications for specific product categories (20).
Absence of harmonized standards for SaMD in the MDR
There are currently no standards harmonized with the MDR, which creates a high degree of uncertainty for SaMD developers about how they fulfil the requirements in the GSPR, which are equally non-specific as in the MDD. The standards that are harmonized under the MDD are the same as those planned for harmonization with the MDR, although according to current plans this will not be completed until May 2024.
The Medical Device Coordination Group (MDCG) issued guidance for cybersecurity December 2019 (21) that on how these requirements should be addressed. It also included a list of 16 standards for various software development aspects including risk management, usability, network safety, and software life cycle development, although they pointed out that these were solely for informative purposes as they were not harmonized. It is therefore unclear what role these may have in ensuring fulfilment of the GSPR.
Guidance and standards regarding artificial intelligence (AI) and machine learning (ML)
There are currently no standards or guidance regarding the design and use of AI/ML. The MDCG has not published any current plans to do so on their publicly available activities list. There is, however, significant pressure from many stakeholders to establish such standards and guidance to ensure the safe use of AI/ML.
A regulatory perspective
The major quality and performance issues identified in Freeman et al. were cross-tabulated with the relevant sections of the current MDD regulations in Table 1, including an assessment of the adequacy of the MDD for the specific issue (1). The analysis suggest that the quality and performance issues were not likely to be related to the applicable requirements for such products in the regulation but rather to methods applied to ensure compliance under the MDD regulation. With the new MDR regulation, with a revised classification and increased control measures of notified bodies and more stringent requirements for clinical evidence it is unlikely that flaws such as those reported in Freeman et al. would be placed on the market (1).
Table 1. Comparison of DHS quality issues identified in the Freeman et al. (1) systematic review vs. regulatory requirements in the MDD.
When DHS products are denoted as class I in the MDD, responsibility for the interpretation of the legislation and harmonized standards lies entirely with the developing company without any involvement of external governance. In high-risk classes, the company and the product will undergo review by a Notified Body. The lack of adequate risk classification for SaMD in MDD and consequent lack of external evaluation is the probable cause for the identified issues in reference (1).
Table 2. Comparing measures to ensure safe use of the products on the market.
Discussion
With the current limitations of how SaMD are regulated under the MDD, it is imperative that stakeholders understand the risks associated with acquiring and using such products. Companies producing class I SaMD products under the MDD can at their discretion claim that both they and their product fulfil the ER, without the need for any external evaluation. As illustrated in the example with of apps for skin cancer (1), such products aim to directly or indirectly influence the diagnosis of cancer, and are currently being placed on the market with inferior performance to clinical standards. This is a potentially major risk public health.
Suggested actions for developers of SaMD
Developers should ensure they have a good understanding of the regulatory requirements that relate to all aspects of the MDD and MDR frameworks to ensure the ability to provide safe products to the patients. In the absence of harmonized standards under the MDR, the advice is to start implementing the standards that currently are harmonized under the MDD until more clarity has been established for the former.
Suggested actions for HTA evaluators and decision makers in the use of SaMD
Considering the limitations of the MDD for SaMD and the ability to place such products on the market until May 2025, it is essential that HTA evaluators and decision makers carefully evaluate the associated risks with using such devices.
For Class I SaMD under the MDD, it should be recognized that there is no independent evaluation of the development of the product nor evaluation of the technical documentation and/or clinical data.
This could be mitigated by only approving the use of SaMD that can demonstrate fulfilment of the MDR requirements, or by requiring competent review that the requirements of the MDD have in fact been fulfilled. Companies would argue that this is prohibitive due to the limitations of Notified Bodies in qualifying them according to the MDR, but from a public health perspective is it reasonable to accept the use of SaMD without such assurances?
Strengths and limitations
This work builds upon the strengths of Freeman et al. (1) in illustrating the consequences of the challenges related to current regulatory situation for SaMD, and adding knowledge of regulatory processes from industry and agencies.
The stakeholders addressed in this study – namely Notified Bodies and regulatory agencies – have not been consulted regarding their view on the conclusions of this article, which may be considered a limitation.
Conclusion
Active engagement with decision makers and evaluators to create an understanding of the risk at hand of using devices approved under the MDD class I is essential. With the evolving use of DHS in healthcare it is important for industry, regulators, and HTA-evaluators to jointly work together for the safe and effective use. This is in particular important in the growing field of applications using machine learning or artificial intelligence.
Data availability statement
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author/s.
Ethics statement
Ethical review and approval was not required for this study in accordance with the local legislation and institutional requirements.
Author contributions
MK is the first and only author and approved the submitted version.
Acknowledgments
The author would like to acknowledge Matt X. Richardson for support with text edition.
Conflict of interest
MK is employed by Synergus RWE.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1. Freeman K, Dinnes J, Chuchu N, Takwoingi Y, Bayliss SE, Matin RN, et al. Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies. Br Med J. (2020) 368:m127. doi: 10.1136/bmj.m127
2. The European MDR Impetus Impacts And Current Status. Available at: https://www.meddeviceonline.com/doc/the-european-mdr-impetus-impacts-and-current-status-0001 (Accessed March 5, 2020).
3. “Software as a Medical Device”: Possible Framework for Risk Categorization and Corresponding Considerations. (2014). p. 30. Available at: https://www.imdrf.org/sites/default/files/docs/imdrf/final/technical/imdrf-tech-140918-samd-framew ork-risk-categorization-141013.pdf (Accessed April 20, 2021).
4. Altenstetter C. Medical device regulation and nanotechnologies: determining the role of patient safety concerns in policymaking: medical device regulation. Law Policy. (2011) 33(2):227–55. doi: 10.1111/j.1467-9930.2010.00337.x
5. CORRIGENDUM to Regulation (EU) 2017/745. (2019). Available at: https://data.consilium.europa.eu/doc/document/ST-13081-2019-INIT/en/pdf
6. COUNCIL DIRECTIVE 93/42/EEC. Available at: https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CONSLEG:1993L0042:20071011:EN:PDF (Accessed February 22, 2020).
7. Institut J. MDR Rule 11: The Classification Nightmare. Available at: https://www.johner-institute.com/articles/regulatory-affairs/and-more/mdr-rule-11-software/, https://www.johner-institute.com/articles/regulatory-affairs/and-more/mdr-rule-11-software/ (Accessed March 5, 2020).
8. MEDDEV 2. 4/1. (2010). Available at: https://ec.europa.eu/docsroom/documents/10337/attachments/1/translations (Accessed April 20, 2021).
9. MEDDEV 2.1/6. (2016);29. Available at: https://ec.europa.eu/docsroom/documents/17921/attachments/1/translations/en/renditions/native (Accessed April 20, 2021).
10. MDCG. MDCG 2019-11 Guidance on qualification and classification of software in regulation (EU) 2017/745 – MDR and regulation (EU) 2017/746 – IVDR. (2019). Available at: https://health.ec.europa.eu/system/files/2020-09/md_mdcg_2019_11_guidance_qualification_classification_software_en_0.pdf (Acc essed April 20, 2021).
11. Apps under the medical devices. (2018):40. Available at: https://www.rivm.nl/bibliotheek/rapporten/2018-0083.pdf (Accessed February 25, 2020).
12. Cohen D. EU Approval system leaves door open for dangerous devices. Br Med J. (2012) 345:e7173. doi: 10.1136/bmj.e7173
13. Cohen D. How a fake hip showed up failings in European device regulation. Br Med J. (2012) 345:e7090. doi: 10.1136/bmj.e7090
14. Cohen D. Faulty hip implant shows up failings of EU regulation. Br Med J. (2012) 345:e7163. doi: 10.1136/bmj.e7163
15. Cohen D. Notified bodies: are they fit for purpose? Br Med J. (2012) 345:e7177. doi: 10.1136/bmj.e7177
16. Cohen D. Manufacturers offered helping hand with EU approval. Br Med J. (2012) 345:e7225. doi: 10.1136/bmj.e7225
17. Godlee F. The scandal of medical device regulation. Br Med J. (2012) 345:e7180. doi: 10.1136/bmj.e7180
18. McCulloch P. The EU's System for regulating medical devices. Br Med J. (2012) 345:e7126. doi: 10.1136/bmj.e7126
19. Cohen D. UK Health secretary promises to tighten regulatory procedures for medical devices. Br Med J. (2012) 345:e7192. doi: 10.1136/bmj.e7192
20. REGULATION (EU) 2017/745 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL. (2017):175. Available at: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32017R0745&from=EN (Accessed February 22, 2020).
21. MDCG 2019-16 Guidance on Cybersecurity for medical devices. (2019). Available at: https://health.ec.europa.eu/system/files/2022-01/md_cybersecurity_en.pdf (Accessed March 3, 2020).
Keywords: regulation, digital health, health technology assessment, medical software, MDR, MDD
Citation: Kyhlstedt M (2022) The need for action by evaluators and decision makers in Europe to ensure safe use of medical software. Front. Med. Technol. 4:1063622. doi: 10.3389/fmedt.2022.1063622
Received: 7 October 2022; Accepted: 3 November 2022;
Published: 29 November 2022.
Edited by:
Rossella Di Bidino, Agostino Gemelli University Polyclinic (IRCCS), ItalyReviewed by:
Arso M. Vukicevic, University of Kragujevac, Serbia© 2022 Kyhlstedt. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Mattias Kyhlstedt mattias.kyhlstedt@synergusrwe.com
Specialty Section: This article was submitted to Regulatory Affairs, a section of the journal Frontiers in Medical Technology