Benefit Assessment and Reimbursement of Digital Health Applications: Concepts for Setting Up a New System for Public Coverage

In Germany, some digital health applications (DiHA) became reimbursable through the statutory health insurance system with the adoption of the Digital Healthcare Act in 2019. Approaches and concepts for the German care context were developed in an iterative process, based on existing concepts from international experience. A DiHA categorization was developed that could be used as a basis to enable the creation of a reimbursed DiHA repository, and to derive evidence requirements for coverage and reimbursement for each DiHA. The results provide an overview of a possible classification of DiHA as well as approaches to assessment and evaluation. The structure of remuneration and pricing in connection with the formation of groups is demonstrated.

Introduction

Digital health applications (DiHA), often also called mHealth applications, are cooperative and/or interactive applications of modern information and communication technologies, aiming to improve care and population health. They have rapidly proliferated in recent years, leading to considerable transformation in the delivery of health services (1, 2). DiHA have a wide range of different functions and are used in various areas (3). The heterogeneous structures of health systems and variability of regulations between countries mean that DiHA face different evaluation processes before inclusion into the benefits basket.

The European Commission Expert Panel on effective ways of investing in health (4) highlighted that the implementation, use, or reimbursement of DiHA should ideally be based on the evidence about their performance: DiHA should be assessed against health system objectives such as access, quality, and contribution to population health, efficiency, and equity. The proof of patient benefit is also a central element in the DiHA evaluation framework of the World Health Organization to conducting research and assessment (5). The benefit for the end user should be described and it should be demonstrated why the application is innovative, and to what extent, it is superior to the standard of care. At the same time, approaches for demonstrating the benefit of DiHA are increasingly emerging, aiming to both do justice to the particular characteristics of these health technologies and guarantee sufficient evidence to satisfy the evidence standards described above (6).

European countries apply different regulations for the payment of DiHA; they can partly be paid voluntarily by patients and individual health insurees, or individual DiHA with certain proof of patient benefit are paid obligatorily by all health insurers. Examples of countries moving in the latter direction with interesting developments in the categorization of DiHA include Belgium (7), England (8, 9), and France (10, 11). This is linked with an increased interest in reimbursement options (12–14) or assessment frameworks of DiHA (15, 16). Germany however is considered a pioneer in the European context, as a statutory reimbursement obligation has already been introduced (17). To bring DiHA into the statutory health insurance (SHI) system in Germany, the Digital Health Care Act (Digitale-Versorgung-Gesetz, DVG) was passed in November 2019 (18). The law created, among other things, the entitlement of insured persons to certain types of DiHA and broadly regulates the reimbursement claims of DiHA, and the requirements that manufacturers must fulfill to be eligible for funding. The subsequent Digital Health Applications Ordinance (Digitale Gesundheitsanwendungen-Verordnung, DiGAV) describes the procedure and specific requirements for manufacturers in detail (19).

Briefly put, to be reimbursed by SHI in Germany, a DiHA must be classified as a class I or IIa medical device according to Regulation (EU) 2017/745 on medical devices or according to Directive 93/42/EEC, serve a specific purpose, and not be used exclusively by healthcare providers/health professionals. DiHA developers must submit an application to the German Institute for Medicines and Medical Devices (Bundesinstitut für Arzneimittel und Medizinprodukte, BfArM), demonstrating that their application meets requirements for safety, quality, functionality, privacy, and data security. They must also demonstrate that the application has positive care effects.

It has been argued that DiHA differ substantially from both medicines and medical devices in their characteristics, and this should be considered in the design of evidence-based pathways toward reimbursement and pricing (20). Perhaps most importantly, DiHA can reflect a care process rather than a single product. At the same time, DiHA compete for the same pot of funds as other health interventions, and the principles underpinning decisions on their reimbursement should be comparable. This article addresses the question of how these supposedly contradictory statements can be united. In the following, approaches developed in the “I.DiGA project” (see Methods Section, next) for a benefit assessment and pricing mechanism for DiHA will be presented. While the project focused on the German context, the approaches described in the following may be useful for other countries considering setting up similar systems.

Methods

Parallel to the German “Digital Health Care” (DVG) legislative process, the “I.DiGA project,” funded by the Federal Ministry of Health, developed approaches for the categorization, evaluation, and pricing/reimbursement of DiHA. The project took place in parallel with the preparation, discussion, and adoption of the DVG and related regulatory documents, and its results served as one of the bases that informed this process. The project had an agile design: contents were developed drawing on existing international frameworks and initiatives, literature research, and expert interviews. These contents were then discussed in seven expert workshops involving a broad range of stakeholders to identify areas in need of refinement and additional analysis, and possible solutions were developed. There were three main areas of investigation: (a) the development of a conceptual model for a DiHA directory that could serve reimbursement purposes; (b) the development of a framework for the evidence-based evaluation of DiHA, encompassing process and methodological requirements; and (c) the development of a remuneration system for DiHA, including a pricing approach.

Results

Developing a “Positive List” for DiHA: Proposal for a DiHA Directory

Given the heterogeneity of DiHA, as well as the availability of several DiHA serving the same purpose, it was necessary to create a taxonomy that would facilitate reimbursement, pricing, and prescribing decisions. We developed a categorization that could serve as the basis for a directory with a triple functionality: grouping DiHA in a meaningful way to enable the determination of the type of evidence required for reimbursability; to identify comparators, against which additional benefits can be demonstrated; and to enable DiHA comparisons for health professionals looking to choose which alternative to prescribe. We proposed a “directory for digital health applications” that groups DiHA according to the following attributes: (i) application area, (ii) target group, (iii) function, and (iv) user. It is listed in Table 1.

TABLE 1

Table 1. Proposal for DiHA directory with criterion values and correspondence to evidence requirement levels (blue shading).

The application area (i) is derived from the 3-digit ICD 10 code of the disease for which the DiHA is to be used according to its certification and this, together with the function, reflects the intended purpose of the DiHA as a medical device. The target group (ii) complements the scope of application and reflects the disease severity or vulnerability of the patients who are to receive the DiHA. The classification varies because the evidence requirements for healthy people may be lower than for highly vulnerable people. The function (iii) is divided into detection, monitoring, and treatment (including alleviation), which are further differentiated according to their exact functionality into (4) Diagnosis, (5) Simple Monitoring, (6) Complex Monitoring, (7) Indirect Intervention, and (8) Direct Intervention. Under “function,” Categories 0–3 represent possible functionalities of DiHA as medical devices according to Regulation (EU) 2017/745, which are, however, not among those meeting the requirements of the 2020 iteration of the DVG for prescribability (these were included in the model to ensure its sustained usability should the scope of the legal framework be expanded in future). The user (iv) of a DiHA may not only determine the risk associated with its use, but also influence the evaluation design and the potential access and remuneration pathways. For the user, a distinction is made as to whether the DiHA is used (1) by the patient or a relative only, or (2) by the patient and healthcare provider. Category (3) is reserved for DiHA that are used exclusively by health professionals and are therefore not covered by the DVG (illustrated here for the same reason as Categories 0–3 under “function”).

For each criterion, a DiHA included in the SHI reimbursement list would be assigned the matching digits depending on taxonomic position, and those would be sequentially merged into a 6-digit number. Additional digits are added at the end to provide a serial number for each DiHA (product) within a single taxonomic position (Table 1).

Evidence Requirements for the Reimburseability of DiHA

The DiHA classification system described above can serve inter alia to link the identified DiHA groups to the level of evidence required for the DiHA to demonstrate positive care effects and qualify for reimbursement. For reasons of usability, the I.DiGA project proposed a simplified distinction between three evidence requirement levels: low, medium, or high. The requirement level for each DiHA group in the matrix is determined by the two attributes “target group” and “function.” The darker the shade of the cell in Table 1, the more rigorous the study design of the evidence required for reimbursability. The final level of evidentiary requirements for each group of DiHA is determined by the highest level assigned to one of the two categories (e.g., a direct intervention for stable chronic patients would fall into the “high” evidence category, determined by the high assignment for DiHA with this function).

To further specify the acceptable evidence per requirement level, considerations from existing frameworks were combined (5, 8, 10). Three fundamental elements of study design were chosen to provide the necessary foundation and retain simplicity. Expectations on study design increase with the level but the comparison against a control group was considered essential for all levels. Only DiHA in the low requirement category can potentially rely on study designs wherein participants form their own comparison group (before/after designs) to qualify for reimbursability. A medium requirement level means it is sufficient to present results from (quasi-)experimental studies with a control group. While the randomized allocation to the intervention and control group is not a prerequisite, all relevant confounding variables should be identified and considered to the maximum possible extent. The analysis must follow the “Intention to treat” (ITT) principle. When DiHA fall into a high requirement category, a randomized controlled study with an analysis following the ITT principle is required (Table 2).

TABLE 2

Table 2. Study design characteristics per requirement level.

Acceptable Outcomes for Demonstrating the Effects of DiHA

Due to their nature, DiHA can have health effects and other effects. Taking the overall aim of improving patient-relevant outcomes, the project relied on the assumption that for the overall effect of DiHA to be positive, either the health effects must be better with the DiHA than without the DiHA, or the other effects must be better if the health effects are the same (i.e., health effects should not be jeopardized even if other effects are better; the general interpretation of the DVG in Germany deviates from this position, see Section Discussion).

For demonstrating the health effects of DiHA, end points that reflect patient-relevant benefits in the form of an improvement in health status, shortening of disease duration, prolongation of life, reduction of side effects, or improvement in quality of life are ideal. On the other hand, typical surrogate parameters can also be acceptable under certain circumstances, as can other intermediate end points such as medication adherence; provided that their correlation with patient-relevant end points is established (i.e., they have been validated).

Similar to the logic for determining the evidence requirement levels described above, acceptable end points depend on the type of DiHA as determined by its function and target group. Table 3 brings together these two attributes with the corresponding level of required evidence and the respective acceptable end points for determining health effects. For DiHA that make a diagnosis, take over complex monitoring, or represent a direct intervention to change health status, relevant health-related end points should be investigated with robust study designs. Surrogate parameters can be used as end points for these DiHA only if non-vulnerable target groups are involved. For the DiHA that correspond to indirect interventions or simple monitoring, the spectrum of health-related end points can be expanded to include other intermediate outcomes.

TABLE 3

Table 3. Matrix for determining the requirement level and acceptable end points for health effects.

As DiHA often reorganize whole processes of care, non-health outcomes can be used to determine reimbursement, provided that the health effects are not compromised. The field of health technology assessment (HTA) has long considered the potential implications of the application of health technologies across a range of domains, and these can also be used to derive outcome categories here (21). Specifically, organizational, social/ethical, and economic aspects were considered particularly relevant for determining a DiHA's eligibility for reimbursement. Building on the EUnetHTA Core Model®, core aspects with associated contextual questions can help identify possible outcome areas and target variables (see Appendixes 1–5 in Supplementary Material). As with health effects, the evaluation of other care effects of DiHA should be based on a comparative analysis, if they are to improve patient outcomes. While established outcomes for the measurement of health and economic effects are available, this is largely not the case for organizational and social/ethical effects. How these can be measured depends on function and target group and cannot be determined uniformly.

Relationship Between Health Effects and Other Care Effects

As described earlier, the I.DiGA project worked on the assumption that other care effects should be used to determine reimbursability only if sufficient evidence has shown that health effects are superior, or at least not inferior to those of the appropriate comparator. Drawing on similar considerations in France (10), the matrix in Table 4 was designed to visualize the interplay between health effects, other care effects, and overall reimbursability. If neither the health effects nor the other care effects are superior to a corresponding comparison, eligibility for reimbursement should not be considered given.

TABLE 4

Table 4. Relationship between health effects and other care effects.

Pricing and Remuneration Systems of DiHA

Principle of Price Determination for DiHA

Based on the usual pricing methods applied or discussed for health technologies, the amount public payers are to spend on DiHA qualifying for reimbursement can be based on (a) the extent of the positive effect proved in studies and/or (b) development and production costs and/or (c) a price comparison (national/European consumer prices or European reimbursement prices). Depending on health system goals and payer criteria, price negotiations can consider only positive health effects, or also other care effects such as increased efficiency. Based on any one or a combination of these three components, an initial base amount to be paid for a DiHA can be calculated, potentially complemented by performance-based remuneration components for actual positive effects achieved by patients using the DiHA. DiHA utilization data or patient-reported outcome and/or experience measures can be used to determine such components.

Considerations on a Remuneration Model

The proposed logic for a remuneration and pricing model builds on the previous ideas of clustering DiHA based on indication, function, and target group and requiring comparative evidence before they can qualify for reimbursement. For DiHA with comparable measured effects and underlying study quality, prices should also be comparable. A new DiHA would only be eligible for a higher price if it were superior in term of positive effects and/or was supported by methodologically better studies than already reimbursed DiHA in the same group. An additional dimension that may contribute to a higher price within a DiHA group concerns the comparator in the submitted studies: a new DiHA that can show benefit compared to existing DiHA would receive a higher price than DiHA that have shown benefit compared only with no DiHA use. If the three criteria positive effects (by type and extent), quality of evidence (including overall study design and quality of individual studies), and comparator are combined, a classification leading to different price tiers can be created (Table 5).

TABLE 5

Table 5. Possible price tiers based on the criteria of study results, quality of evidence, and comparator.

Discussion and Policy Implications

The aim of the I.DiGA project was to provide a basis for a workable system of implementing DiHA in the public reimbursement system, particularly for the German context. With the DVG, Germany is considered a pioneer, where patient-facing DiHA are now reimbursable by the SHI system. However, the project aimed at providing conceptual guidance usable beyond the context of the DVG as passed in 2020, regarding, for instance, an expansion in the scope of regulated DiHA. It should be helpful for relevant stakeholders thinking about how to set up a public system for DiHA access in any country.

Several countries have developed or are developing approaches to better understand and evaluate DiHA for health system use; however, these are at different stages of development and clearly dependent on the country's health system set-up and coverage decision-making processes so far (4, 22, 23). Concerns regarding privacy, efficacy, usability, and implementation of DiHA are common characteristics of all these initiatives. These could potentially be alleviated by clear evaluation guidelines from the relevant decision-makers, as they have now been developed in Germany (24).

The system to determine the reimbursability of DiHA presented in this paper has similarities with other approaches, such as the Belgian validation pyramid (7), but also important differences, such as the categorization based on target group and function. The categorization matrix is more granular than existing alternatives for a formal app repository, such as the National Health Service (NHS) App (9). Regarding evidentiary requirements and related pricing options, our model draws on the NICE framework from England (8), and the Medical Device and Health Technology Evaluation Committee (CNEDiMTS) framework (10), where, however, the number of evidence levels and the level of detail of reimbursement situations differ.

A systematic review of evaluation frameworks for mobile health applications found that the majority of the identified frameworks that examined safety—only a few assessed potential harm and only one explicitly considered a comparison group (25). However, it has been recognized that health systems must provide adequate approaches for DiHA evaluation and reimbursement (5, 26), and HTA can provide a methodological basis (27). To make informed decisions about the reimbursement, prescription and use of DiHA, stakeholders, clinicians, and users need guidance that adapts to diverse local requirements, standards, and current content (28). To create a standard set of requirements, state regulatory agencies should collaborate with clinical stakeholders (22). Assessment tools for DiHA that aim to measure risks and benefits are proliferating (29). Methodological considerations for evaluating DiHA are increasingly developed, for instance, primarily for specific DiHA groups [e.g., for mental health, see (30), or with a more comprehensive scope (31)].

The work in the I.DiGA project confirmed the national and international relevance of the topic. Even though there are more and more approaches toward DiHA evaluation and reimbursement decisions, none have really been tested yet in the real world. Consequently, the implementation of DiHA reimbursement processes should be closely monitored and evaluated, and shared in international dialogue. The concepts developed in the I.DiGA project can provide an initial basis but will likely need to be adapted once the implementation of current provisions in Germany and other countries has been evaluated. The exchange of knowledge and experience at European level could contribute to the improvement and expansion of relevant assessment frameworks; existing initiatives should be supported, and funding should be made available for implementation research.

The study was developed based on the regulations and particularities of the German health system; this may be considered a limitation, or at least a challenge for transferability. However, even if other countries have different starting points, our considerations can be adopted to some extent. Another limitation of our work pertains to the selective consideration of existing literature and country examples for the evaluation of DiHA, as no systematic search was carried out. A number of new publications may have become available since this work was concluded. The insights in this paper are mainly conceptual in nature and have not been validated yet; however, they partially draw on existing and tested regulatory levers for other health technologies in the German health system.

Conclusion

The I.DiGA project developed concepts to categorize, evaluate, and price DiHA for reimbursement. These can be used individually or in combination to inform the formalization of any DiHA reimbursement system. Fundamentally, DiHA that want to be covered by statutory health systems need to provide robust evidence of patient benefit, which can be used to determine reimbursement eligibility and price. As other European countries are also increasingly developing approaches for categorizing or reimbursing DiHA, cross-border cooperation for the development of efficient evaluation methods which consider the particularities of DiHA is indicated, and has the potential of expanding reimbursement opportunities for developers.

Author Contributions

HL, DP, and RB wrote the manuscript. All authors conceived the contents of the project and revised the manuscript. All authors contributed to the article and approved the submitted version.

Funding

The I.DiGA project was funded by the German Federal Ministry of Health (Grant Reference: ZMVI1-2519FSB800).

Conflict of Interest

VS was employed by HelloBetter—GET.ON Institut für Online Gesundheitstrainings GmbH. DS and KK were employed by fbeta GmbH. CP was employed by D+B Rechtsanwälte.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We would like to thank Sabine Fuchs for her input, and all participants who attended the expert workshops and gave content-related impulses.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2022.832870/full#supplementary-material

References

1. Thiel R, Deimel L, Schmidtmann D, Piesche K, Hüsing T, Rennoch J, et al. Health System Comparison Focus Digitalization. #SmartHealthSystems. International comparison of digital strategies. Bertelsmann Foundation 2018 (2019).

Google Scholar

2. Williams GA, Fahy N. Building and maintaining public trust to support the secondary use of personal health data. Eur Observ Health Syst Policies. (2019) 25:7–10.

Google Scholar

3. Fahy N, Williams GA. Use of digital health tools in Europe: before, during and after COVID-19. Policy Brief . (2021) 42:34. Available online at: https://eurohealthobservatory.who.int/publications/i/use-of-digital-health-tools-in-europe-before-during-and-after-covid-19 (accessed December 9, 2021).

PubMed Abstract | Google Scholar

4. European Commission. Assessing the Impact of Digital Transformation of Health Services. Report of the Expert Panel on effective ways of investing in Health (EXPH). (2019). Available online at: https://ec.europa.eu/health/expert_panel/sites/expertpanel/files/docsdir/022_digitaltransformation_en.pdf (accessed December 9, 2021).

Google Scholar

5. World Health Organization. Monitoring and Evaluating Digital Health Interventions: A Practical Guide to Conducting Research and Assessment (2016). Available online at: https://apps.who.int/iris/bitstream/handle/10665/252183/9789241511766-eng.pdf;jsessionid=BFA562487E3057944410F7A9A13219FD?sequence=1 (accessed December 9, 2021).

Google Scholar

6. Dick S, O'Connor Y, Heavin C. Approaches to mobile health evaluation: a comparative study. Inform Syst Manag. (2020) 37:75–92. doi: 10.1080/10580530.2020.1696550

CrossRef Full Text | Google Scholar

7. Mhealthbelgium. Validation Pyramid of mHealth, Belgium. (2022). Available online at: https://mhealthbelgium.be/validation-pyramid (accessed February 4, 2022).

Google Scholar

8. National Institute for Health Care Excellence (NICE). Evidence Standards Framework for Digital Health Technologies (2019). Available online at: https://www.nice.org.uk/Media/Default/About/what-we-do/our-programmes/evidence-standards-framework/digital-evidence-standards-framework.pdf (accessed December 9, 2021).

Google Scholar

9. NHS. NHS App. (2022). Available online at: https://www.nhs.uk/nhs-app/ (accessed February 4, 2022).

Google Scholar

10. French National Authority for Health (HAS). Évaluation des dispositifs médicauxpar la CNEDiMTSGuide sur les spécificités d'évaluation clinique d'un dispositif médical connecté (DMC) en vue de sonaccès au remboursement. (2019). Available online at: https://www.has-sante.fr/upload/docs/application/pdf/2019-02/guide_sur_les_specificites_devaluation_clinique_dun_dmc_en_vue_de_son_acces_au_remboursement.pdf (accessed December 9, 2021).

Google Scholar

11. L'Assurance Maladie. La liste des produits et prestations—LPP (2021). Available online at: https://www.ameli.fr/medecin/exercice-liberal/remuneration/nomenclatures-codage/liste-produits-prestations-lpp (accessed February 4, 2022).

Google Scholar

12. Jeindl R, Wild C. Framework zur Unterstützung von Refundierungsentscheidungen zu digitalen Gesundheitsanwendungen (mHealth) und dessen (retrospektive) Anwendung an ausgewählten Beispielen. AIHTA Projektbericht Nr.: 134; 2020. Wien: HTA Austria – Austrian Institute for Health Technology Assessment GmbH (2020). Available online at: https://eprints.aihta.at/1279/1/HTA-Projektbericht_Nr.134.pdf (accessed February 4, 2022).

Google Scholar

13. Degelsegger-Márquez A, Panteli D. Towards reimbursement of mobile digital health applications? An international comparison of practices. Eur J Public Health. (2020) 30(Supplement_5):ckaa165-1327. doi: 10.1093/eurpub/ckaa165.1327

CrossRef Full Text | Google Scholar

14. European Med Tech and IVD Reimbursement Consulting. Reimbursement Landscape for Health Apps in Europe. (2020). European Med Tech and IVD Reimbursement Consulting.

Google Scholar

15. Lagan S, Sandler L, Torous J. Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps. BMJ Open. (2021) 11:e047001. doi: 10.1136/bmjopen-2020-047001

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Hensher M, Cooper P, Dona SWA, Angeles MR, Nguyen D, Heynsbergh N, et al. Scoping review: Development and assessment of evaluation frameworks of mobile health apps for recommendations to consumers. J Am Med Inform Assoc. (2021) 28:1318–29. doi: 10.1093/jamia/ocab041

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Gerke S, Stern AD, Minssen T. Germany's digital health reforms in the COVID-19 era: lessons and opportunities for other countries. NPJ Digit Med. (2020) 3:94. doi: 10.1038/s41746-020-0306-7

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Gesetz für eine bessere Versorgung durch Digitalisierung und Innovation (Digitale-Versorgung-Gesetz; DVG) (2019). SGB V; Bundesgesetzblatt (2019), I, Nr. 49, S. 2562–2584 (2019). Available online at: https://www.bgbl.de/xaver/bgbl/start.xav?startbk=Bundesanzeiger_BGBl&start=%2F%2F%2A%5B%40attr_id=%27bgbl119s2562.pdf%27%5D#__bgbl__%2F%2F*%5B%40attr_id%3D%27bgbl119s2562.pdf%27%5D__1610879899265 (accessed December 9, 2021).

Google Scholar

19. Bundesanzeiger Verlag. Verordnung über das Verfahren und die Anforderungen zur Prüfung der Erstattungsfähigkeit digitaler Gesundheitsanwendungen in der gesetzlichen Krankenversicherung (Digitale Gesundheitsanwendungen-Verordnung - DiGAV) (2021). Available online at: https://www.bgbl.de/xaver/bgbl/start.xav?startbk=Bundesanzeiger_BGBl&jumpTo=bgbl120s0768.pdf#__bgbl__%2F%2F*%5B%40attr_id%3D%27bgbl120s0768.pdf%27%5D__1625152759885 (accessed December 9, 2021).

Google Scholar

20. European Commission. Proposed Guiding Principles for Reimbursement of Digital Health Products Solutions. Prepared for adoption by a SubGroup of the eHealth Stakeholder Group (2019). Available online at: https://www.medtecheurope.org/wp-content/uploads/2019/04/30042019_eHSGSubGroupReimbursement.pdf (accessed December 9, 2021).

Google Scholar

21. EUnetHTA. EUnetHTA Joint Action 2, Work Package 8. HTA Core Model ® version 3.0 (2016). Available online at: https://eunethta.eu/wp-content/uploads/2018/03/HTACoreModel3.0-1.pdf (accessed December 9, 2021).

Google Scholar

22. Mathews SC, McShea MJ, Hanley CL, Ravitz A, Labrique AB, Cohen AB. Digital health: a path to validation. NPJ Digit Med. (2019) 2:38. doi: 10.1038/s41746-019-0111-3

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Karpathakis K, Libow G, Potts HWW, Dixon S, Greaves F, Murray E. An evaluation service for digital public health interventions: user-centered design approach. J Med Internet Res. (2021) 23:e28356. doi: 10.2196/28356

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM) Das Fast Track Verfahren für digitale Gesundheitsanwendungen (DiHA) nach § 139e SGB V. Ein Leitfaden für Hersteller, Leistungserbringer und Anwender. (2020). Available online at: https://www.bfarm.de/SharedDocs/Downloads/DE/Service/Beratungsverfahren/DiHA-Leitfaden.pdf?__blob=publicationFile (accessed December 9, 2021).

Google Scholar

25. Moshi MR, Parsons J, Tooher R, Merlin T. Evaluation of mobile health applications: is regulatory policy up to the challenge? Int J Technol Assess Health Care. (2019) 35:351–60. doi: 10.1017/S0266462319000461

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Alami H, Gagnon M-P, Fortin J-P. Digital health and the challenge of health systems transformation. Mhealth. (2017) 3:31. doi: 10.21037/mhealth.2017.07.02

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Vukovic V, Favaretti C, Ricciardi W, Waure C de. Health technology assessment evidence on e-health/m-health technologies: evaluating the transparency and thoroughness. Int J Technol Assess Health Care. (2018) 34:87–96. doi: 10.1017/S0266462317004512

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Camacho E, Hoffman L, Lagan S, Rodriguez-Villa E, Rauseo-Ricupero N, Wisniewski H et al. Technology evaluation and assessment criteria for health apps (TEACH-Apps): pilot study. J Med Internet Res. (2020) 22:e18346. doi: 10.2196/18346

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Levine DM, Co Z, Newmark LP, Groisser AR, Holmgren AJ, Haas JS et al. Design and testing of a mobile health application rating tool. NPJ Digit Med. (2020) 3:74. doi: 10.1038/s41746-020-0268-9

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Torous J, Andersson G, Bertagnoli A, Christensen H, Cuijpers P, Firth J et al. Towards a consensus around standards for smartphone apps and digital mental health. World Psychiatry. (2019) 18:97–8. doi: 10.1002/wps.20592

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Henson P, David G, Albright K, Torous J. Deriving a practical framework for the evaluation of health apps. Lancet Digital Health. (2019) 1:e52–4. doi: 10.1016/S2589-7500(19)30013-5

PubMed Abstract | CrossRef Full Text | Google Scholar