ORIGINAL RESEARCH article
Sec. Drugs Outcomes Research and Policies
Volume 11 - 2020 | https://doi.org/10.3389/fphar.2020.608068
A Minimal Information Model for Potential Drug-Drug Interactions
- 1Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, United States
- 2Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, United States
- 3Department of Public Health Sciences, Clemson University, Clemson, SC, United States
- 4Pharmacy Consulting International (PCI), New York City, NY, United States
- 5Department of Software Engineering, Bahçeşehir University, Istanbul, Turkey
- 6Clinical Data Science, AstraZeneca, Cambridge, United Kingdom
- 7Institute of Data Science, Maastricht University, Maastricht, Netherlands
- 8Department of Computer Science, Georgia State University, Atlanta, GA, United States
- 9Fraunhofer Institute for Applied Information Technology, RWTH Aachen University, Aachen, Germany
- 10Department of Health Outcomes and Biomedical Informatics, University of Florida, Gainesville, FL, United States
- 11Mayo Clinic, Rochester, MN, United States
- 12Open Source Electronic Health Record Alliance, Washington, DC, United States
- 13IBM Research, Yorktown Heights, NY, United States
- 14The European Bioinformatics Institute, Birney Research Group, London, United Kingdom
- 15Genelex Corporation, Seattle, WA, United States
- 16School of Pharmacy, University of Washington, Seattle, WA, United States
- 17Southeast Louisiana Veterans Health Care System, New Orleans, LA, United States
- 18Department of Pharmacotherapy, University of Utah, Salt Lake City, UT, United States
- 19Department of Computer Science, Norwegian University of Science and Technology, Trondheim, Norway
- 20Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, United States
- 21School of Information Science, University of Illinois, Champaign, IL, United States
Despite the significant health impacts of adverse events associated with drug-drug interactions, no standard models exist for managing and sharing evidence describing potential interactions between medications. Minimal information models have been used in other communities to establish community consensus around simple models capable of communicating useful information. This paper reports on a new minimal information model for describing potential drug-drug interactions. A task force of the Semantic Web in Health Care and Life Sciences Community Group of the World-Wide Web consortium engaged informaticians and drug-drug interaction experts in in-depth examination of recent literature and specific potential interactions. A consensus set of information items was identified, along with example descriptions of selected potential drug-drug interactions (PDDIs). User profiles and use cases were developed to demonstrate the applicability of the model. Ten core information items were identified: drugs involved, clinical consequences, seriousness, operational classification statement, recommended action, mechanism of interaction, contextual information/modifying factors, evidence about a suspected drug-drug interaction, frequency of exposure, and frequency of harm to exposed persons. Eight best practice recommendations suggest how PDDI knowledge artifact creators can best use the 10 information items when synthesizing drug interaction evidence into artifacts intended to aid clinicians. This model has been included in a proposed implementation guide developed by the HL7 Clinical Decision Support Workgroup and in PDDIs published in the CDS Connect repository. The complete description of the model can be found at https://w3id.org/hclscg/pddi.
Ensuring that medication therapy occurs safely and to the maximum benefit of any given patient is of great interest to clinicians. One possible threat to patient safety comes from exposure to drug combinations that could interact to cause treatment failure, toxicity, or other unanticipated adverse drug reactions. While some potentially interacting combinations can benefit patients (e.g., by reducing the dose required for an expensive drug), drug interactions are more often a patient safety concern. Clinically important events attributable to drug interaction exposure occur in 5.3–14.3% of inpatients, and are responsible for up to 231,000 emergency department visits each year in the United States alone (Magro et al., 2012; Centers forDisease Control, 2018). A meta-analysis of 13 studies conducted on 3 continents found that 22.2% of hospital admissions associated with an adverse drug event were attributable to drug-drug interactions (interquartile range 16.6–36.0%) (Dechanont et al., 2014).
The development and delivery of clear and concise information about potential adverse events associated with drug-drug interactions is a long-standing clinical informatics challenge. Drug experts generate summaries of potential drug-drug interactions (PDDI) evidence from primary sources including the peer-reviewed scientific literature, drug product labeling, and case reports. These knowledge artifacts reach clinicians in the form of published drug information compendium, clinical decision support rules, and interaction checking applications. However, the knowledge content of the numerous resources describing PDDIs are far from uniform, with variations in goals and practice (Romagnoli et al., 2017) leading to significant discrepancies between resources (Wang et al., 2010; Saverno et al., 2011; Ayvaz et al., 2015; Ekstein et al., 2015; Fung et al., 2017). For example, a recent study comparing potential psychiatric drug interactions from six drug interaction database programs found agreement on the PDDI category for only 66% of 100 drug pairs. Fung et al compared 8.6 million unique pairs across 3 commercial PDDI knowledge bases and found that only 5% of PDDIs were mentioned in all 3KBs (Fung et al., 2017). We think a major issue underlying these discrepancies is that there is currently no broadly accepted standards to guide knowledge artifact creators in the organization and content of PDDI information. Given the costs of adverse events associated with drug interactions (Magro et al., 2012; Dechanont et al., 2014; Centers for Disease Control, 2018), improvements in the quality and utility of available information should contribute to improved care and reduced costs. As a means of bridging this gap, we have developed a minimal information model capable of presenting a common schema for describing evidence surrounding a PDDI.
Providing clinical guidance on the management of potential drug-drug interaction evidence requires synthesis and interpretation of multiple, possibly conflicting claims from evidence sources providing various levels of rigor and detail, including new drug announcements, published results of controlled trials, and spontaneous adverse event reports. Our investigation of the practices of experts involved in evaluating drug-drug interaction evidence found a range of personalized practices for both interpretation and synthesis of recommendations (Romagnoli et al., 2017). These variations in process and interpretation of evidence hinder sharing and comparison of recommendations, perhaps contributing to the aforementioned lack of consistency across information resources. Despite these variations, previous efforts established some recommended best practices for describing PDDIs. In 2005, van Roon, et al. (van Roon et al., 2005) published a model including a five-stage rating of evidence quality and a six-stage rating of seriousness of adverse events. A more recent effort by Seden et al. adapts the widely used GRADE (Grading of Recommendations Assessment, Development and Evaluation) methodology to evaluate PDDIs (Seden et al., 2017). This method is used by the various disease focused PDDI knowledge resources created by a collaborative effort led by the University of Liverpool (University of Liverpool, 2020a; University of Liverpool, 2020b; University of Liverpool, 2020c; University of Liverpool, 2020d). The Drug Interaction Probability Scale (DIPS) (Horn et al., 2007) provides a widely adopted structured means of assessing the likelihood of a drug-drug interaction in a specific case.
A series of efforts have attempted to build upon these tools to develop consensus models for PDDI information. A 2011 Delphi study involving 69 experts identified five key descriptors (patient status, probability of occurrence, risk factors, reaction severity, and evidence quality) (Riedmann et al., 2011). A conference series conducted in 2013–2014 developed usability recommendations for drug-drug interaction clinical decision support (Payne et al., 2015), evidence evaluation recommendations (Scheife et al., 2015), and guidance for selecting PDDIs for inclusion in clinical decision support systems (Tilson et al., 2016). A 2016 review of PDDI conceptual models identified 15 approaches with wide variations in granularity and coverage of key elements (Herrero-Zazo et al., 2016).
Building on all of these efforts, our goal was to develop community consensus around a common core set of elements to be included in PDDI knowledge artifacts. The new minimal information model would be used by drug experts to ensure that the PDDI knowledge artifacts they create have the core relevant information needed by the clinician end users. As a minimum information model, it should provide sufficient structure to be broadly useful without requiring substantial curatorial effort. Also, the minimal information model should focus exclusively on what information is to be collected, without addressing how it should be formatted or stored. This approach promotes flexibility and ease of use at the expense of possibilities for computational analysis (Brazma, 2009).
Critical to the domain of PDDIs, the information model should also help creators of PDDI knowledge artifacts include contextual factors that can help clinicians who use the artifacts to more specifically apply the knowledge to specific patient cases. This is because the specificity of a PDDI knowledge artifact to individual patient characteristics play a major role in acceptance of its recommendations (van der Sijs et al., 2006; Seidling et al., 2014). Using examples from PDDI clinical decision support, an in situ qualitative study of prescribers’ interaction with electronic medication alerts showed that prescribers bypassed the alert and then searched for patient-specific data that they needed when alerts failed to provide contextual information (Russ et al., 2012). Similarly, quality improvement projects have found that making PDDI alerts more appropriate to clinical context can improve alert acceptance (Daniels et al., 2019).
Development of the PDDI minimal information model was conducted under the auspices of the Health Care and Life Sciences Community Group (HCLSCG) of the World Wide Web Consortium (W3C - acronyms used this paper are summarized in Table 1). As a volunteer group, HCLSCG brings together experts interested in the use of Semantic Web tools in both medical and research applications. (Health Care and Life Sciences Community Group, 2018). Working with members of the HCLSCG group, including participants with both technical and clinical experience, we took a user-centered design approach to development of the minimal information model through four complementary activities: the selection of PDDIs used to inform the design of the minimal information model; the identification of user stories to define requirements; the construction of definitions for the elements of the information model; and the development of preliminary knowledge representation guidelines for the minimal information model.
The team started working collaboratively on the minimum information model in January of 2016. Discussion occurred using monthly “all-team” web meetings and additional web meetings focused on the development of use cases and exemplar PDDIs. After agreeing on the primary goals of the project, the team began a detailed discussion of information items mentioned in published consensus recommendations (Payne et al., 2015; Scheife et al., 2015) and development of real-world use cases. The team then used anonymous online surveys followed by team discussions to arrive at consensus on which information items to include in the minimum information model and the formal and user-centered definitions for the information items. A public draft version of the Community Group report was continuously edited as suggestions and disagreements were discussed and resolved. As a report reflecting consensus among the team reached maturity, drafts were sent out to the greater W3C community listserv for comments until a two week period passed without further suggestions. The final report was published in May of 2019 and is available online (https://w3id.org/hclscg/pddi). The next subsections summarize the methods for developing key knowledge artifacts included in the report. Further details on the information model’s development are available in the Appendix of the online report.
2.1 Decision Trees
Prior work by participants in the interest group sought to develop evidence-based clinical algorithms that consider a patient’s electronic health record information to provide a clinician with actionable information tailored to the patient’s specific context. Nelson et al. (2017); Rosko et al. (2017); The algorithms are formulated as decision trees, each containing a series of binary questions designed to explicitly constrain the decisions needed to interpret the appropriate response to a PDDI, leading to any of three different recommended actions—no special precautions; assess risk and take action if necessary; and use only if benefits outweigh the risk. Each decision tree should therefore result in a complete and unambiguous process for describing a PDDI.
The team developed decision trees for 14 PDDIs identified through an iterative process involving regular meetings with focused sub-teams. The 14 PDDIs (Table 2) were selected to demonstrate how an information model might help address known challenges with PDDI evidence. For example, two interactions (tamoxifen—paroxetine and potassium - potassium-sparing diuretics) were chosen to represent situations where information about the interaction can (and should) be contextualized for specific patients or clinical circumstances. The interaction between monoamine oxidase inhibitors - and indirect sympathomimetics was chosen because, the it applies for all drugs belonging to both drug classes. In contrast, tyrosine kinase inhibitors—proton pump inhibitors was chosen because the interaction is only relevant for kinase inhibitors that have pH dependent absorption.
TABLE 2. A total of 14 PDDI decision trees were created to represent different situations that developers of PDDI knowledge artifacts face. Knowledge artifacts following the minimum information model were created for each PDDI using logic flow diagrams, narrative explanations, and citations. Ten knowledge artifacts have been deposited in Zenodo. The other four are available from the authors upon request.
Analyses of questions needed to build the 14 decision trees, and their answers, informed selection of items for inclusion in the minimal information model. As part of this process, value sets for categories of medications (aldosterone antagonists, non-steroidal anti-inflammatory drugs, etc.) and clinical conditions (gastrointestinal bleeding, intracranial hermorrhage, etc.) were developed for several of the PDDI decision trees. Value sets are collections of codes from one or more terminologies that are useful for representing the specific drugs, conditions, observations, or other items of interest that have been coded in electronic health records. For example, a value set for paroxetine could include all clinical drug codes from the RxNorm terminology that are used when the electronic health records stores a prescription for any form of that medication. Value sets can be extensional, meaning that all included codes are simply listed, or intensional, meaning that the codes are specified using a workflow or logical definition. The value sets we created for the note were extensional and are available in the appendix of https://w3id.org/hclscg/pddi and also at the National Library of Medicine’s Value Set Authority Center (https://vsac.nlm.nih.gov/—search for value sets created by the “Meaningful Drug Interaction Alerts” organization).
2.2 User Stories
As our goal was to develop an information model to help PDDI knowledge artifact developers provide the information necessary to help clinicians make prescribing choices informed by accurate information regarding PDDIs, we developed a set of user stories to demonstrate the application of the minimal information model and potential utility for clinical challenges. These user stories focused on the challenges facing individuals involved in collating drug-drug interaction information, developing treatment plans, building clinical decision support systems, and otherwise providing appropriate information to clinicians. Each user story discussed instances where PDDI evidence described in the minimal information model might have been necessary to help provide effective care. Thus, pharmacists, physicians, nurses, librarians, and systems developers were considered to be potential users of the PDDI information model.
Initial stakeholder descriptions and scenarios were developed and used as the basis for further brainstorming with the assistance of a user experience expert. These efforts led to the development of a master list of tasks, users, information needs, and barriers to drug-drug interaction based decision-making in a variety of situations. A core set of user types was selected for development of user stories based on the scope of the minimum information model.
User stories for these core user types were developed based on an initial information needs list created by interest group members. This list was supplemented based on the results of a recent study of PDDI information needs of drug information compendium editors (Romagnoli et al., 2017). Where possible, user stories were based on PDDIs suggested by the interest group’s PDDI experts. All user stories were reviewed during team meetings to solicit feedback and comments. Based on interest group member suggestions, user stories were edited for clinical relevance, accuracy, and appropriateness. Elements of user stories were highlighted based on a color-coded key to indicate the minimum information model information item in question. Medication reconciliation use cases were specifically targeted as a compelling demonstration of the importance of the elements of the minimal information model.
After agreeing on a set of conceptual information items for the information model that were grounded in the user stories, we developed a draft definitions for them. The definitions were informed by entities present in the Drug-Drug Interaction Evidence Ontology (DIDEO) (Brochhausen et al., 2014) and the Drug-Drug Interactions Ontology (DINTO) (Herrero-Zazo et al., 2015). These draft definitions were then shared with all team members who were asked to provide suggestions for clarification. The final definitions, agreed upon by all report authors, were added to a new ontology called MPIO (http://www.ontobee.org/ontology/MPIO) for use in the minimal information model.
The complete model, including detailed use cases, two examples of application to PDDI (narrative and JSON), and other supporting information, can be found at https://w3id.org/hclscg/pddi.
3.1 User Stories
Twelve user stories involving eight types of users focused on key clinical challenges illustrate the potential uses of the minimal information model (Table 3). Three of these user stories focus on the critical task of medication reconciliation; others covered a range of topics, including treatment planning, evaluation of management options, screening for drug-drug interactions, and synthesis for dissemination. Several classes of user stories are repeated from the perspectives of different stakeholders, including physicians, nurses, pharmacists, drug compendium editors, librarians, and clinical decision support system staff. All user stories are annotated to indicate items from the minimal information model (an example is given in Table 4). As the initial focus was on clinical needs, researchers and regulators are not included explicitly in these initial user stories.
TABLE 3. Goals and users for user stories illustrating the use of the minimal information model. The complete user stories are at https://w3id.org/hclscg/pddi.
TABLE 4. Treatment planning user story example, annotated with instances of information model items.
3.2 Elements of the PDDI Minimal Information Model
The minimal information model contains 10 core elements (drugs involved, clinical consequences, seriousness, operational classification statement, recommended action, mechanism of interaction, contextual information/modifying factors, frequency of exposure to the PDDI, frequency of harm for persons who have been exposed to the PDDI, and evidence about a suspected drug-drug interaction). The definitions and eight detailed best practice recommendations are provided in https://w3id.org/hclscg/pddi. In this paper, the definitions are presented using two example PDDIs (warfarin and non-steroidal anti-inflammatory drugs and tyrosine-kinase inhibitors and proton pump inhibitors) in Tables 5, 6. A brief description of each core element provided here.
TABLE 5. The minimal information model and example elements as applied to potential interactions between Warfarin and non-steroidal anti-inflammatory drugs (NSAIDS).
TABLE 6. The minimal information model and example elements as applied to potential interactions between Tyrosine kinase inhibitors and Proton Pump Inhibitors.
3.2.1 Drugs Involved
Minimal information model PDDI Knowledge artifacts need to indicate the drugs involved in the PDDI with either extensional or intensional value sets consisting of concepts from widely adopted vocabularies such as RxNorm (National Institutes of Health, 2020a), or the Anatomical Therapeutic Chemical Classification System (WHO Collaborating Center for Drug Statistics Methodology, 2020) for drugs.
3.2.2 Clinical Consequences
Minimal information model PDDI Knowledge artifacts need to indicate the known or anticipated clinical consequences of exposure to the PDDI. This is important because many knowledge sources, including drug product labeling, report PDDIs that are based on pharmacologic properties (e.g., pharmacokinetics) without specifying what a potential clinical consequence. Without a clear clinical consequence, it is very hard for clinicians to asses the potential risks to the patient. Where possible, knowledge artifacts should indicate the clinical consequences using either extensional or intensional value sets consisting of concepts from widely adopted vocabularies such as ICD-10 (World Health Organization, 2020) or SNOMED-CT (SNOMED International, 2020).
Minimal information model PDDI Knowledge artifacts need to report if each known or anticipated clinical consequence is serious or not. Serious clinical consequences include death, life-threatening hospitalization, congenital anomaly, disability, or permanent impairment or damage. The seriousness entity was added to the model to bridge the gap between the community of pharmacy practitioners and creators of drug compendia and drug-drug interaction software tools, who commonly discuss PDDIs in terms of severity (Abarca et al., 2004; Roblek et al., 2015; Romagnoli et al., 2017), and the pharmacovigilance community, who describe events leading to death, hospitalizaiton, or persistent harm as being serious (WHO-Uppsala Monitoring Centre, 2020). The concept of severity is not included in the model. In place of severity, the report recommends the use of an operational classification statement (described below) which is thought be generally more clear and useful in a clinical decision making context.
An Operational Classification Statement is short risk classification statement that suggests a specific management criteria for a specific patient context. Minimal information model PDDI Knowledge artifacts need to report at least one such statement. The report suggests the use of the following categories from the OpeRational ClassificAtion of Drug Interactions system (Hansten et al., 2001):
• Avoid Combination (Risk of combination outweighs benefit)
• Usually Avoid Combination (use only under special circumstances)
• Interactions for which there are clearly preferable alternatives for one or both drugs.
• Interactions to avoid by using an alternative drug or other therapy unless the benefit is judged to outweigh the increased risk.
• Minimize Risk (Assess risk and take one or more of the following actions if needed)
• Consider alternatives: Alternatives may be available that are less likely to interact.
• Circumvent: Take action to minimize the interaction (without avoiding combination).
• Monitor: Early detection can minimize the risk of an adverse outcome.
• No Special Precautions (Risk of adverse outcome appears small)
• Ignore (Evidence suggests that the drugs do not interact)
Adopters of the minimum information model can expand on these or use their own operational classification statement system. In either case, recommended actions should be linked, if possible, to justifying evidence.
The Recommended Action element can be used independently, as a suggestion for dealing with any instance of the PDDI, or as modified by some contextual information/modifying factors (see below), to indicate guidance specific to patient factors or given drug formulations. A well written recommended action provides an evidence-based strategy to mitigate the potential clinical consequences of a drug-drug interaction such as a drug change, adjust drug dose, and monitor lab values. It should provide specific detailed suggestions that complement the Operational Classification Statement. The three basic recommended actions can be the same as the operational classification statements (No special precautions, Assess risk and take action if necessary, and Use only if benefit outweighs risk). Adopters of the minimum information model could expand on these or use their own system. In either case, recommended actions should be linked, if possible, to justifying evidence.
If known, a minimum information model PDDI knowledge artifact needs to include a statement about the mechanism of the PDDI. The description should be written for a clinician audience and include details that help the clinician decide what course of management action to take. To reduce ambiguity, the description may refer to specific drugs or health conditions using codes or value sets from widely used terminologies.
Contextual information/modifying factors are necessary for alerts that are both sensitive and specific. Like clinical consequences, minimum information model PDDI knowledge artifacts should indicate each known factor be stated as specifically as possible.
Many potential interactions are based on a small number of case reports or inference from pharmacologic properties. The two information items Frequency of exposure to the PDDI and Frequency of harm for persons who have been exposed to the PDDI help clarify if any evidence exists on the public health impact of the PDDI. While potentially very helpful for making a risk/benefit assessment about exposure to a PDDI, accurate frequency counts for may be unavailable in most cases. If so, “unknown” should be indicated to convey the limits of available information. In this way, minimum information model knowledge artifacts can help make apparent evidence gaps in need of further clinical research.
3.2.5 Evidence About a Suspected Drug-Drug Interaction
A minimum information model PDDI knowledge artifact needs to include evidence support for or in refutation of a given drug-drug interaction in humans. This evidence could potentially include data resulting from clinical studies, clinical observation, physiological experiments, or extrapolation based on drug-drug interaction mechanisms. The report’s best practice recommendations indicate that links to evidence are particularly essential to the artifact’s operational classification statement(s) recommended action(s).
A more detailed description of the model and its application can be found in the W3C note, available at https://w3id.org/hclscg/pddi.
3.3 A Word About Knowledge Representation
This section is intended for readers interested in knowledge representation aspects of the information model and could be skipped by non-interested readers. Ideally, representations of PDDI information would be fully-computable, expressed in a formal representation with well-specified semantics. However, the ambiguity of existing free-text descriptions of medications, interactions, contextual information, and other key elements makes this a long-term vision rather than a short-term goal. To maximize opportunities for automated analysis, the PDDI information model should be described using existing terminologies and ontologies wherever possible, while acknowledging that accurate capture of free text details can be useful as a last resort. Specific suggestions include:
3.3.1 Use of Biomedical Ontologies to Represent Items from the PDDI Information Model
All of the 10 core elements of the information model are represented in the Minimum PDDI Information Ontology (MPIO) (Boyce et al., 2019). MPIO classes extend classes from the Basic Formal Ontology (BFO) (Arp et al., 2015), supporting integration with the DIDEO (Brochhausen et al., 2014) and DINTO (Herrero-Zazo et al., 2015) ontologies mentioned above, as well as with ontologies relevant to descriptions of experiments, such as the Ontology for Biomedical Investigations (Bandrowski et al., 2016) and adverse events (He et al., 2014).
3.3.2 Representation of Model Elements as Information Content Entities
As statements of possible relationships, PDDI models do not specifically refer to objects that exist in the world. The theoretical nature of these statements is somewhat at odds with the accepted use of the BFO to describe existing objects. To avoid this difficulty, PDDIs are modeled as subclasses of the BFO-compliant class information content entity from the information artifact ontology (IAO) (Ruttenberg, 2020), indicating that each element is a statement that conveys some information, rather than a statement about an existing interaction in the world.
3.3.3 Use of Multiple Terminologies at Multiple Granularities
Drugs, contextual information/modifying factors, mechanisms of interaction, clinical consequences, and other elements of the PDDI model may be represented by multiple concepts from multiple sources, including RxNorm (National Institutes of Health, 2020), ATC (WHO Collaborating Center for Drug Statistics Methodology, 2020), ICD10 (World Health Organization, 2020), SNOMED-CT (SNOMED International, 2020). These concepts might be represented with varying degrees of specificity, ranging from specific products to broad classes of drugs, such as NSAIDs. Although the minimal information model suggests that concepts from accepted vocabularies should be used whenever possible, specific sources and granularities are not specified. Rather, concepts should be presented unambiguously through the combination of identifiers both for the source vocabulary and for the specific concept. If a potential interaction involves multiple drugs or outcomes that cannot be represented by a single concept, value sets might be used to describe multiple concepts. In cases where concepts cannot be represented by ontology terms, textual descriptions might be used as a last resort. When used, free-text summaries should be a specific as possible. Although not ideal, these unstructured descriptions are preferable to the omission of potentially useful information.
PDDIs present significant challenges in information representation and interpretation. Variations in descriptions of drugs, adverse events, sources of evidence, and patient characteristics are often confounded by challenges associated with incomplete information and evidence from differing sources, leading to differences in interpretation that might leave clinicians without clear, actionable guidance.
The proposed PDDI minimal information model builds on both prior work in minimal information models (Brazma et al., 2001; Taylor et al., 2008; McQuilton et al., 2016) and discussions with drug-drug interaction experts (Payne et al., 2015; Tilson et al., 2016), presenting a set of 10 items identified as most critical for conveying available understanding of PDDIs. These data elements, along with eight best practice recommendations and the proposed guidance on knowledge representation, present a first step toward a broader goal of fully-computable and semantically well-specified representations for drug-drug interaction information.
The PDDI minimal information model is already having an impact on health information standards and technology development activities. The model has been adopted by the HL7 Clinical Decision Support Workgroup, which works within the HL7 organization to develop clinical decision support standards (HL7, 2020a). The PDDI model has also be included in a new PDDI clinical decision support implementation guide that the HL7 community recently successfully balloted as a reference standard for trial use (Nguyen et al., 2019; HL7 Clinical Decision Support Workgroup, 2020). The implementation guide provides guidance for actionable PDDI alerts using sharable clinical decision support (CDS) artifacts that adhere to the minimum information model and the health information technology standards CDS Hooks (HL7, 2020b), Clinical Quality Language (CQL) (HL7, 2020c), and Fast Healthcare Interoperability Resources (FHIR). (Bender and Sartipi, 2013; Mandel et al., 2016; HL7, 2020b; HL7, 2020c; HL7, 2020d). Together, these efforts provide technical guidance for representing clinical data as stored in electronic medical records, querying systems for relevant data, and building tools that might integrate pertinent information into clinical workflows. Adoption of the PDDI model by these efforts provides a pathway for the inclusion of the PDDI information in widely-used clinical tools.
Realizing the vision of meaningful and highly actionable PDDI information will require additional work aimed at addressing several issues that have been raised by reviewers of early versions of the proposed model. Additional guidance on the distinction between the severity and seriousness elements may be needed to help unfamiliar users populate and interpret those terms appropriately. Additional examples, potentially including PDDIs described as being severe but not serious, might be particularly useful in this regard.
Experience with exploration of the warfarin-NSAID PDDI illustrates the importance of providing support for the use of customized value sets. As textual descriptions of PDDIs routinely lump concepts into categories (such as NSAIDs) that are not well-represented in existing ontologies, value sets provide a key tool for enabling appropriate specificity. Although the use of existing value sets developed through community efforts (National Institutes of Health, 2020b) might be the preferred approach, we anticipate that some PDDI descriptions will require novel value sets. A robust PDDI information ecosystem would enable sharing and reuse of such value sets wherever possible.
Although initially focused on meeting clinical needs, the proposed minimal information model might also be used to support broader technical, regulatory, and research goals. The W3C note (https://w3id.org/hclscg/pddi) provides justification for potential uses of the PDDI model for enhancing drug-drug interaction statements in structured product labels. The minimum information model can also be used to point out limitations in available information. Although clearly useful when available, the frequency of harm and frequency of exposure elements will often necessarily be specified as “unknown”. As large-scale observational efforts such as the Observational Health Data Sciences and Informatics (OHDSI) Hripcsak et al. (2015) effort continue to grow, retrospective extraction of such prevalence information from large cohorts is more of a practical possibility. For PDDIs identified through clinical trials, prevalence of harm within participating cohorts might provide information appropriate for these elements. The definition of well-specified models for expressing harm ratios will be an important step toward computable PDDI representations. Future enhancements might include expansion to include additional recommended data items designed to support researchers, regulators, and other perspectives.
Prior work on knowledge representation for PDDI evidence might be used to provide additional structure for the evidence item in the information model. As knowledge schemas designed to represent assertions from the research literature, Micro- and nano-publications (Schneider et al., 2015), might be used to model specific evidence statements. Particularly when combined with semantic classes from the Drug-Drug interaction and drug-drug interaction evidence ontology (Brochhausen et al., 2014) and the Drug-Drug Interactions Ontology (DINTO) (Herrero-Zazo et al., 2015), might bring needed structure appropriate for the interpretation and classification of evidence types and sources.
Appropriate tooling may be necessary to encourage potential users to adopt the PDDI information model. The MIAME model was eventually supported by a range of tools, including an XML markup language (Spellman et al., 2002), a spreadsheet format (Rayner et al., 2006), and the MAGE-TAB collection of tools supporting validation, visualization, and other key tasks (Rayner et al., 2009). The definition of the minimal information model contains preliminary examples of possible XML representations, but more work would be needed to develop a full schema. Similarly, further design work will be needed to support spreadsheet formats, a formal scheme, validation tools, and other support necessary to minimize the burden of completing compliant representations of PDDI information. Such efforts should leverage recent work in semantic metadata annotation (Egyedi et al., 2017).
Shared models of information describing the risks of PDDIs have the potential to improve the applicability of this information to clinical care, thus reducing adverse events, increasing medication safety, and improving outcomes. We present a proposed model containing 10 information items identified through a community consensus process: the drugs involved in a potential DDI, the clinical consequences, seriousness, an operational classification statement, recommended action, mechanism of interaction, contextual information/modifying factors, evidence about the suspected drug-drug interaction, frequency of exposure, and frequency of harm. These items, together with preliminary knowledge representation recommendations, present a first step toward improved collection, curation, and communication of PDDI information. We are committed to working with the community to promote and expand this model, with the goal of developing fully-computable information exchange of PDDI information. Further details are available at https://w3id.org/hclscg/pddi.
Data Availability Statement
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.
RB proposed the work and led the development of the information model. HH contributed to the development of the model and led the drafting and organization of the manuscript. All authors participated in the development of the minimum information model as described in the manuscript. They also have read, reviewed, and approved the final manuscript.
This work was supported by National Library of Medicine Grant No. R01 LM011838 and R15LM012941. Additional support was provided by grants U18 HS027099, R01HS025984 and R21HS023826 from the Agency for Healthcare Research and Quality (AHRQ). Any opinions, findings, and conclusion or recommendations expressed on the site are those of the site maintainers and do not necessarily reflect the views of AHRQ.
Conflict of Interest
Author EG was employed by the company Pharmacy Consulting International. Author OH was employed by IBM. Author BH was employed by Genelex.
The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
The full list of authors of the minimal information model can be found at https://w3id.org/hclscg/pddi.
Abarca, J., Malone, D. C., Armstrong, E. P., Grizzle, A. J., Hansten, P. D., Van Bergen, R. C., et al. (2004). Concordance of severity ratings provided in four drug interaction compendia. J. Am. Pharmaceut. Assoc. 44, 136–141. doi:10.1331/154434504773062582
Arp, R., Smith, B., and Spear, A. D. (2015). Building ontologies with basic formal ontology. Cambridge, MA: MIT Press.
Ayvaz, S., Horn, J., Hassanzadeh, O., Zhu, Q., Stan, J., Tatonetti, N. P., et al. (2015). Toward a complete dataset of drug–drug interaction information from publicly available sources. J. Biomed. Inf. 55, 206–217. doi:10.1016/j.jbi.2015.04.006
Bandrowski, A., Brinkman, R., Brochhausen, M., Brush, M. H., Bug, B., Chibucos, M. C., et al. (2016). The ontology for biomedical investigations. PLoS One 11, e0154556. doi:10.1371/journal.pone.0154556 |
Bender, D., and Sartipi, K. (2013). Hl7 FHIR: an agile and RESTful approach to healthcare information exchange. Proceedings of the 26th IEEE international symposium on computer-based medical systems, Porto, Portugal, June 20–22, 2013, 326–331. doi:10.1109/CBMS.2013.6627810
Boyce, R. D., Brochhausen, M., and Herrero-Zazo, M. (2019). MPIO minimum PDDI information ontology. Available at: http://www.obofoundry.org/ontology/mpio.html#:∼:text=MPIO%20(Minimum%20PDDI%20Information%20Ontology,alignment%20with%20DIDEO%20or%20DINTO (Accessed August 15, 2020). [Dataset]
Brazma, A., Hingamp, P., Quackenbush, J., Sherlock, G., Spellman, P., Stoeckert, C., et al. (2001). Minimum information about a microarray experiment (MIAME)–toward standards for microarray data. Nat. Genet. 29, 365–371. doi:10.1038/ng1201-365 |
Brazma, A. (2009). Minimum information about a microarray experiment (MIAME)–successes, failures, challengeS. ScientificWorldJournal 9, 420–423. doi:10.1100/tsw.2009.57 |
Brochhausen, M., Schneider, J., Malone, D., Empey, P. E., Hogan, W. R., and Bocye, R. D. (2014). Towards a foundational representation of potential drug-drug interaction knowledge. The first international workshop on drug interaction knowledge representation (DIKR), Houston, TX, October 6–7, 2014.
Centers for Disease Control. Faststats—emergency department visits (2018). Online: https://www.cdc.gov/nchs/fastats/emergency-department.htm (Accessed June 7, 2018). [Dataset]
Chung, L., Chakravarty, E. F., Kearns, P., Wang, C., and Bush, T. M. (2005). Bleeding complications in patients on celecoxib and warfarin1. J. Clin. Pharm. Therapeut. 30, 471–477. doi:10.1111/j.1365-2710.2005.00676.x
Daniels, C. C., Burlison, J. D., Baker, D. K., Robertson, J., Sablauer, A., Flynn, P. M., et al. (2019). Optimizing drug-drug interaction alerts using a multidimensional approach. Pediatrics 143, e20174111. doi:10.1542/peds.2017-4111 |
Dechanont, S., Maphanta, S., Butthum, B., and Kongkaew, C. (2014). Hospital admissions/visits associated with drug–drug interactions: a systematic review and meta-analysis. Pharmacoepidemiol. Drug Saf. 23, 489–497. doi:10.1002/pds.3592 |
Egorin, M. J., Shah, D. D., Christner, S. M., Yerk, M. A., Komazec, K. A., Appleman, L. R., et al. (2009). Effect of a proton pump inhibitor on the pharmacokinetics of imatinib. Br. J. Clin. Pharmacol. 68, 370–374. doi:10.1111/j.1365-2125.2009.03466.x |
Egyedi, A., O’Connor, M., Martínez-Romero, M., Willrett, D., Hardi, J., Graybeal, J., et al. (2017). “Embracing semantic technology for better metadata authoring in biomedicine,” in SWAT4HCLS: semantic web applications and tools for health care and life sciences. Available at: http://ceur-ws.org/Vol-2042/paper14.pdf. Rome, Italy: CEUR-WS Workshop Proceedings.
Ekstein, D., Tirosh, M., Eyal, Y., and Eyal, S. (2015). Drug interactions involving antiepileptic drugs: assessment of the consistency among three drug compendia and fda-approved labels. Epilepsy Behav. 44, 218–224. doi:10.1016/j.yebeh.2015.02.009 |
E.R. Squibb and Sons, L.L.C. (2018). Tech. rep. SPRYCEL- dasatinib tablet [package insert]. East Brunswick, NJ: E.R. Squibb and Sons, L.L.C.
Fung, K. W., Kapusnik-Uner, J., Cunningham, J., Higby-Baker, S., and Bodenreider, O. (2017). Comparison of three commercial knowledge bases for detection of drug-drug interactions in clinical decision support. J. Am. Med. Inf. Assoc. 24, 806–812. doi:10.1093/jamia/ocx010
Hansten, P. D., Horn, J. R., and Hazlet, T. K. (2001). Orca: operational classification of drug interactions. J. Am. Pharmaceut. Assoc. 41, 161–165. doi:10.1016/S1086-5802(16)31244-X
He, Y., Sarntivijai, S., Lin, Y., Xiang, Z., Guo, A., Zhang, S., et al. (2014). Oae: the ontology of adverse events. J. Biomed. Semant. 5, 29. doi:10.1186/2041-1480-5-29
Health Care and Life Sciences Community Group (2018). Semantic web in health care and life sciences community group. Online: https://www.w3.org/community/hclscg/. (Accessed May 31, 2018). [Dataset]
Herrero-Zazo, M., Segura-Bedmar, I., Hastings, J., and Dinto, M. P. (2015). Using owl ontologies and swrl rules to infer drug–drug interactions and their mechanisms. J. Chem. Inf. Model. 55, 1698–1707. doi:10.1021/acs.jcim.5b00119.PMID:26147071|
Herrero-Zazo, M., Segura-Bedmar, I., and Martínez, P. (2016). Conceptual models of drug-drug interactions: a summary of recent efforts. Knowl. Base Syst. 114, 99–107. doi:10.1016/j.knosys.2016.10.006
HL7 (2020b). CDS Hooks. Available at: https://cds-hooks.hl7.org/ (Accessed May 15, 2020). [Dataset].
HL7 Clinical Decision Support Workgroup (2020). Potential drug-drug interaction (PDDI) CDS implementation guide. Available at: http://hl7.org/fhir/uv/pddi/2020SEP/index.html (Accessed September 20, 2015). [Dataset]
HL7 (2020c). Clinical Quality Language (CQL). Available at: https://cql.hl7.org/ (Accessed September 20, 2015). [Dataset]
HL7 (2020d). FHIR v4.0.1. Available at: https://www.hl7.org/fhir/ (Accessed September 20, 2015). [Dataset]
HL7 (2020a). HL7 clinical decision support workgroup home. Available at: https://confluence.hl7.org/display/CDS/WorkGroup+Home (Accessed September 20, 2011). [Dataset]
Horn, J. R., Hansten, P. D., and Chan, L. N. (2007). Proposal for a new tool to evaluate drug interaction cases. Ann. Pharmacother. 41, 674–680. doi:10.1345/aph.1H423 |
Hripcsak, G., Duke, J. D., Shah, N. H., Reich, C. G., Huser, V., Schuemie, M. J., et al. (2015). Observational health data sciences and informatics (OHDSI): opportunities for observational researchers. Stud. Health Technol. Inf. 216, 574–578. doi:10.3233/978-1-61499-564-7-574
Magro, L., Moretti, U., and Leone, R. (2012). Epidemiology and characteristics of adverse drug reactions caused by drug–drug interactions. Expet Opin. Drug Saf. 11, 83–94. doi:10.1517/14740338.2012.631910 |
Malone, D. C., Hutchins, D. S., Haupert, H., Hansten, P., Duncan, B., Van Bergen, R. C., et al. (2005). Assessment of potential drug–drug interactions with a prescription claims database. Am. J. Health Syst. Pharm. 62, 1983–1991. doi:10.2146/ajhp040567 |
Mandel, J. C., Kreda, D. A., Mandl, K. D., Kohane, I. S., and Ramoni, R. B. (2016). Smart on fhir: a standards-based, interoperable apps platform for electronic health records. J. Am. Med. Inf. Assoc. 23, 899–908. doi:10.1093/jamia/ocv189
Masclee, G. M., Valkhoff, V. E., Coloma, P. M., de Ridder, M., Romio, S., Schuemie, M. J., et al. (2014). Risk of upper gastrointestinal bleeding from different drug combinations. Gastroenterology 147, 784792.e9. doi:10.1053/j.gastro.2014.06.007
McQuilton, P., Gonzalez-Beltran, A., Rocca-Serra, P., Thurston, M., Lister, A., Maguire, E., et al. (2016). Biosharing: curated and crowd-sourced metadata standards, databases and data policies in the life sciences. Database 2016, baw075. doi:10.1093/database/baw075 |
Millennium Pharmaceuticals, Inc. (2020). Iclusig-ponatinib hydrochloride tablet, film coated [package insert]. Cambridge, MA: Millennium Pharmaceuticals, Inc., Tech. Rep.
National Institutes of Health (2020a). RxNorm. Available at: https://www.nlm.nih.gov/research/umls/rxnorm/ (Accessed September 20, 2015). [Dataset]
National Institutes of Health (2020b). Value set authority center. Available at: https://vsac.nlm.nih.gov/ (Accessed September 20, 2015). [Dataset]
Nelson, S., Horn, J., Kapusnik-Uner, J., Desai, B., and Boyce, R. (2017). Should drug-drug interaction seriousness ratings Be taken seriously? Interactive panel. Philadelphia PA: AMIA iHealth 2017.
Nguyen, B. P., Reese, T., Decker, S., Malone, D., Boyce, R. D., and Beyan, O. (2019). Implementation of clinical decision support services to detect potential drug-drug interaction using clinical Quality Language. Stud. Health Technol. Inf. 264, 724–728. doi:10.3233/SHTI190318
Novartis (2020). TASIGNA-nilotinib capsule [package insert]. East Hanover, NJ: Novartis, Tech. Rep.
Payne, T. H., Hines, L. E., Chan, R. C., Hartman, S., Kapusnik-Uner, J., Russ, A. L., et al. (2015). Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J. Am. Med. Inf. Assoc. 22, 1243–1250. doi:10.1093/jamia/ocv011
Pfizer (2020). BOSULIF- bosutinib monohydrate tablet, film coated [package insert]. New York, NY: Pfizer, Tech. Rep.
Rayner, T. F., Rezwan, F. I., Lukk, M., Bradley, X. Z., Farne, A., Holloway, E., et al. (2009). Magetabulator, a suite of tools to support the microarray data format mage-tab. Bioinformatics 25, 279–280. doi:10.1093/bioinformatics/btn617 |
Rayner, T. F., Rocca-Serra, P., Spellman, P. T., Causton, H. C., Farne, A., Holloway, E., et al. (2006). A simple spreadsheet-based, miame-supportive format for microarray data: mage-tab. BMC Bioinf. 7, 489. doi:10.1186/1471-2105-7-489
Riedmann, D., Jung, M., Hackl, W. O., and Ammenwerth, E. (2011). How to improve the delivery of medication alerts within computerized physician order entry systems: an international delphi study. J. Am. Med. Inf. Assoc. 18, 760–766. doi:10.1136/amiajnl-2010-000006
Roblek, T., Vaupotic, T., Mrhar, A., and Lainscak, M. (2015). Drug-drug interaction software in clinical practice: a systematic review. Eur. J. Clin. Pharmacol. 71, 131–142. doi:10.1007/s00228-014-1786-7 |
Romagnoli, K. M., Nelson, S. D., Hines, L., Empey, P., Boyce, R. D., and Hochheiser, H. (2017). Information needs for making clinical recommendations about potential drug-drug interactions: a synthesis of literature review and interviews. BMC Med. Inf. Decis. Making 17, 21. doi:10.1186/s12911-017-0419-3 |
Rosko, S., Hansten, P., Horn, J., Malone, D., Romero, A., and Boyce, R. (2017). Toward shareable individualized drug interaction alerts,” AMIA summit on clinical research informatics. 2017 AMIA summit on clinical research informatics. San Francisco, CA, March 27–30, 2017.
Russ, A. L., Zillich, A. J., McManus, M. S., Doebbeling, B. N., and Saleem, J. J. (2012). Prescribers’ interactions with medication alerts at the point of prescribing: a multi-method, in situ investigation of the human–computer interaction. Int. J. Med. Inf. 81, 232–243. doi:10.1016/j.ijmedinf.2012.01.002
Ruttenberg, A. (2020). Information artifact ontology (IAO). Available at: https://github.com/information-artifact-ontology/IAO (Accessed September 20, 2015). [Dataset]
Saverno, K. R., Hines, L. E., Warholak, T. L., Grizzle, A. J., Babits, L., Clark, C., et al. (2011). Ability of pharmacy clinical decision-support software to alert users about clinically important drug–drug interactions. J. Am. Med. Inf. Assoc. 18, 32–37. doi:10.1136/jamia.2010.007609
Scheife, R. T., Hines, L. E., Boyce, R. D., Chung, S. P., Momper, J. D., Sommer, C. D., et al. (2015). Consensus recommendations for systematic evaluation of drug–drug interaction evidence for clinical decision support. Drug Saf. 38, 197–206. doi:10.1007/s40264-014-0262-8 |
Schneider, J., Brochhausen, M., Rosko, S., Ciccarese, P., Hogan, W. R., Malone, D., et al. (2015). “Formalizing knowledge and evidence about potential drug-drug interactions,” International workshop on biomedical data mining, modeling, and semantic integration (BDMI2015), Bethlehem, October 11, 2015.
Seden, K., Gibbons, S., Marzolini, C., Schapiro, J. M., Burger, D. M., Back, D. J., et al. (2017). Development of an evidence evaluation and synthesis system for drug-drug interactions, and its application to a systematic review of HIV and malaria co-infection. PLoS One 12, e0173509. doi:10.1371/journal.pone.0173509 |
Seidling, H. M., Klein, U., Schaier, M., Czock, D., Theile, D., Pruszydlo, M. G., et al. (2014). What, if all alerts were specific - estimating the potential impact on drug interaction alert burden. Int. J. Med. Inf. 83, 285–291. doi:10.1016/j.ijmedinf.2013.12.006
SNOMED International (2020). Get snomed ct. (Available at: https://www.snomed.org/snomed-ct/get-snomed (Accessed September 20, 2015). [Dataset]
Spellman, P. T., Miller, M., Stewart, J., Troup, C., Sarkans, U., Chervitz, S., et al. (2002). Design and implementation of microarray gene expression markup language (mage-ml). Genome Biol. 3, RESEARCH0046. doi:10.1186/gb-2002-3-9-research0046 |
Taylor, C. F., Field, D., Sansone, S. A., Aerts, J., Apweiler, R., Ashburner, M., et al. (2008). Promoting coherent minimum reporting guidelines for biological and biomedical investigations: the MIBBI project. Nat. Biotechnol. 26, 889–896. doi:10.1038/nbt.1411 |
Tilson, H., Hines, L. E., McEvoy, G., Weinstein, D. M., Hansten, P. D., Matuszewski, K., et al. (2016). Recommendations for selecting drug–drug interactions for clinical decision support. Am. J. Health Syst. Pharm. 73, 576–585. doi:10.2146/ajhp150565 |
University of Liverpool (2020a). Cancer drug interactions from radboud UMC and university of Liverpool. Available at: https://cancer-druginteractions.org/ (Accessed September 20, 2011). [Dataset]
University of Liverpool (2020d). Liverpool COVID-19 interactions. Available at: https://www.covid19-druginteractions.org/ (Accessed September 20, 2011). [Dataset]
University of Liverpool (2020c). Liverpool HEP interactions. Available at: https://www.hep-druginteractions.org/ (Accessed September 20, 2011). [Dataset]
University of Liverpool (2020b). Liverpool HIV interactions. Available at: https://www.hiv-druginteractions.org/ (Accessed September 20, 2011). [Dataset]
van der Sijs, H., Aarts, J., Vulto, A., and Berg, M. (2006). Overriding of drug safety alerts in computerized physician order entry. J. Am. Med. Inf. Assoc. 13, 138–147. doi:10.1197/jamia.M1809
van Roon, E. N., Flikweert, S., le Comte, M., Langendijk, P. N., Kwee-Zuiderwijk, W. J., Smits, P., et al. (2005). Clinical relevance of drug-drug interactions. Drug Saf. 28, 1131–1139. doi:10.2165/00002018-200528120-00007 |
Wang, L. M., Wong, M., Lightwood, J. M., and Cheng, C. M. (2010). Black box warning contraindicated comedications: concordance among three major drug interaction screening programs. Ann. Pharmacother. 44, 28–34. doi:10.1345/aph.1M475 |
WHO Collaborating Center for Drug Statistics Methodology (2020). ATC/DDD index 2020. Available at: https://www.whocc.no/atc_ddd_index/ (Accessed September 20, 2015). [Dataset]
WHO-Uppsala Monitoring Centre (2020). Glossary of pharmacovigilance terms. Available at: https://www.who-umc.org/global-pharmacovigilance/global-pharmacovigilance/glossary/ (Accessed September 20, 2015). [Dataset]
World Health Organization (2020). WHO—international classification of diseases. Available at: http://www.who.int/classifications/icd/en/ (Accessed September 20, 2015). [Dataset]
Yin, O. Q. P., Giles, F. J., Baccarani, M., le Coutre, P., Chiparus, O., Gallagher, N., et al. (2012). Concurrent use of proton pump inhibitors or H2 blockers did not adversely affect nilotinib efficacy in patients with chronic myeloid leukemia. Canc. Chemother. Pharmacol. 70, 345–350. doi:10.1007/s00280-012-1881-3 |
Keywords: drug-drug interaction, adverse drug events, minimal information model, clinical informatics, knowledge representation
Citation: Hochheiser H, Jing X, Garcia EA, Ayvaz S, Sahay R, Dumontier M, Banda JM, Beyan O, Brochhausen M, Draper E, Habiel S, Hassanzadeh O, Herrero-Zazo M, Hocum B, Horn J, LeBaron B, Malone DC, Nytrø Ø, Reese T, Romagnoli K, Schneider J, Zhang L(Y) and Boyce RD (2021) A Minimal Information Model for Potential Drug-Drug Interactions. Front. Pharmacol. 11:608068. doi: 10.3389/fphar.2020.608068
Received: 18 September 2020; Accepted: 29 December 2020;
Published: 08 March 2021.
Edited by:Ugo Moretti, University of Verona, Italy
Reviewed by:Emanuel Raschi, University of Bologna, Italy
Tanja Mueller, University of Strathclyde, United Kingdom
Copyright © 2021 Hochheiser, Jing, Garcia, Ayvaz, Sahay, Dumontier, Banda, Beyan, Brochhausen, Draper, Habiel, Hassanzadeh, Herrero-Zazo, Hocum, Horn, LeBaron, Malone, Nytrø, Reese, Romagnoli, Schneider, Zhang and Boyce. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Harry Hochheiser, email@example.com