# EDUCATING HEALTH PROFESSIONALS IN GENOMIC MEDICINE: EVIDENCE-BASED STRATEGIES AND APPROACHES

EDITED BY : Sylvia Ann Metcalfe, Clara Gaff and Michael Dougherty PUBLISHED IN : Frontiers in Genetics

#### Frontiers eBook Copyright Statement

The copyright in the text of individual articles in this eBook is the property of their respective authors or their respective institutions or funders. The copyright in graphics and images within each article may be subject to copyright of other parties. In both cases this is subject to a license granted to Frontiers. The compilation of articles constituting this eBook is the property of Frontiers.

Each article within this eBook, and the eBook itself, are published under the most recent version of the Creative Commons CC-BY licence. The version current at the date of publication of this eBook is CC-BY 4.0. If the CC-BY licence is updated, the licence granted by Frontiers is automatically updated to the new version.

When exercising any right under the CC-BY licence, Frontiers must be attributed as the original publisher of the article or eBook, as applicable.

Authors have the responsibility of ensuring that any graphics or other materials which are the property of others may be included in the CC-BY licence, but this should be checked before relying on the CC-BY licence to reproduce those materials. Any copyright notices relating to those materials must be complied with.

Copyright and source acknowledgement notices may not be removed and must be displayed in any copy, derivative work or partial copy which includes the elements in question.

All copyright, and all rights therein, are protected by national and international copyright laws. The above represents a summary only. For further information please read Frontiers' Conditions for Website Use and Copyright Statement, and the applicable CC-BY licence.

ISSN 1664-8714 ISBN 978-2-88966-059-9 DOI 10.3389/978-2-88966-059-9

#### About Frontiers

Frontiers is more than just an open-access publisher of scholarly articles: it is a pioneering approach to the world of academia, radically improving the way scholarly research is managed. The grand vision of Frontiers is a world where all people have an equal opportunity to seek, share and generate knowledge. Frontiers provides immediate and permanent online open access to all its publications, but this alone is not enough to realize our grand goals.

#### Frontiers Journal Series

The Frontiers Journal Series is a multi-tier and interdisciplinary set of open-access, online journals, promising a paradigm shift from the current review, selection and dissemination processes in academic publishing. All Frontiers journals are driven by researchers for researchers; therefore, they constitute a service to the scholarly community. At the same time, the Frontiers Journal Series operates on a revolutionary invention, the tiered publishing system, initially addressing specific communities of scholars, and gradually climbing up to broader public understanding, thus serving the interests of the lay society, too.

#### Dedication to Quality

Each Frontiers article is a landmark of the highest quality, thanks to genuinely collaborative interactions between authors and review editors, who include some of the world's best academicians. Research must be certified by peers before entering a stream of knowledge that may eventually reach the public - and shape society; therefore, Frontiers only applies the most rigorous and unbiased reviews.

Frontiers revolutionizes research publishing by freely delivering the most outstanding research, evaluated with no bias from both the academic and social point of view. By applying the most advanced information technologies, Frontiers is catapulting scholarly publishing into a new generation.

#### What are Frontiers Research Topics?

Frontiers Research Topics are very popular trademarks of the Frontiers Journals Series: they are collections of at least ten articles, all centered on a particular subject. With their unique mix of varied contributions from Original Research to Review Articles, Frontiers Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author by contacting the Frontiers Editorial Office: researchtopics@frontiersin.org

# EDUCATING HEALTH PROFESSIONALS IN GENOMIC MEDICINE: EVIDENCE-BASED STRATEGIES AND APPROACHES

Topic Editors: Sylvia Ann Metcalfe, Murdoch Childrens Research Institute, Royal Children's Hospital, Australia Clara Gaff, Melbourne Genomics Health Alliance, Australia Michael Dougherty, University of Colorado, United States

Citation: Metcalfe, S. A., Gaff, C., Dougherty, M., eds. (2020). Educating Health Professionals in Genomic Medicine: Evidence-Based Strategies and Approaches. Lausanne: Frontiers Media SA. doi: 10.3389/978-2-88966-059-9

# Table of Contents


Nirmala D. Sirisena and Vajira H. W. Dissanayake

	- Alexander Fedotov, Jordan Nestor, Chunhua Weng, Robb K. Rowley,

Georgia L. Wiesner, Gail P. Jarvik and Guilherme Del Fiol

*36 Ensuring Best Practice in Genomic Education and Evaluation: A Program Logic Approach*

Amy Nisselle, Melissa Martyn, Helen Jordan, Nadia Kaunein, Alison McEwen, Chirag Patel, Bronwyn Terrill, Michelle Bishop, Sylvia Metcalfe and Clara Gaff


Siobhan Simpson, Anneke Seller and Michelle Bishop

*86 The African Genomic Medicine Training Initiative (AGMT): Showcasing a Community and Framework Driven Genomic Medicine Training for Nurses in Africa*

Victoria Nembaware, African Genomic Medicine Training Initiative and Nicola Mulder


Belinda J. McClaren, Emily A. King, Erin Crellin, Clara Gaff, Sylvia A. Metcalfe and Amy Nisselle on behalf of the Australian Genomics Workforce & Education Working Group

# Editorial: Educating Health Professionals in Genomic Medicine: Evidence-Based Strategies and Approaches

#### Sylvia A. Metcalfe1,2 \*, Michael J. Dougherty <sup>3</sup> and Clara L. Gaff 1,2,4

*<sup>1</sup> Murdoch Childrens Research Institute, Royal Children's Hospital, Melbourne, VIC, Australia, <sup>2</sup> Department of Pediatrics, Melbourne Medical School, University of Melbourne, Melbourne, VIC, Australia, <sup>3</sup> School of Medicine, University of Colorado, Aurora, CO, United States, <sup>4</sup> Melbourne Genomics Health Alliance, Parkville, VIC, Australia*

Keywords: genomics education, evaluation, needs assessment, continuing professional education, program logic

#### **Editorial on the Research Topic**

#### **Educating Health Professionals in Genomic Medicine: Evidence-Based Strategies and Approaches**

With the rapid advancement of genomic technologies, particularly in the area of testing for human disease, genomics is being increasingly integrated into clinical care across many health disciplines. Nonetheless, there has been perceived lack of relevance by some non-genetic specialist health professionals and many challenges exist for genomic medicine to be successfully implemented (Joyner and Paneth, 2019). All specialists in genomic medicine will play an important role in preparing their non-specialist colleagues for this transformation in clinical care. New and innovative strategies both for education and system change will be required.

This special topic focusses on how evidence-based strategies and approaches can be used to develop and successfully implement education of health professionals in genomic medicine. This issue includes 12 articles on education in African countries, Australia, Canada, England, the Netherlands, Sri Lanka, and the United States of America, covering the educational needs of health care providers in genomic medicine and examples of emerging and successful educational activities, including their development, implementation, evaluation and outcomes.

The issue begins with two mini-review articles. The first, by Crellin et al., discusses the important role that a person's perceived need for learning plays in effective education, which we know from adult learning theory. They therefore reviewed the literature examining medical specialists' perceptions of genomics, drawing on studies from the earlier "genetic" era (due to the paucity of empirical studies published to date in the "genomic" era). They emphasize that the educational needs of medical specialists should be investigated to determine "if there is a need" and "how to meet the need," before tailoring educational interventions, and to encourage that education be considered part of a wider clinical implementation strategy. In her mini-review, Cornel et al. summarizes more than 30 years of research in Amsterdam on education for non-genetics experts. She notes that while some improvements have been seen in genetic competence, subsequent impacts on clinical practice and population health have been challenging to measure.

Nisselle et al. expand on the challenges of measuring the effectiveness of genomics education and advocate the use of program logic to develop and evaluate education interventions. Program logic models can help describe the inputs (such as stakeholder engagement and needs assessments), activities (such as development and delivery of the program), and intended outcomes of a program. Program logic models also include where and how evaluation can be targeted. The authors describe

#### Edited and reviewed by:

*Dov Greenbaum, Yale University, United States*

\*Correspondence: *Sylvia A. Metcalfe sylvia.metcalfe@mcri.edu.au*

#### Specialty section:

*This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics*

Received: *29 February 2020* Accepted: *05 June 2020* Published: *07 August 2020*

#### Citation:

*Metcalfe SA, Dougherty MJ and Gaff CL (2020) Editorial: Educating Health Professionals in Genomic Medicine: Evidence-Based Strategies and Approaches. Front. Genet. 11:696. doi: 10.3389/fgene.2020.00696*

**5**

the development of a generic program logic for genomics education that took place at a workshop with international experts in 2018 in Melbourne, Australia. They then report the results of testing the program logic in four diverse educational contexts and show that the model can be applied as a tool in multiple ways.

Several articles report on needs assessments to inform targeted education to a variety of health professionals. Saleh et al. conducted qualitative interviews with nurses, midwives, and allied health professionals in Australia to identify perceived genetic knowledge and education needs. They found that there was interest in genomics, tempered with uncertainty around how to access reliable resources and how to deal with challenges in incorporating education in clinical practice. In a separate large qualitative needs assessment with medical specialists in Australia, McClaren, Crellin et al. found that their participants believed confidence and skills in genomics clinical care require experiential learning (i.e., learning through reflection on doing); this mode of learning also includes interacting with their peers, especially "genomic champions," experts in their own specialty who have gained genomics expertise. Further findings from this study informed the development of a national survey, which is described in a second article by McClaren, King et al.. This paper describes the methodology, which used a mixed-methods approach and included additional interviews with education providers and a Delphi panel of experts, followed by piloting. To add to the rigor of survey development, the items were also informed by a theoretical framework of behavior change, the COM-B model: capability, opportunity, motivation and behavior.

In England, as genomic medicine is being rolled out through the National Health Service, there is a national coordinated approach to educating and upskilling health professionals. To inform these programs, Simpson et al. undertook a crossprofessional training needs analysis using a national survey and found that online learning was preferred by many. Their findings are providing an evidence base to inform resource development and an understanding of the motivations to engage in learning, which can aid in resource design. Among the suite of resources produced in England is a 3-week Massive Open Online Course (MOOC) on whole genome sequencing. Bishop et al. describe the rationale for choosing this type of learning resource, the process of development, including the recruitment and training of mentors, and the short-term evaluation outcomes.

Carroll et al. focused on primary care practitioners (family physicians) in Canada. They used a questionnaire to understand current involvement and confidence in genomic medicine, as well as attitudes about clinical validity, how genomic medicine could be integrated into primary care practice, and necessary resources and education. Their findings have informed the development of a website containing evidence-based resources, including pointof-care tools.

Clinical decision making and information for genomics education purposes can also be supported by tools embedded into electronic health records. Williams et al. in the USA describe some of the barriers to the effective use of electronic health records in supporting the clinical practice of genomics, and they identify "lessons learned" and several testable, potential solutions.

The studies and activities discussed so far are based in developed countries. In their opinion piece, Sirisena and Dissanayake from Sri Lanka begin by articulating the challenges of integrating genomic medicine in low- and middle-income countries and then discuss strategies for genomics education in these countries. In the final paper to be mentioned in this editorial, Nembaware et al. report on developments of the African Genomic Medicine Training Initiative (AGMT), in which they report on a program of training for nurses across 11 countries in Africa. They describe undertaking both a general and a targeted needs assessment and the construction of nurse personas to develop and map core competences adapted to the needs of the African continent. These personas and competences then informed the curriculum and evaluation plan. A blendedlearning course was subsequently implemented using trained community-based facilitators in virtual and physical classrooms in 19 different sites across Africa, with outcomes to be reported in due course.

Taken collectively, it is evident that major stakeholders in genomic healthcare systems recognize the importance of evaluation in education delivery and outcomes. Several examples include those responsible for health professional education (e.g., Health Education England, the Centre for Genetics Education, Australia), the professional genetics community (e.g., African Genomic Medicine Training Initiative), research networks (e.g., eMERGE), academic researchers and networks (e.g., Australian Genomics Health Alliance), and clinical providers (e.g., those using resources produced by Genetics Education Canada-Knowledge Organization and Geisinger's GenomeFIRST program). The challenges of ensuring sustained, effective education at a scale that ultimately and significantly improves patient care makes robust evaluation all the more important. Education funders, as well as those delivering education, need to be confident that an approach is—or can be—effective. The use of frameworks, such as those proposed by Nisselle et al., could move the field from evaluation of individual education programs to meta-analysis, yielding a robust body of knowledge to guide educators about interventions with strong evidence of effectiveness. This will advance the ultimate goal of improved patient care by educating clinicians about best practices in genomic medicine.

# AUTHOR CONTRIBUTIONS

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

# ACKNOWLEDGMENTS

The authors would like to thank all of the authors who contributed to this special issue.

# REFERENCES

Joyner, M. J., and Paneth, N. (2019). Promises, promises, and precision medicine. J. Clin. Invest. 129, 946–948. doi: 10.1172/JCI126119

**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2020 Metcalfe, Dougherty and Gaff. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Evidence-Based Genetic Education of Non-Genetic-Expert Physicians: Experiences Over Three Decades in Amsterdam

#### *Martina C. Cornel\**

*Amsterdam UMC, Vrije Universiteit Amsterdam, Clinical Genetics, Section Community Genetics, Amsterdam Public Health Research Institute, Amsterdam, Netherlands*

To study and improve the competences of health-care workers in the domain of genetics, attention needs to be paid to attitudes, activities, knowledge, and changes in performance. Three decades of research on genetic education for non-genetic-experts in Amsterdam are summarized, including both local and international collaborative efforts. Evidence shows that assessment of learners' needs and the definition of competences have driven slow but gradual improvement in genetics competence among non-geneticists. Attitudes and behavior are mainly influenced by face-to-face training. eLearning modules can serve to increase knowledge in a large number of participants in a rapidly changing field. Materials developed for accredited courses will sometimes be used for reference or just in time learning. Taking a theoretically informed evaluation approach, it has been possible to demonstrate satisfaction, improved knowledge, and self-reported behavioral change, although measuring effects on health-care practice and population health remains challenging. A flexible approach is needed to serve learners' needs in a field with many upcoming challenges.

Keywords: knowledge, genetics, genetic education, eLearning, competences, curriculum, continuing professional development

# INTRODUCTION

In the last decades, genetics and genomics research has generated many new insights, but the implications for health care so far have been modest. The publication of the sequence of the human genome was seen as a potential turning point. On 26 June 2000, Francis Collins stood next to the President of the United States, who announced the publication of the first survey of the entire human genome, and stated that "It will revolutionize the diagnosis, prevention, and treatment of most, if not all, human diseases" (Collins, 2010). However, 10 years later, Francis Collins concluded that while the revolution had not yet arrived, a few powerful new drugs against cancer and predictive genetic tests for a dozen conditions had become available (Collins, 2010). To enable the revolution, education of health-care providers was presented as one of the factors needed. This education should increase genetic knowledge and skills in physicians in domains outside of clinical genetics. In the last few decades, several studies have been performed in the Community Genetics Research group in Amsterdam, the Netherlands, in collaboration with colleagues from other countries, to improve genetic knowledge and skills and evaluate the approaches used. The tradition in Amsterdam is

#### *Edited by:*

*Michael Dougherty, University of Colorado Denver, United States*

#### *Reviewed by:*

*Melissa Martyn, Murdoch Childrens Research Institute, Australia Kate Louise McGregor Dunlop, NSW Health, Australia*

*\*Correspondence: Martina C. Cornel mc.cornel@amsterdamumc.nl*

#### *Specialty section:*

*This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics*

*Received: 01 April 2019 Accepted: 05 July 2019 Published: 02 August 2019*

#### *Citation:*

*Cornel MC (2019) Evidence-Based Genetic Education of Non-Genetic-Expert Physicians: Experiences Over Three Decades in Amsterdam. Front. Genet. 10:712. doi: 10.3389/fgene.2019.00712*

**8**

characterized by a strong evidence-based approach. In this paper, I will review lessons learned from three decades of genetic educational research. In some studies, there was a focus on activities, attitudes, and knowledge, as defined by experts, but there was also attention on educational needs as defined by nonexperts. Evaluation took place on different levels, including satisfaction, increase of knowledge, behavioral change, and use in health-care practice.

### ACTIVITIES AND ATTITUDES

In the Netherlands, general practitioners (GPs) have a role as gatekeepers in health care. If a couple has questions about their risk of having a child with congenital anomalies, they will first go to their GP, who can refer them to a specialist such as a clinical geneticist or obstetrician. A PhD thesis published in 1997, reported how well GPs in 1989 performed their task with regard to identifying and informing couples who are at increased risk of having a child with a congenital disorder (De Smit, 1997; Baars et al., 2003). A random sample of 124 GPs from the province of Noord-Holland (including Amsterdam) received a questionnaire, and 74% responded. Genetic counseling was defined as the provision of information on the chances of hereditary diseases and congenital disorders and on the possibilities of genetic examination, prenatal diagnosis, and pregnancy termination. Ten years later, the same GPs were investigated again, and 72% responded (Baars et al., 2003). The GPs recorded information on potential risk factors in their database for "previous child with congenital anomaly" in 57% (in 1989) and 63% (in 1999) and information on a serious congenital disorder in the close family in 15 and 13%, respectively. Information on consanguinity was recorded in 19 and 23%, respectively. GPs often gave oral information and, rarely, written information. In 1989, 82% supported directive counseling, and in 1999, this percentage had increased to 87% [measured as (strong) support for the statement "Genetic counseling should push the decisions of women and their partners on carrying out prenatal diagnosis in the right direction"]. The stance of clinical geneticists was that genetic counseling should be nondirective, especially for reproductive decisions, given the preference sensitive and value-laden decisions. The percentage of GPs that reported that they "(almost) always" referred women to a clinical geneticist for genetic counseling increased from 20% to 37%. The authors concluded that there was limited improvement in the GPs' activities over the 10-year time span (Baars et al., 2003). Around that time, an epidemiological study in the Northern Netherlands showed that 17% of couples who had a child with a congenital disorder were referred for genetic counseling (Cornel et al., 1992), and 10 years later, this percentage was 18% (Sikkens et al., 2002). The authors concluded that despite the increasing familiarity with genetics, the uptake of genetic counseling had not increased.

#### Curricula

Given the slow improvement of activities and attitudes, one might wonder what was taught in medical schools and specialist training. Given the fast developments in genetics, internationally, it was felt that more insight was needed in the content of medical education, and the "Genetic Education for Nongenetic Health Professionals" project (GenEd) was performed in 11 European countries (Challen et al., 2005). Wide variation was reported, and many countries lacked explicit genetics in their undergraduate, postgraduate, and continuing education. As part of the GenEd project, the medical curricula in eight medical faculties in the Netherlands in the year 2002 were investigated, as well as genetics in postgraduate training for non-genetics health-care professionals (Plass et al., 2006). Written documentation was studied and checked for accuracy with the genetic educators from each medical school. All medical curricula in The Netherlands used a list of "final goals of basic medical training." Two of the 328 health issues were genetic: "request for genetic evaluation" and "suspicion of genetic/ congenital anomaly," and three were frequently used in the context of genetics: "increased risk (positive test result of screening)," "request for preventive evaluation," and "request for information." Health issues were formulated in a rather general way, making it hard to identify specific fields of medicine. Genetics was relatively invisible in the curricula, often being integrated within a course (e.g. reproduction, developmental disorders). Thus, Plass et al. reported that it was hard to estimate the time spent on genetics.

As a very general development, many medical faculties in the Netherlands around 2002 used "problem-based" and increasingly "competence-based" curricula. Competences became the formal backbone of medical education in the Canadian Medical Education Directions for Specialists (CanMEDS) framework (Frank and Langer, 2003), which was used increasingly in medical faculties for graduate and postgraduate trainings. Thus, similar results might have been found for other basic sciences or fields of applied medicine (e.g., anatomy, histology, pathology, rehabilitation). Plass et al. (2006) included postgraduate training in their analysis. Out of 27 medical specialist training programs, only three (other than clinical genetics) indicated formal genetics training (obstetrics and gynecology, neurology and paediatrics) (Plass et al., 2006). The training of MDs for intellectually disabled people, which was not a recognized medical specialism at that time, also formally included genetics. As for continued education, MDs were obliged to follow postgraduate training, but they were free to choose from many topics. Only a few genetics courses were available. A postgraduate genetics course for obstetricians/ gynecologists existed, and a genetics course for cardiologists was being developed. The authors expressed the concern that genetics education was not only invisible but also insufficient.

#### KNOWLEDGE

Before the year 2000, clinical genetics had a strong focus on children with congenital anomalies. The population of patients referred was mainly children and their parents: couples looking for a diagnosis for their child and often considering reproductive decisions. Couples with a relative with a congenital anomaly were also referred for reproductive planning. After 2000, oncogenetics became a more frequent reason for referral. Medical curricula had not changed very much, and genetic issues were scarcely mentioned in the official final training goals. A study was done to evaluate knowledge of genetics relevant for daily practice in students nearing graduation (Baars et al., 2005b). Out of 855 questions on genetics selected from medical examinations and literature, 215 questions were selected for an examination administered by computer. These 215 questions were assessed by clinical geneticists for their relevance to daily medical (non-genetic) practice and classified as "essential," "desirable," and "too specialized." Participants were students in the final years of clerkships in seven out of eight of the medical faculties in the Netherlands. None of the students scored over 95% for "essential" knowledge, approximately a quarter of the students scored 60% or more for "desirable" knowledge, and most of the students scored over 40% for "too specialized" knowledge. Of the participants, 93% failed according to the cutoff score as defined by non-genetic health-care providers. Apparently, their knowledge was relatively good for issues that were less relevant for daily practice, while "essential" knowledge was often insufficient. It was hypothesized that in genetic education, too much attention is paid to specialized topics. The advice was that time spent on genetics should be spent more efficiently and should focus on knowledge that is relevant for daily practice (Baars et al., 2005b). While much of the research in Amsterdam focused on medical students and primary care physicians, the studies on knowledge also included gynecologists and pediatricians (Baars et al., 2005a). Average scores increased from GPs to gynecologists, pediatricians, and the clinical geneticist validation group. Overall genetic knowledge showed deficiencies for non-geneticist health-care providers. There was a specific lack of knowledge about DNA testing (Baars et al., 2005a).

# COMPETENCES NEEDED

To develop curricula for medical faculties, it is essential to define what a health-care professional needs to know and which competences are needed. A group of relevant health professionals and patients developed a set of core competences for different groups of health-care providers: GPs; genetic nurses/midwives; medical specialists in fields other than genetics; specialist nurses, specialist midwives, and specialist allied health professionals; specialist dentists; clinical geneticists; genetic specialist nurses or genetic counselors; molecular geneticists; cytogeneticists; and biochemists/biomedical scientists (Skirton et al., 2010). This was done in a collaborative project funded by the European Union: EuroGentest and under the auspices of the European Society of Human Genetics Education Committee. An exhaustive process of consultation took place, both with relevant health professionals and patient groups.

General competences include: to recognize individuals who may have a genetic condition; to be able to discuss this with patients and to refer; and to manage patients with a genetic condition and coordinate the care with other health-care workers. More specific competences were defined for clinical geneticists, genetic nurses or genetic counselors, molecular geneticists, cytogeneticists, and biochemists. These sets of competences can help countries to adjust their education and genetic service delivery systems for the future, according to a coherent set of standards (Skirton et al., 2010).

# DEVELOPING EDUCATIONAL MODULES BASED ON NEEDS

Building on the core competences defined by Skirton et al. (2010) and starting with assessment of the needs of primary health-care professionals, a comprehensive educational program for genetics was developed (Houwink et al., 2015). Given the fast developments in genetics, a flexible approach was chosen, which would also be suitable for future challenges in other fields of genetics. Midwives and GPs first reported their needs in a focus group study (Houwink 2011), after which prioritization took place in a Delphi procedure (Houwink et al., 2012). The top three genetic competencies were "recognizing signals that can indicate a hereditary component of a disease," "evaluating indications for referral to a clinical genetics centre," and "knowledge of the possibilities and limitations of genetic tests" (Houwink et al., 2012). These general competencies could in theory be applied in different fields (e.g. reproduction, cardiogenetics, oncogenetics). As the focal theme of the Dutch College of General Practitioners (NHG) was oncology, the competences were elaborated for oncogenetics. Three products were developed: an online continuing professional development module on oncogenetics (G-eCPD), a live genetic CPD module (interactive program taking oncogenetics as a model condition), and a supportive website (www.huisartsengenetica.nl, "GP and genetics"). For the evaluation of learning outcomes, Kirkpatrick's model was used (Kirkpatrick, 1967). The first level of Kirkpatrick's involves satisfaction, the second level knowledge, the third level behavioral change, and the fourth and highest level organizational change and health gain.

The eCPD was evaluated in a randomized controlled trial in 80 GPs (Houwink et al., 2014). Satisfaction was high, knowledge increase showed moderate effect sizes, also at 6 months follow-up (Houwink et al., 2014). The evaluation of learning outcomes at the lower levels is relatively simple, but providing evidence of behavioral change, organizational change, and health gain are challenging. The difficulty is partly related to the follow-up needed. As for the website, visitor numbers and percentage returning visitors could be reported. Website visitors often looked for information on basic genetics (drawing family trees, family history taking), which was not expected initially (Houwink et al., 2015). Participants of the live training reported more frequent referral of patients to the clinical genetics centers (68%) vs. 29% of participants of the eCPD (Houwink et al., 2015). On a regional population level, however, referral did not increase in the year after the modules. This might be due to the small number of participants and small number of referrals as compared with that of the entire region. A longer follow-up time and modules on other topics (e.g. reproduction and development, cardiogenetics) might be needed to achieve significantly more referrals by GPs.

# ECPD AND WEBSITE ON MULTIPLE TOPICS FOR MULTIPLE COUNTRIES

Many European countries face similar challenges related to genetic education. A European Union postgraduate education project, Gen-Equip, led by Prof. Heather Skirton, developed online continuing professional development (CPD) modules on nine topics (Paneque et al., 2017). As the challenges for genetics in primary care in different countries are very similar, the joint efforts made it possible to develop similar materials in six European languages. The online modules are supported by a website and webinars. The materials are available for free. Knowledge and skills increased significantly, and self-reported behavior changed (Jackson et al., 2019). Just like in the Houwink study, not only increasing skills in collecting family information and drawing pedigrees were mentioned but also knowing how to explain genetics to patients. Behavioral change was evidenced by participants who organized genetic training for their colleagues. While the modules were accredited for continuous education, users frequently did not ask for a certificate but came back for the materials to use "just-in-time."

#### CURRENT SITUATION IN NETHERLANDS

Clinical geneticists are involved in face-to-face training to groups of not only primary care physicians but also a diversity of other specialties and medical students. Online modules are developed for a range of rare diseases (e.g. monogenic subtypes of diabetes) and general issues (e.g. recognizing rare diseases), some of these in collaboration with patient organizations. Specialists other than clinical geneticists can now order some DNA tests (mainstreaming), and some specific modules for these purposes have been developed. A problem of this fragmented approach is that learners may not see certain challenges until they face them in practice. If they are unknowingly unable on, for instance, variants of unknown significance or the responsibilities

# REFERENCES


toward family members, they may not request support until they are overwhelmed. While some of the funding for previous genetic education projects came from the National Genomics Initiative, currently, no specific large-scale funding is available. The limited availability of funding leads to fragmentation, where the evidence-based approach to education and evaluation may be more difficult to achieve on a long-term and/or national scale. Evaluation at the higher levels of Kirkpatrick's requires a long-term involvement and may be difficult to achieve without dedicated funding.

# CONCLUSION

In the last decades, both genetic services and medical education underwent major changes. Problem-based learning and competence-based curricula gained importance, as did online learning modules. Given the underuse of the potential of genetics for health care, all of these strategies can help to improve the knowledge and skills relevant for daily practice. The challenge is to adapt to external changes in terms of technology and resources and patients' and learners' needs; particularly, learners are unknowingly unable for some aspects. Using an evidencebased approach to the development of modules can help to have most impact: learners' needs can be served best, and the flexible approach can integrate the challenges of tomorrow.

# AUTHOR CONTRIBUTIONS

MC wrote the article.

# ACKNOWLEDGMENTS

I would like to thank all colleagues who worked as researchers and participants on the studies cited.


primary care: a focus group study with multiple perspectives. *BMC Fam. Pract*. 12, 5. doi: 10.1186/1471-2296-12-5


Kirkpatrick, D. (1967). *Evaluation of training*. New York, NY: McGraw-Hill.


congenital anomaly in the Northern Netherlands. *Am. J. Med. Genet*. 112, 133– 137. doi: 10.1002/ajmg.10680

Skirton, H., Lewis, C., Kent, A., and Coviello, D. A. (2010). Members of Eurogentest Unit 6 and ESHG Education Committee. Genetic education and the challenge of genomic medicine: development of core competences to support preparation of health professionals in Europe. *Eur. J. Hum. Genet*. 18, 972–977. doi: 10.1038/ejhg.2010.64

**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Cornel. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Preparing Medical Specialists to Practice Genomic Medicine: Education an Essential Part of a Broader Strategy

*Erin Crellin1,2,3, Belinda McClaren1,2,3, Amy Nisselle1,2,3, Stephanie Best1,4, Clara Gaff1,2,3\* and Sylvia Metcalfe1,2,3*

*1 Australian Genomics Health Alliance, Melbourne, VIC, Australia, 2 Genomics in Society, Murdoch Children's Research Institute, Melbourne, VIC, Australia, 3 Department of Paediatrics, The University of Melbourne, Melbourne, VIC, Australia, 4 Australian Institute of Health Innovation, Macquarie University, Sydney, NSW, Australia*

#### *Edited by:*

*Go Yoshizawa, OsloMet—Oslo Metropolitan University, Norway*

#### *Reviewed by:*

*Scott McGrath, University of Nebraska Omaha, United States Divya Talwar, Children's Hospital of Philadelphia, United States*

> *\*Correspondence: Clara Gaff cgaff@unimelb.edu.au.*

#### *Specialty section:*

*This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics*

*Received: 26 April 2019 Accepted: 26 July 2019 Published: 11 September 2019*

#### *Citation:*

*Crellin E, McClaren B, Nisselle A, Best S, Gaff C and Metcalfe S (2019) Preparing Medical Specialists to Practice Genomic Medicine: Education an Essential Part of a Broader Strategy. Front. Genet. 10:789. doi: 10.3389/fgene.2019.00789*

Developing a competent workforce will be crucial to realizing the promise of genomic medicine. The preparedness of medical specialists without specific genetic qualifications to play a role in this workforce has long been questioned, prompting widespread calls for education across the spectrum of medical training. Adult learning theory indicates that for education to be effective, a perceived need to learn must first be established. Medical specialists have to perceive genomic medicine as relevant to their clinical practice. Here, we review what is currently known about medical specialists' perceptions of genomics, compare these findings to those from the genetics era, and identify areas for future research. Previous studies reveal that medical specialists' views on the clinical utility of genomic medicine are mixed and are often tempered by several concerns. Specialists generally perceive their confidence and understanding to be lacking; subsequently, they welcome additional educational support, although specific needs are rarely detailed. Similar findings from the genetics era suggest that these challenges are not necessarily new but on a different scale and relevant to more specialties as genomic applications expand. While existing strategies developed for genetic education and training may be suitable for genomic education and training, investigating the educational needs of a wider range of specialties is critically necessary to determine if tailored approaches are needed and, if so, to facilitate these. Other interventions are also required to address some of the additional challenges identified in this review, and we encourage readers to see education as part of a broader implementation strategy.

Keywords: medical specialist, workforce, genomic medicine, preparedness, theory, genomic education, review

# INTRODUCTION

Genomic medicine (i.e., the use of genomic information to guide diagnostic and treatment decisions) promises to transform the way medicine is practiced (Collins and McKusick, 2001; Williams, 2019). However, numerous challenges must be overcome for this promise (illustrated in **Table 1**) to be realized, including developing a competent workforce (Manolio et al., 2013; Bowdin et al., 2016). Medical specialists without specific genetic qualifications (defined herein as doctors specialized

**13**

TABLE 1 | Case study illustrating the promise of genomic medicine, derived from existing literature (Notarangelo and Fleisher, 2017; Stray-Pedersen et al., 2017).

#### Case

Rose is in her late teens and is continually in and out of hospital; she has suffered from serious, recurrent lung infections and autoimmune disease since childhood. Rose is suspected to have a primary immunodeficiency (PID), but a precise diagnosis remains elusive, despite repeated cellular and genetic testing.

#### Utility of Genomic Testing

Rose's immunologist recently heard about the diagnostic utility of genome sequencing, considers Rose a suitable candidate and hopes sequencing numerous genes in parallel may provide Rose with a more specific diagnosis and, potentially, help inform her treatment.

#### Outcome

Genomic testing pinpoints the genetic variant responsible for Rose's PID. This variant leads to overactivation of a protein that drives lymphocyte proliferation. Targeting this overactivated cell pathway with a readily available immunosuppressant is known to alleviate the severity of patients' disease. An immunosuppressant paradoxically helps treat an immunodeficiency. Without a genomic diagnosis, Rose's immunologist would never have thought of prescribing such a drug.

in a field other than general/family practice or clinical/medical genetics) will be key players in this workforce; however, their preparedness to practice genomic medicine has long been questioned (Guttmacher et al., 2001; Slade and Burton, 2016). It is widely feared that limited medical specialist knowledge and/or skills may see genomic tests misused or not used at all, to the detriment of patient care (Passamani, 2013; Korf et al., 2014; Burton et al., 2017). Consequently, there have been calls for educational efforts across the spectrum of medical training [i.e., from medical school and specialty training to continuing medical education (CME) (Guttmacher et al., 2007; McGrath and Ghersi, 2016)], with upskilling practicing medical specialists *via* CME the focus of this review.

#### Towards Effective Genomic Education

In response to the broader call for increased genomic education for medical specialists, the concept of what constituted a "prepared" medical specialist began to be considered. Vassy et al. (2015), drawing upon the competencies developed by Korf et al. (2014), proposed physicians would be sufficiently prepared if they had the knowledge and skills required to navigate genomic medicine and incorporate it into patient care. They stressed that as genomic practices are likely to be diverse, the nature of the knowledge and skills required will likely vary for different medical specialists. Yet, specific details as to how this might be successfully achieved were lacking from these early claims.

Here, we build upon this work and define preparedness as having the competence (knowledge, skills, and attitudes) and confidence to practice genomic medicine (whether it be identifying and referring suitable patients, or ordering and interpreting genomic tests) and propose that it could be achieved with greater efficacy and efficiency if CME approaches were grounded in adult learning theory.

#### Adult Learning Theory

According to adult learning theory, education of adults is most effective when they recognize a need to learn (i.e., when they are interested) and when education is tailored to the needs they self-identify, which arise from their work setting (Grant, 2002; Knowles et al., 2015). Problem-centered learning is preferred, as adults are keen to acquire knowledge and skills that are immediately applicable to real-life settings.

It is critical to emphasize that one cannot PRESUME a need to learn exists (Metcalfe et al., 2008) or that medical specialists will even be receptive to genomic medicine. After all, advances in genomics are not occurring in isolation; medical specialists have numerous competing learning demands and areas of interest (Feero et al., 2014). Investigating medical specialists' willingness to learn and potential educational needs is an essential first step and, as Reed et al. (2016) and others (Gaff et al., 2007; Houwink et al., 2011; Houwink et al., 2014) show, facilitates the design and delivery of effective educational interventions.

Here, we review what is currently known about medical specialists' perceptions of genomic medicine. Do specialists see a role for genomic medicine in their specialty, now or in the future? Would they feel confident using genomic tests? What do they know or think they should know about genomic testing and its use in clinical practice?

#### REVIEW METHODOLOGY

Initial searches of empirical literature on medical specialists' perceptions of genomics yielded limited results. As we believed that useful insights could be gained from medical specialists' earlier experiences with genetics, our literature search was subsequently broadened to include perceptions of genetic tests, too. Here, we define a genetic test as that which analyzes a single gene one at a time and a genomic test as that which analyses scores of (or all) genes simultaneously [see Brittain et al. (2017) for an overview of gene panels, whole exome sequencing (WES), and whole genome sequencing (WGS)]. Searches were conducted in MEDLINE, Embase, and PubMed using the search strategy detailed in the **Supplementary Material**, with articles focused on both germline and somatic testing examined.

#### GENETICS, GENOMICS, AND MEDICAL SPECIALISTS: A COMPLEX RELATIONSHIP Perceived Utility and Concerns

Views regarding the perceived relevance of genetics to conditions seen in clinical practice and utility of genetic testing varied across and within the specialties studied in the literature (Wilkins-Haug et al., 2000a; Hoop et al., 2008a; Hoop et al., 2008b; Harris et al., 2013; Myers et al., 2016; Amara et al., 2018; Diamonstein et al., 2018; Loss et al., 2018). For example, genetics was considered highly relevant and useful in obstetrics and pediatrics (Diamonstein et al., 2018) but less so in psychiatry (Hoop et al., 2008b) and general internal medicine (Diamonstein et al., 2018). Perceived utility is known to influence test use (Sanson-Fisher, 2004), exemplified by oncologists' rapid embrace of KRAS1

<sup>1</sup>Kirsten ras (KRAS) tumor mutation.

genetic testing for metastatic colorectal cancer when they were convinced that such testing would usefully inform treatment decisions (Harris et al., 2013).

Although the value of genetic testing was often recognized, a number of concerns, primarily relating to test access and implications for patients, were often raised across studies (Freedman et al., 2003; Finn et al., 2005; Harris et al., 2013; Salm et al., 2014; Myers et al., 2016). Perceptions of genomics, which largely emanate from the oncology field to date, are proving to be similarly mixed, with perceived benefits often tempered by a host of concerns, some old, some new.

Some oncologists (Gray et al., 2014; Chow-White et al., 2017; Johnson et al., 2017), pediatric neurologists (Jaitovich Groisman et al., 2017), and neonatologists (Knapp et al., 2019) believed that genomic tests would be useful for facilitating diagnoses and family planning, guiding treatment selection, or aiding disease surveillance. Yet, across studies, many specialists questioned the current utility of genomic testing (Miller et al., 2014; Chow-White et al., 2017; Deininger et al., 2019; Knapp et al., 2019), with few treatments available and genomic information yet to be fully deciphered.

Of the limited studies conducted to date, concerns raised included genomic test access and cost (Helman et al., 2016; Chow-White et al., 2017; Jaitovich Groisman et al., 2017), lack of evidence and clinical guidelines (Bonter et al., 2011; Stanek et al., 2012; Amara et al., 2018), and the potential for genomic tests to cause psychological harm or impede insurance access (Johnson et al., 2017; Deininger et al., 2019; Knapp et al., 2019). These concerns linger from the genetics era, with additional worries arising from the complexity, volume, and uncertain nature of the data generated (Miller et al., 2014; Christensen et al., 2016; Gray et al., 2016; Knapp et al., 2019). For instance, some oncologists (Gray et al., 2016; Weipert et al., 2018) and cardiologists (Christensen et al., 2016) participating in various genomics studies were worried about being burdened with the responsibility of disclosing additional findings (e.g., cancer predispositions or conditions that lay outside their specialty). Other oncologists (Miller et al., 2014; McCullough et al., 2016) were troubled by the potential for additional findings to cause undue worry or distract patients or parents from their/their child's primary condition. Yet, despite holding numerous concerns, specialists often saw the infiltration of genomics into medicine as inevitable (Selkirk et al., 2013; Chow-White et al., 2017; Jaitovich Groisman et al., 2017), an inevitability, as indicated in the section that follows, for which few felt prepared.

# Understanding and Confidence

Medical specialists' perceived or actual knowledge of genetic concepts, conditions, and/or testing have long been shown to be highly variable and frequently poor (Hofman et al., 1993; Hunter et al., 1998; van Langen et al., 2003; Baars et al., 2005; Hoop et al., 2008b; Nippert et al., 2011; Klitzman et al., 2013). Canadian specialists surveyed by Hunter et al. (1998), for instance, had poor knowledge of the availability of genetic tests for specific conditions, with further studies suggesting that knowledge levels have not improved since. For example, the majority of European primary care specialists (pediatricians and obstetrician–gynecologists) surveyed by Nippert et al. (2011) expressed limited confidence in their ability to identify/explain inheritance patterns and perform other such tasks, and most US specialists surveyed by Klitzman et al. (2013) perceived their genetic knowledge to be very/somewhat poor. That said, genetic knowledge often varied by specialty. Specialties and subspecialties (for example, cardiologists subspecialized in cardiogenetics) with greater genetics exposure often had higher perceived or actual knowledge of genetic concepts, conditions, and/or testing (Hofman et al., 1993; Pichert et al., 2003; van Langen et al., 2003; Baars et al., 2005; Nippert et al., 2011). These studies imply that the impetus to know about genetics is greatest when it is perceived to be directly relevant to one's clinical practice, in line with adult learning theory.

Comfort to discuss or use genetics in practice, while often low (Klitzman et al., 2013), also differed across specialties, reflected in the various roles that specialists were willing to assume. Neurologists, for instance, appeared to be more comfortable ordering genetic tests and interpreting and discussing test results compared with psychiatrists (Finn et al., 2005; Salm et al., 2014; Zhou et al., 2014; Dominguez-Carral et al., 2017). Self-confidence was often a product of genetics experience (with neurologists in the preceding example having cause to use genetic tests more frequently than psychiatrists) and a predictor of future test use (Freedman et al., 2003; Salm et al., 2014). Moreover, those who had received some genetic education were often more confident and knowledgeable and used genetics more frequently (Hofman et al., 1993; Wilkins-Haug et al., 2000b; Hoop et al., 2008b; Nippert et al., 2011), supporting a role for education in facilitating competent practice.

A similar lack of preparedness is emerging from the genomics literature, with specialists mostly expressing low confidence in their understanding of, and ability to use, somatic or germline genomic tests (Bonter et al., 2011; Selkirk et al., 2013; Gray et al., 2014; Amara et al., 2018; Deininger et al., 2019; Knapp et al., 2019) but self-reporting familiarity with basic genetic concepts (Stanek et al., 2012; Chow-White et al., 2017). Knowledge and confidence have often been shown to be highest among oncologists compared with other specialties (Bonter et al., 2011; Stanek et al., 2012); however, confidence levels are even relatively low among this experienced group of genetic/genomic test users, particularly with regards to germline results (Chow-White et al., 2017; Johnson et al., 2017; Weipert et al., 2018).

Findings from somatic and/or germline studies with oncologists (Chow-White et al., 2017; Johnson et al., 2017; Weipert et al., 2018) and neonatologists (Knapp et al., 2019) indicate comprehending and communicating genomic information, with colleagues or patients, will be challenging for most. However, there is some evidence, albeit from a small qualitative study of pediatric oncologists involved with tumor genomics (McCullough et al., 2016), to suggest that some individuals do not consider genomic information as any more complex or daunting to communicate than the tasks they currently perform.

Given specialists' perceived lack of preparedness to practice genomic medicine, it is unsurprising that many strongly supported additional education, training, and resources, such as clinical guidelines (Bonter et al., 2011; Selkirk et al., 2013; Chow-White et al., 2017; Jaitovich Groisman et al., 2017; Weipert et al., 2018; Deininger et al., 2019). Yet, few studies have explored specialists' preferences in any great depth (Selkirk et al., 2013; Weipert et al., 2018; Deininger et al., 2019). The survey of oncologists involved with a tumor genomics study by Chow-White et al. (2017) suggests that specialists are keen to learn about the practicalities of genomic testing and that participating in genomics research is a useful means of gaining knowledge and skills, but further evidence is severely lacking.

Education is likely to work best when key conditions are met; for instance, clinical utility is recognized. Neurologists, in the study by Jaitovich Groisman et al. (2017), who saw a use for genome sequencing in their future practices were far more supportive of education compared with those who did not. Clearly, exploring medical specialists' perceptions of genomic medicine is a useful starting point for gauging interest in, and need for, educational support.

Self-confidence and perceived competence remain strong predictors of future (somatic or germline) test use (Gray et al., 2014; Johnson et al., 2017). A qualitative study by Weipert et al. (2018) also suggests that these constructs are a product of one's work setting. Several oncologists in their study felt that community-based practitioners had less exposure to genomics than their counterparts working in academic/tertiary settings and therefore would be less competent ordering and interpreting genomic tests. This perception is yet to be verified, though. Confidence and perceived competence additionally appear to be products of genomic education and experience (Bonter et al., 2011; Stanek et al., 2012; Selkirk et al., 2013; Amara et al., 2018), once again supporting a role for education in facilitating competent practice.

#### ROLE OF EDUCATION IN PREPARING MEDICAL SPECIALISTS TO PRACTICE GENOMIC MEDICINE

This review suggests a range of factors (e.g., perceived utility and consequences, confidence, experience level, education, and resources available) are likely to influence medical specialists' preparedness to practice genomic medicine, echoing findings from a systematic review by Paul et al. (2018) of factors influencing medical specialists' use of genetic tests. In light of this, it is worthwhile reflecting on the role education will play in mainstreaming genomic medicine. To do so, we need to take a step back and see where education fits within a broader genomic medicine implementation strategy.

Implementation science is a field that uses a range of behavior change theories to systematically study ways of getting evidence into practice (Michie et al., 2005; Bauer et al., 2015). As part of a learning healthcare system, focused on continual improvement and where collaboration among diverse stakeholders including clinicians is critical, implementation science can provide the mechanism for considering future implementation strategies (Chambers et al., 2016; Gaff et al., 2017). Different theories can be used to study potential barriers and facilitators, guide the selection and implementation of suitable behavior change interventions, and subsequently evaluate intervention efficacy (Lynch et al., 2018). Theories are used to create generalizable results and because it is widely recognized that theory-driven approaches are more likely to work (Bauer et al., 2015).

The Capability, Opportunity, and Motivation Model of Behavior (COM-B model) is a theory commonly used to identify barriers and facilitators to the adoption of new practices like genomic medicine and to aid the selection of behavior change interventions, for example, educational supports (Michie et al., 2011; McDonagh et al., 2018). According to this model, three interacting constructs result in a behavior (e.g., referring patients/ordering genomic tests to guide diagnostic and treatment decisions): capability, the requisite knowledge and skills; opportunity, the support of a well-resourced environment and one's peers; and motivation, the self-belief that one is capable of performing a given task and the practice will lead to positive, not negative, outcomes. These constructs, illustrated in the context of genomic medicine, are further detailed in **Table 2**.

As various aspects of these three constructs can enhance or impede behavior change (Michie et al., 2011), a suite of interventions will likely be needed to see genomic medicine successfully integrated into routine practice. The capability and motivation constructs can be amenable to education and training, supported by evidence from systematic reviews (Grol and Grimshaw, 2003; Paneque et al., 2016; Talwar et al., 2017), which consistently show that education can help improve competence (knowledge, skills, and attitudes) and confidence.

TABLE 2 | The Capability, Opportunity, Motivation Behavior (COM-B) model adapted from Michie et al. (2011) to apply to genomic medicine. Three intersecting constructs will likely determine the successful use of genomic medicine in medical specialist practice. The capability and motivation constructs can be amenable to education and training.


However, behavior change with education alone is rare. This may be because: a) long-term outcomes are difficult to measure (although attempts to measure these outcomes should be considered prior to developing educational interventions; Talwar et al., 2017) and b) other interventions need to be delivered alongside education for education to have additional impact (Grol and Grimshaw, 2003). CME in genomic medicine is likely to be most effective as an essential part of a broader implementation strategy.

Since multidisciplinary teamwork is known to be critical for successful implementation (Chambers et al., 2016), upskilling other health professions who work alongside medical specialists and providing interdisciplinary education to those working in the same clinical setting, previously identified as being relevant to genetics (Gaff et al., 2008), is additionally and importantly needed.

#### SUMMARY AND CONCLUSIONS

Findings from the very limited empirical studies conducted to date (largely in the field of oncology) suggest that medical specialists' perceptions of genomic medicine are likely to be complex. Mixed views on the clinical utility of genomic medicine currently exist, with perceived benefits frequently tempered by several concerns. At the same time, specialists generally consider the arrival of genomic medicine inevitable. Most do not feel prepared for this inevitability and perceive a lack of understanding and confidence. While little evidence exists, there is indication that CME in genomic medicine is likely to be broadly welcomed.

Similar findings from the genetics era suggest that these challenges are not necessarily new in the genomics era but occur on a larger scale and are likely to be relevant to more specialties as genomic applications expand across medicine (Burton et al., 2017; Knapp et al., 2019). Informing medical specialists that genomics is, in many ways, a continuation of genetics may be reassuring to those daunted by the impending arrival of genomic medicine. It also suggests that existing strategies for genetic education and training may be transferable to genomic education and training. Given the limited resources available for genomic education, repurposing and sharing educational materials, where possible, through online repositories will be important (Nisselle, submitted). Equally important will be improving the quality of evaluation approaches, noting that while existing educational strategies may be transferable across different settings, evaluation of these strategies will likely need to be different (Talwar et al., 2017). Current efforts often lack methodological rigor, are infrequently guided by theory, and rarely include follow-up data to determine long-term impact. The COM-B model introduced in this review could be one theory used to guide evaluation approaches.

To test the hypothesis that existing educational strategies may be transferable, a perceived need must be confirmed. Detailed explorations of educational needs should be undertaken, in a wider range of medical specialties and more diverse settings (most studies reviewed arose from academic/tertiary hospitals; the needs of community-based practitioners are largely unknown). Findings from the genetics era revealed that the perceived relevance of genetics varied across specialties. Whether this remains the case in the genomics era is unknown and worth investigating.

We are investigating the perceptions, experiences, and education and training needs of varied health professionals (including medical specialists) in the Workforce & Education program of the Australian Genomics Health Alliance (Stark et al., 2019). Guided by the principles of adult learning theory, which have previously informed health professional genetic/genomic needs assessments (Gaff et al., 2007; Metcalfe et al., 2008; Reed et al., 2016), we seek to investigate the perceived relevance of genomic medicine to clinical practice and document education and training needs, should they exist. Establishing this evidence base will be critical to facilitate the implementation of tailored educational supports.

# AUTHOR CONTRIBUTIONS

EC, CG, SM, AN, and BM conceived the idea for the manuscript, and EC conducted the literature searches, reviewed the literature, and drafted and revised the manuscript. CG, SM, AN, and BM provided intellectual input throughout and revised drafts. SB provided substantial input into the implementation science component and revised drafts. All authors approved the final version and agree to be accountable for all aspects of the work.

# FUNDING

This work was supported by the Victorian Government's Operational Infrastructure Support Program and a grant from the Australian National Health & Medical Research Council (GNT1113531); the contents are solely the responsibility of the individual authors and do not reflect the views of the NHMRC. EC was supported by The University of Melbourne Helen R. Freeman Scholarship.

# ACKNOWLEDGMENTS

The authors would like to thank Poh Chua for her assistance with literature searches, Kushani Jayasinghe for her assistance with inclusion and exclusion criteria, and Emily King for her input on describing the application of the COM-B model to genomic medicine.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fgene.2019.00789/ full#supplementary-material

#### REFERENCES


attitudes, behaviors, and needs for training. *J. Genet. Couns.* 23, 156–163. doi: 10.1007/s10897-013-9624-0


**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest**.**

*Copyright © 2019 Crellin, McClaren, Nisselle, Best, Gaff and Metcalfe. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Strategies for Genomic Medicine Education in Low- and Middle-Income Countries

*Nirmala D. Sirisena\* and Vajira H. W. Dissanayake*

*Human Genetics Unit, Faculty of Medicine, University of Colombo, Colombo, Sri Lanka*

Keywords: developing world, genomic education, genomic literacy, genomic medicine, healthcare

# INTRODUCTION

Implementing genetic and genomic medicine is dependent to a large extent on the successful training of a genomics workforce with expertise in interpreting, communicating, and integrating genomic information in a clinical setting. In order to effectively implement genomic medicine at different levels of healthcare delivery, strategies for establishing training in core competencies of genetics and genomics targeted at the undergraduate, postgraduate, and continuing professional development levels need to be in place. Several approaches have been adopted in Western countries like the UK and USA to ensure that their healthcare workforce is adequately trained and competent to effectively use genetic and genomic information in their professional practice. However, this is not the case in most low- and middle-income countries (LMICs) located in regions of East Asia and the Pacific, Central and South Asia, Latin America, and the Caribbean, North and Sub-Saharan Africa, with a gross national income of \$1,026–\$3,995. In many of these countries, this necessity has been plagued by numerous challenges stemming from the lack of local capacity to plan and carry out the required training of the healthcare workforce. The other contributory factors are the scarcity of adequate funding for training as well as establishing core facilities needed for delivering these services around which such training programs could be implemented and delivered (Sirisena and Dissanayake, 2018). Herein, we provide a concise overview of the various challenges faced in achieving genomic literacy for integrating genomic medicine into the healthcare setting in LMICs and potential strategies for overcoming such limitations.

#### *Edited by:*

*Sylvia Ann Metcalfe, Murdoch Childrens Research Institute (MCRI), Australia*

#### *Reviewed by:*

*Alice Matimba, Wellcome Genome Campus, United Kingdom*

#### *\*Correspondence:*

*Nirmala D. Sirisena nirmala@anat.cmb.ac.lk*

#### *Specialty section:*

*This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics*

*Received: 03 July 2019 Accepted: 05 September 2019 Published: 08 October 2019*

#### *Citation:*

*Sirisena ND and Dissanayake VHW (2019) Strategies for Genomic Medicine Education in Low- and Middle-Income Countries. Front. Genet. 10:944. doi: 10.3389/fgene.2019.00944*

# SITUATION ANALYSIS OF THE CHALLENGES

Several initiatives have promoted genomic research and infrastructure and capacity development in many LMICs such as the Human Heredity and Health in Africa (H3Africa), the Qatar Genome Project, the Mexico National Institute of Genomic Medicine (INMEGEN), and the Collaborative African Genomics Network (CAfGEN) (Tekola-Ayele and Rotimi, 2015; Mlotshwa et al., 2017; Mboowa and Sserwadda, 2019). As genomic technologies rapidly advance and genomic sequencing becomes increasingly affordable, even in the LMICs, immense volumes of genomic data are generated with potential for guiding clinical decision making in the healthcare setting. However, the advent of such clinically actionable genomic information creates a dilemma as most healthcare providers in these countries are not competent in interpreting and communicating these results due to inadequate genomics knowledge and skills, thereby depriving patients from making informed decisions regarding personalized, targeted disease screening, prevention, diagnostics, and treatment approaches that can influence health and disease management (Metcalfe et al., 2002; Guttmacher et al., 2007; Cohn et al., 2015; Mboowa and Sserwadda, 2019). Thus, in the current genomic era, it is vital that LMICs take necessary measures to educate and build up a

**20**

healthcare workforce that is effectively trained to integrate genetic and genomic information into their clinical practice (de Abrew et al., 2014; Sirisena et al., 2016a). However, the practical challenges for implementing such educational initiatives are quite diverse (Guttmacher et al., 2007; de Abrew et al., 2014; Sirisena et al., 2016b; Sirisena and Dissanayake, 2018). Some of them are addressed below.

A major impediment in most LMICs is the lack of personnel trained in genetics, genomics, and bioinformatics who could serve as a core team to plan and develop training programs and clinical and laboratory facilities around which such programs could be implemented and delivered (Wonkam et al., 2010; de Abrew et al., 2014). The lack of adequate infrastructure such as cytogenetic and molecular genetic laboratories and tools for providing quality training and services is a huge setback. Disparities in health priorities in most LMICs is an important factor contributing to lack of sufficient funding for developing, implementing, and sustaining genomic-based educational initiatives (Sirisena and Dissanayake, 2018). Consequentially, this has led to the slow pace of translation of genomics research from the bench to the bedside resulting in a lack of perception of the clinical relevance of genomics and its clinical utility and potential benefit for improving health-related patient outcomes (Tekola-Ayele and Rotimi, 2015).

Lack of access to genomic-based educational resources and e-learning tools in local languages for training at the secondary, tertiary, and continuing professional development levels, lack of access to internet facilities and/or the skills and confidence to use web-based learning resources by some healthcare providers are further deterrents (Mitropoulos et al., 2015). Another limiting factor is the time constraints of busy healthcare providers who find it difficult to keep up with the rapid pace of clinical genomic advances and thereby tend to pursue only those educational opportunities that cater for the immediate needs of their patients (Skirton et al., 2010; de Abrew et al., 2014; Tekola-Ayele and Rotimi, 2015).

Additional challenges include institutional matters and differences in the educational systems across the LMICs, such as the structure and sequence of existing undergraduate medical curricula resulting in significant differences in the content and delivery of genomic education. Misconceptions and flawed assumptions among medical students and health professionals that diseases fall strictly into genetic and non-genetic categories rather than into a continuum of interaction between genetic and non-genetic components are other limiting factors (Skirton et al., 2010; de Abrew et al., 2014; Tekola-Ayele and Rotimi, 2015).

#### STRATEGIES AND WAY FORWARD

Genomics‐related educational initiatives to improve the genetic and genomic literacy among healthcare professionals in LMICs would require a multi-faceted approach, depending on the national priorities and the financial capabilities of each country. The pre-requisites needed for the development of genetic and genomics literacy include the following: recognition of the need, definition of the knowledge and skills required, development and implementation of educational initiatives and evaluation to assess the achievement of the desired outcomes (Gaff et al., 2007; Thurston et al., 2007; Skirton et al., 2010; de Abrew et al., 2014). Some of the core areas in which healthcare professionals need to develop competency in include: genetic variation in health and disease, the role of the family history in determining the modes of inheritance of genetic disorders and assessment of genetic risk, indications for referral for genetic evaluation and testing, assessing the clinical validity and utility of genetic testing for specific clinical conditions, ordering and interpreting genetic and genomic tests, communicating genomic information effectively, genetic counselling and facilitating informed decision making by patients, integrating genetic information into clinical management decisions, and the complex ethical and psychosocial issues related to genetics and genomics (Guttmacher et al., 2007; Telner et al., 2008; Korf, 2013). Educational programs on genetics and genomics should ideally incorporate a pre-service education component for those in training prior to their onset of clinical practice as well as a continuing education component along with professional practice guidelines to cater for those currently in clinical practice (de Abrew et al., 2014; Korf et al., 2014; Manolio et al., 2015). Five entrustable professional activities (EPAs) that encompass a basic set of genomic skills with clinical applications across different levels of healthcare and between medical specialties have been identified. They include: family history, genomic testing, genomic-guided therapeutics, somatic cancer genomics, and microbial genomic information (Korf et al., 2014; Institute of Medicine, 2015).

Education and training in the basics of genomics and bioinformatics could be introduced at the level of preundergraduate education while more advanced training could be at the undergraduate and postgraduate levels. It is also necessary to address any misconceptions among medical students and health professionals and create awareness that genomics underlies the whole of pathophysiology and constitutes the fundamental science of health and disease and should therefore not be treated solely as a medical specialty having implications for only a few areas of clinical practice (Guttmacher et al., 2007).

Even though basic genetics content focusing mainly on the rare Mendelian disorders is integrated into the basic sciences courses of most medical undergraduate curricula in varying depths and durations, there is a need for it to be applied across the entire curriculum, ending with real life patients during clinical training through inclusion of case-based clinical examples to illustrate the genetic and genomic determinants and mechanisms underlying common complex diseases such as heart disease, hypertension, diabetes mellitus, etc. (Korf, 2002; Guttmacher et al., 2007; Telner et al., 2008). Such approaches would facilitate bridging the gap between the basic sciences and the clinical training and instill in the trainee the perception that genetics and genomics is clinically relevant. In situations where such integration is non-existent, revision of the undergraduate medical curricula to incorporate genetics and genomics modules tailor-made towards disease conditions that are relevant in the local context is warranted (Mboowa and Sserwadda, 2019).

At the postgraduate level, training in genetics and genomics should be incorporated into both specialty and sub-specialty training programs. It is also important that examinations for licensure and certification should include a substantial number of genetics-related questions. Additional strategies for building bridges between genetics and genomics and other specialties include the establishment of joint specialty training programs that combine medical genetics and genomics with another major discipline and the development of subspecialty training for individuals trained outside of medical genetics and genomics and introduction of new graduate programs in genetics and genomics (Mboowa and Sserwadda, 2019; Sirisena and Dissanayake, 2019). Thus, it would be necessary for the largely public funded academic institutions in LMICs to take the necessary steps to request for increased allocation of funds from national budgets for the realization of the above outcomes (Sirisena and Dissanayake, 2017).

In order to cater for the needs of healthcare professionals currently in clinical practice who have not received training in basic genetics content during their undergraduate medical training, continuing professional development programs incorporating basic educational materials should be introduced by hospitals and health systems, professional medical associations and societies to equip clinicians to provide some genetic services on their own, while providing clear guidelines for referral to genetics specialists when necessary (Houwink et al., 2011; Sirisena and Dissanayake, 2019). Specialized training such as genomics workshops, seminars, postgraduate courses, and massive open online courses also provide opportunities for clinicians to obtain up-to-date information with the purpose of improving genetics and genomics knowledge, attitudes, and skills in a cost-effective and time-efficient manner (Sirisena et al., 2016a; Mboowa and Sserwadda, 2019). Many organizations have developed or are in the process of developing such point-of-care, electronic decision-support systems and continuing professional development courses for healthcare providers based on the best practices in adult learning, such as interactivity, case-based learning, and skill-focused objectives (Reed et al., 2016). A needs-driven, learner-centric, evidencebased, outcomes-oriented, and practice-embedded continuing medical education system has been shown to contribute to improved quality of care and patient outcomes (Institute of Medicine, 2015).

#### REFERENCES


Alternative effective educational strategies include providing access to online or print versions of medical genetics and genomics journals as well as textbooks containing the core knowledge and the latest advances in the field of genetics and genomics (Guttmacher et al., 2007). Due to the rapid pace at which genomics technology advances, educational strategies need to be designed in such a way so as to keep the workforce continually up-to-date in diagnostic and therapeutic measures, especially pertaining to pharmacogenomics and the use of tumor genomic data for precise molecular diagnosis of cancer, selecting targeted therapy, and monitoring of response to treatment (Slade et al., 2016).

Extensive barriers would first need to be overcome for the successful integration of the advances in genetic and genomic technologies into clinical care. In this regard, national healthcare system planners, administrators, and policy makers of LMICs would first need to establish collaborative ties and seek technical assistance from healthcare institutions abroad and from inter-governmental agencies such as the World Health Organization (Wonkam et al., 2010; Tekola-Ayele and Rotimi, 2015). This would enable them to overcome the existing lack of local human, technological, and financial resources. It would also empower them to advance genetic and genomic capacity building by fostering the development of genomic educational initiatives beginning from premedical education through medical specialty training to subspecialty training. Such collaborative efforts would help lay a solid foundation for building up the genomic literacy of the healthcare workforce in their respective countries within the context of their national economic and socio-cultural uniqueness.

#### AUTHOR CONTRIBUTIONS

NS gathered the literature data and wrote the manuscript. VD critically revised the manuscript for important intellectual content. Both authors read and approved the final manuscript.


preparation of health professionals in Europe. *Eur. J. Hum. Genet.* 18, 972–977. doi: 10.1038/ejhg.2010.64


**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Sirisena and Dissanayake. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Genomic Information for Clinicians in the Electronic Health Record: Lessons Learned From the Clinical Genome Resource Project and the Electronic Medical Records and Genomics Network

#### *Edited by:*

*Sylvia Ann Metcalfe, Murdoch Childrens Research Institute (MCRI), Australia*

#### *Reviewed by:*

*Janice Fletcher, South Australia Pathology, Australia Yvonne Bombard, University of Toronto, Canada Chloe Mighton, University of Toronto, Canada, in collaboration with reviewer YB*

#### *\*Correspondence:*

*Marc S. Williams mswilliams1@geisinger.edu*

#### *Specialty section:*

*This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics*

*Received: 03 July 2019 Accepted: 03 October 2019 Published: 29 October 2019*

#### *Citation:*

*Williams MS, Taylor CO, Walton NA, Goehringer SR, Aronson S, Freimuth RR, Rasmussen LV, Hall ES, Prows CA, Chung WK, Fedotov A, Nestor J, Weng C, Rowley RK, Wiesner GL, Jarvik GP and Del Fiol G (2019) Genomic Information for Clinicians in the Electronic Health Record: Lessons Learned From the Clinical Genome Resource Project and the Electronic Medical Records and Genomics Network. Front. Genet. 10:1059. doi: 10.3389/fgene.2019.01059*

*Marc S. Williams1\*, Casey Overby Taylor1,2, Nephi A. Walton1, Scott R. Goehringer1, Samuel Aronson3, Robert R. Freimuth4, Luke V. Rasmussen5, Eric S. Hall6, Cynthia A. Prows7, Wendy K. Chung8, Alexander Fedotov9, Jordan Nestor10, Chunhua Weng11, Robb K. Rowley12, Georgia L. Wiesner13, Gail P. Jarvik14 and Guilherme Del Fiol15*

*1 Genomic Medicine Institute, Geisinger, Danville, PA, United States, 2 Department of Medicine, Johns Hopkins University, Baltimore, MD, United States, 3 Partners HealthCare, Boston, MA, United States, 4 Department of Health Sciences Research, Mayo Clinic, Rochester, MN, United States, 5 Department of Preventive Medicine, Northwestern University, Chicago, IL, United States, 6 Department of Pediatrics, University of Cincinnati College of Medicine, and Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States, 7 Divisions of Human Genetics and Patient Services, Cincinnati Children's Hospital Medical Center, Cincinnati, OH, United States, 8 Departments of Pediatrics and Medicine, Columbia University, New York, NY, United States, 9 Irving Institute for Clinical and Translational Research, Columbia University, New York, NY, United States, 10 Department of Medicine, Division of Nephrology, Columbia University, New York, NY, United States, 11 Department of Biomedical Informatics, Columbia University, New York, NY, United States, 12 National Human Genome Research Institute, Bethesda, MD, United States, 13 Division of Genetic Medicine, Department of Internal Medicine, Vanderbilt University Medical Center, Nashville, TN, United States, 14 Departments of Medicine (Medical Genetics) and Genome Sciences, University of Washington, Seattle, WA, United States, 15 Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, United States*

Genomic knowledge is being translated into clinical care. To fully realize the value, it is critical to place credible information in the hands of clinicians in time to support clinical decision making. The electronic health record is an essential component of clinician workflow. Utilizing the electronic health record to present information to support the use of genomic medicine in clinical care to improve outcomes represents a tremendous opportunity. However, there are numerous barriers that prevent the effective use of the electronic health record for this purpose. The electronic health record working groups of the Electronic Medical Records and Genomics (eMERGE) Network and the Clinical Genome Resource (ClinGen) project, along with other groups, have been defining these barriers, to allow the development of solutions that can be tested using implementation pilots. In this paper, we present "lessons learned" from these efforts to inform future efforts leading to the development of effective and sustainable solutions that will support the realization of genomic medicine.

Keywords: genomics, electronic health record, education, clinical decision support, infobutton, knowledge synthesis, interoperability, implementation

# INTRODUCTION

Genomic information is increasingly used in clinical care. However, genomics can only improve healthcare if clinicians and patients are able to identify when genomic information may be useful and, given the durable nature of genomic information, coupled with increased knowledge that enhances interpretation over time, apply the information over the patient's life span. Clinicians without genetic training consistently state they are unprepared to use genomic information to care for their patients (Mikat-Stevens et al., 2015; Pet et al., 2019). There is also concern about where to find reliable information to guide the use of genomic results. Traditional educational approaches to improve genomic knowledge are necessary but insufficient, given the dynamic nature of genomic discovery and rapidly changing knowledge relevant to the use of genomics in the care of patients. This necessitates innovative approaches to storage, knowledge synthesis, representation, retrieval, and presentation, ideally integrated into a redesigned clinician workflow supporting the delivery of relevant genomic information provided "just in time" to support clinical care. The electronic health record (EHR) ecosystem is expected to play a key role in this area (Hoffman 2007; Hoffman and Williams, 2011). In this paper, we will review the lessons learned from two large projects developing approaches to educate clinicians within the EHR.

# MATERIALS AND METHODS

#### Setting

The work was done in two large research projects funded by the National Human Genome Research Institute (NHGRI).

The Electronic Medical Records and Genomics (eMERGE) Network1 was initially funded in 2007 with the goal of developing and studying the EHR as a tool for genomic research. It is currently completing its third cycle of funding. Phase 1 was a proof of concept that demonstrated that EHR data can be used to develop reliable clinical phenotypes, which can subsequently be used for genomic discovery (primarily for genome-wide association studies). In Phase 2, in addition to expanding the phenotyping work of Phase 1, the network began to explore how the EHR could be used to deliver genomic results to clinicians and patients *via* pilot implementations. Phase 3 has been focused on the implementation of genomic medicine in the clinic, where 25,000 participants were sequenced using targeted next-generation sequencing (eMERGEseq)2 . This custom assay sequenced a set of 109 actionable genes as well as other single nucleotide variants (SNVs), including genes from version 1 of the American College of Medical Genetics and Genomics (ACMG) secondary findings list (Green et al., 2013). Sites received results to return to participants (Kullo et al., 2014; Jarvik et al., 2014; eMERGE Consortium, 2019).

The Clinical Genome Resource (ClinGen) project was initially funded in 2013. The goal of this project is to increase the medical

community's knowledge about the relationship between genes and health. The primary task is building a knowledge base that defines the clinical relevance of genes and variants for use in precision medicine and research.

Recognizing the importance of the EHR to support the return of results, the Electronic Health Record Integration (EHRI) Working Group was established in Phase 2 of the eMERGE project3 . The EHRI studied use of the EHR to store genomic test reports and present the results to clinicians and patients. The EHR is also being used to capture patient outcomes related to the return of results. Several tools have been developed by the EHRI that have the potential to impact clinician education. ClinGen established an Electronic Health Record Working Group (EHR WG) tasked with identifying strategies to provide access to ClinGen through the EHR. Liaisons were established between the eMERGE EHRI and ClinGen EHR WG committees to coordinate efforts and accelerate progress. Through their respective evaluation of EHR functionality, the groups developed strategies to accomplish these goals.

Given the novel nature of the problems and resulting strategies, little prior work was available to guide the groups' respective efforts. Therefore, an exploratory approach was used where potential solutions to problems were developed through an informal group process. Volunteers then tested the prototype solutions in development environments associated with the EHR. The results of these pilot implementations are brought back to the groups for discussion and iterative improvement of the tools. This process, while informal, is informed by conceptual frameworks, or desiderata, for genomic data and clinician education and decision support proposed by Masys et al. (2012) and Welch et al. (2014).

While this paper focuses on two specific initiatives, the NHGRI has other funded projects that are using the EHR for genomic medicine. Liaisons to the relevant workgroups and projects are in place to coordinate efforts and disseminate successful strategies. These will be discussed below.

# RESULTS

In 2012 Masys et al., defined a set of technical desiderata for the integration of genomic data into the EHR (Masys et al., 2012). Analysis of these desiderata by the EHRI and EHR WG has identified numerous barriers that impact the ability to represent ClinGen and eMERGE information in the EHR environment. All of the identified barriers will impact the ability to fully use genomic information as a part of healthcare, and as such, no formal prioritization of impact was performed. There are certain dependencies that exist which were the subject of discussion to fully understand the relationships between the barriers. There was also recognition that some barriers could be overcome using existing platforms and resources to develop local solutions to inform more generalizable approaches, while other barriers would require changes to EHR systems, or international standards that were outside of the direct control of the working groups, although information obtained through trial implementation could be

<sup>1</sup>https://www.genome.gov/Funded-Programs-Projects/Electronic-Medical-Records-and-Genomics-Network-eMERGE.

<sup>2</sup>https://emerge.mc.vanderbilt.edu/the-emergeseq-platform/.

<sup>3</sup>https://emerge.mc.vanderbilt.edu/projects-2/ehr-integration/.

shared with these external entities to inform their development. The information that follows represents a qualitative but pragmatic synthesis of the barriers and potential solutions.

## Standards

Arguably the most important and foundational barrier encountered is the limited ability to transmit gene and variant information as standards-compliant, structured data. This is due to several limitations including: inadequate standards for representing core genomic information, such as gene and variant names and variant classification; lack of standards surrounding the naming and delineation of genetic disease; limited interfaces to access EHR data and external information; suboptimal user experience accessing external resources within the EHR; and lack of input from geneticists, clinicians, and informaticians into vendor design to develop improvements. These limitations have a downstream impact on the ability to provide clinician education within the EHR environment through clinical decision support (CDS) capabilities, including access to point-of-care, just-in-time information relevant for the care of the patient and the ability to integrate this information and associated knowledge within other clinical applications that are critical to clinician workflow. In light of these limitations, some incremental progress towards the goal has been achieved. One example is through the use of a standardsbased CDS capability available in the EHR, generally known as "infobuttons" (Del Fiol et al., 2012). Ancillary genomic systems that augment EHR functionality have also been used to provide needed functionality. Improvements in both EHR and ancillary genomic systems, combined with more robust data interfaces such as Fast Healthcare Interoperability Resources (FHIR) (Alterovitz et al., 2015), are providing opportunities for new approaches. These issues are summarized in **Table 1**, and each will be discussed in detail below (Herr et al., 2015; Tenenbaum et al., 2016).

Several standards are required to implement the accurate rendering and scalable delivery of information to the clinician


*eMERGE, Electronic Medical Records and Genomics Network; ClinGen, Clinical Genome Resource; XML, Extensible Markup Language; EHR, electronic health record; HL7, Health Level 7; FHIR, Fast Healthcare Interoperability Resources; GA4GH, Global Alliance for Genomic Health; EHRI, Electronic Health Record Integration; CSER, Clinical Sequencing Exploratory Research; SMART, Substitutable Medical Applications, Reusable Technologies; ACMG, American College of Medical Genetics and Genomics; CPIC, Clinical Pharmacogenetics Implementation Consortium.*

regarding genetic testing results. These involve the representation of patient genetic data; the representation of knowledge about genes, variants, and related phenotypes in a manner that can reflect knowledge updates; the robust definition of "genetic phenotypes"; the definition of interfaces to external knowledge resources; and the content and structure of information presented to the provider (**Table 1**).

#### Storage of Genomic Data

Using genomic data in clinical practice will challenge the storage and computing capacity of current EHR systems. The potential volume of an entire genomic sequence, as opposed to a smaller number of genotypes, is beyond the capacity of current EHR systems. One solution to this problem is the ancillary genomic system (Starren et al., 2013). Much like an imaging archiving system, an ancillary genomic system can offer federated storage solutions optimized for the heterogeneity and size of genomic data and results. For example, an institution could receive from different laboratories a file containing star alleles for pharmacogenetic test results, an Extensible Markup Language (XML) file containing identified variants as part of a custom panel, or even a Variant Call Format (VCF) file for more expanded sequencing data, for the same patient. These data range in size from bytes to kilobytes to megabytes, respectively, and require distinct indexing approaches for fast retrieval. To leverage these data, an ancillary genomic system can perform specialized processing and be linked to the EHR to provide synthesized deeper views into genomic test results and associated data. Three eMERGE sites have developed and implemented versions of a genomic ancillary system. A prototype ancillary genomic system to support pharmacogenomic testing and reporting was implemented at Northwestern University (Rasmussen et al., 2019). Similarly, Mayo Clinic developed a genomic data warehouse (Horton et al., 2017). Partners HealthCare created a distributed system focused on managing indication-specific genetic testing (Aronson et al., 2012). However, open specifications for broadly targeted versions of such systems remain underdefined, and no open source solutions are currently available, although a few commercial systems have been developed to support pharmacogenomic data and single-gene or panel genetic testing. Ancillary genomic systems will be referenced in subsequent sections, emphasizing a key role in supporting the use of genomic information. Of note, EHR vendors are rapidly moving to cloud solutions to increase storage and accessibility of data while preserving EHR performance characteristics. These solutions have not yet been applied to genomic data, and there is concern that EHR vendors don't understand the complexity of genomic data and haven't been able to capture discrete results at the level of detail that is required in clinical care.

#### Representation of Patient Genetic Data

Perhaps the most fundamental gap in all EHR implementations is the lack of a standardized, structured format for genetic data. Most of the data regarding genomic variants exists in the EHR as a scanned document stored in portable document format (PDF) (Shirts et al., 2015). Information in this form is static and does not provide an electronic point of reference to launch clinical information resources. Further, the naming conventions vary within and across institutions, so that tracking and monitoring results is difficult. To overcome the limited functionality of static documents, several healthcare systems have manually entered these results into data fields in the EHR, such as listing pharmacogenomic phenotypes as allergies or genetic findings as items on the problem list (Ohno-Machado et al., 2018). While these solutions are far from ideal, they do allow for CDS to be executed based on this information. However, as genomic results increase in number and complexity, *ad hoc* workarounds such as these become untenable due to the increasing amount of resources needed to maintain them and the risk for error inherent in any manual process. Up until this point, ancillary genomic systems, connected to the EHR, have been required to implement knowledge update–driven CDS (Aronson et al., 2012).

In addition to unstructured PDF reports, genomic test results also can be added to the EHR using Health Level 7 (HL7) Version 2 (v2) messages, which are widely supported across many clinical systems. The HL7 Clinical Genomics working group published an Implementation Guide to support the exchange of genomic data using v2 messaging and the Logical Observation Identifiers Names and Codes (LOINC) code system4 . This approach enables the genomic results to be entered as structured data, which facilitates its use as part of a CDS system, but due to the message structure and the content available in LOINC, v2 messages are limited in their ability to render highly discrete genomic data components with semantic precision.

An emerging standard that has the potential to address some of these issues is the HL7 FHIR standard5 . FHIR builds on prior HL7 standards but takes advantage of widely used web services technology, which facilitates implementer adoption. The HL7 v2 Clinical Genomic Report data structure does not map directly to the structures in the FHIR Genomics Reporting Implementation Guide. Harmonization of these two standards followed by implementation in laboratory information systems could accelerate the communication of genomic results between labs and clinics. A subgroup within HL7 Clinical Genomics is developing an information model for the clinical genomics domain, which is intended to provide common semantics for clinical genomics and serve as a harmonization point for genomic standards. Representatives from eMERGE and ClinGen are involved in this process and actively share lessons learned from site-specific implementation efforts.

In our experience through eMERGE, both the HL7 Clinical Genomic Report and the FHIR genomics standards have significant gaps, which hinders the adoption of these standards for clinical use. Given the heterogeneity of how genomic information is documented in the EHR, in preparation to establish a consensus format in eMERGE, the EHRI workgroup co-chairs designed and conducted informal interviews of eMERGE and Clinical Sequencing Exploratory Research (CSER) sites6 (Shirts et al., 2015). The goal of these interviews was to understand the intended use of genomic test reports and their requirements for

<sup>4</sup>https://www.hl7.org/implement/standards/product\_brief.cfm?product\_id=23. 5https://www.hl7.org/implement/standards/product\_brief.cfm?product\_id=343. 6https://cser-consortium.org/.

transferring the reports and associated data from the laboratory to the sites. In summary, we found that sites wanted the reports in both a PDF and structured format, as well as the complete raw data files. Regarding transfer, secure file transfer protocol (SFTP) was available and acceptable to all sites; however, the ability to use a web service for transfer was not available at all sites. On the basis of these findings, the eMERGE Network created a consensus interface format to enable interorganizational transmission of genetic test results and genetic knowledge updates (available on GitHub)7 . There is now an active effort within eMERGE to convert its existing XML format into a network-specific profile for genomic data.

Concerns exist about the risk of a privacy breach or discrimination based on the presence of genomic data in the EHR. This was the subject of a review article led by the EHRI and Ethical, Legal, and Social Issues working groups of eMERGE (Hazin et al., 2013). To date, there is limited evidence that this represents a significant problem, and additional protections specific to genetic data exist at the state and national level to protect against the inappropriate use of this information by health insurers and employers. As noted above, genetic information already is present in the EHR, albeit in a form less amenable to discovery. The incremental risk of providing the information in a more accessible form is offset by the improved ability of clinicians to use the information to improve patient care and outcomes. Therefore, this was not identified as a priority barrier by the respective working groups.

#### Translating Variant Knowledge Into Genetic Phenotypes

Assuming that genomic data can enter the EHR in a consistent, adequately structured electronic format, in order for the data to be used, it must be combined with standardized, computable genomic knowledge, which might exist at the variant, gene, and ultimately "genetic phenotype" levels. The genetic phenotype is a concept linking variant and gene knowledge to a defined patient characteristic or disease whose risk is associated with genetic variant(s) for which information can be delivered to clinicians (**Figure 1**). An example of such a phenotype is a patient with a pathogenic variant in the gene *BRCA1* [Online Mendelian Inheritance in Man (OMIM) gene #113705]8 , which is associated with increased risk of developing breast, ovarian, and prostate cancers. The genetic phenotype associated with this pathogenic *BRCA1* variant is most commonly called "hereditary breast and ovarian cancer syndrome." Such characteristics (such as the *BRCA1*-associated cancers) do not need to be present in the patient as the phenotype may consist of a risk or predisposition, as shown in **Table 2**. Precisely defining these genetic phenotypes requires more research. eMERGE and some ancillary genomic systems model the linkage between variants and diseases or pharmacogenomic effects. However, a more robust model that incorporates gene-level knowledge and relevant associated information (termed knowledge artifacts) is needed. A standard

for these phenotypes is a necessary prerequisite as it serves as the launching point of genomic information resources. An early example of this is the Monarch initiative9 that is categorizing phenotypes from humans and other species to support discovery10. While not intended as a clinical resource, ClinGen has begun to incorporate some of the Monarch knowledge to support gene and variant annotation that ultimately yields information of relevance to clinicians.

The granularity with which we standardize these genetic phenotypes and how that defines the focus of the information delivered is an important consideration. Precision medicine dictates that management is driven by a patient's genetic variant results coupled with other relevant data. However, with millions of possible variants influencing human health and disease, the maintenance of information delivery at the variant level becomes a daunting task, as most diseases are driven by one of hundreds or thousands of different pathogenic variants in a gene. For example, *BRCA1* has more than 2,969 pathogenic variants asserted in the Clinical Variant Resource11 (ClinVar) as of May 2019. Genetic variant classification is done by laboratories as part of the result reporting process (Richards et al., 2015). The knowledge generated through this process can be captured in structured form. It can then be transmitted to the EHR ecosystem as structured results and, if necessary, revised as structured general "knowledge updates" when more is learned about a particular variant (Aronson et al., 2012). The eMERGE interface format supports transmission of knowledge updates related to these linkages. The ancillary genomic system approach has also

<sup>7</sup>https://github.com/emerge-ehri/results-schema.

<sup>8</sup>https://omim.org/entry/113705?search=113705&highlight=113705.

<sup>9</sup>https://monarchinitiative.org/.

<sup>10</sup>https://monarchinitiative.org/page/about.

<sup>11</sup>https://www.ncbi.nlm.nih.gov/clinvar/.


been used clinically to manage these types of knowledge updates. Ideally, once this information reaches the EHR, it would then be combined with other genetic and non-genetic knowledge to determine patient genetic phenotypes. This last step is currently underdeveloped within EHR ecosystems.

The complexity of genetic disease underscores the importance of having a genetic phenotype as a point of decision making and information delivery in the EHR. There are cases such as with alpha-1-antitrypsin deficiency for which specific variants are associated with variable severity, and environmental factors such as smoking dramatically alter the risk of developing chronic obstructive pulmonary disease and, by necessity, alter the recommended care (Al Ashry and Strange, 2017). It is unrealistic to expect that clinicians will wade through pages of documentation to discover the specific risks associated with that variant. Thus, having the most pertinent information delivered according to the relevant combination of variants and clinical variables is a key goal of CDS. This problem will increase exponentially as we apply genetic variation and non-genetic modifiers to each patient. More effort to increase the granularity of genetic phenotypes may save substantial time and effort on the part of the clinician in the long run, as well as provide better care.

The lack of standardized terminologies for genetic phenotypes for use in result reporting can lead to clinician confusion, while also impacting interoperability and implementation of CDS. Consider the genetic phenotype "hereditary breast and ovarian cancer syndrome." While this term is in common use, the lack of a standard terminology could result in one lab reporting the genetic phenotype as "*BRCA1*- and *BRCA2* associated hereditary breast cancer," while another may report it as "breast–ovarian cancer, familial 1." In the former case, a clinician unfamiliar with the gene–disease association may only provide information about breast cancer, which is not consistent with evidence-based recommendations. This was a significant issue in pharmacogenomics for which use of different terms (extensive metabolizer, normal metabolizer) for the same pharmacogenetically defined phenotype led to confusion (Caudle et al., 2017). Assignment of these variants to the correct phenotype is critical, as the phenotype is the data element to which all information resources are mapped, and it is a key criterion for CDS interventions. Without standardization, the healthcare system must resort to either manual assignment of the phenotype or mapping of phenotypes for each laboratory they use and for every condition for which the laboratory tests. In recognition of this issue, the Clinical Pharmacogenetics Implementation Consortium (CPIC) led an effort to harmonize terms for reporting that incorporated the input of non-specialist clinicians to develop a standard terminology for reporting that is consistent and unambiguous, thus enhancing clinician understanding. A related effort to harmonize terms describing phenotypes and outcomes involving the eMERGE Outcomes working group, and the ClinGen Actionability Working Group (Williams et al., 2018) provides a basis for work by informaticists to create terminology standards to enhance interoperability. ClinVar and ClinGen as public repositories could play a decisive role in managing the known associations between variants and genes, and the resulting genetic phenotypes.

#### Clinical Decision Support for Clinical Genomics

It is not logical nor feasible for EHR vendors and most healthcare systems to create and maintain large-scale genomic knowledge resources for clinicians. This reality necessitates the ability of the EHR to access external knowledge content and CDS capabilities, ideally through scalable standards-based approaches as proposed by Welch et al. (2014) and Shellum et al. (2016). Our previous summary of opportunities for genomic CDS illustrates that there is much we can learn from implementing CDS in the pre-genomic era (Overby et al., 2013). CDS can be organized into three general categories: passive, asynchronous (or semi-active), and active (Lobach et al., 2012). Passive CDS provides just-in-time access to information resources triggered by the clinician when a clinical question is raised. Asynchronous CDS presents aggregated information to a clinician to support patient-specific care reassessments based on new knowledge, or as part of quality improvement and care initiatives for a group of patients outside of an individual patient encounter. Based on EHR user events (e.g., chart opening, medication prescription, laboratory results review), active CDS provides information to clinicians in real time at the point of care specific to the patient encounter anticipating that clinicians will not always be aware that information is needed to make a clinical decision.

Several CDS modalities, such as alerts and reminders (active or asynchronous CDS), population health management dashboards (asynchronous CDS), infobuttons (passive CDS), and integrated information displays (active, asynchronous, or passive) can be used to help providers integrate clinical genomics into routine patient care decisions. For example, *alerts* can prompt providers when a patient may benefit from a certain pharmacogenomic test or when the result of a test warrants changes in the patient's medication or management (active CDS) (Herr et al., 2019). *Reminders* (active or asynchronous CDS) serve as a checklist to help providers follow various evidence-based preventive measures, including cancer screening approaches (such as an accelerated schedule for routine colonoscopies in a patient with a genetic predisposition to developing colorectal cancer) that are personalized based on clinical genomics (Aronson et al., 2012). Patient-specific knowledge alerts (asynchronous CDS) can alert clinicians outside of an encounter when new information emerges on a variant previously identified in a patient. *Population health management* (asynchronous CDS) uses a different approach, whereby patient records are automatically scanned to identify and aggregate those who meet criteria for certain genetic evaluation or care based on a previously reported genetic result (Kohlmann et al., 2019). *Infobuttons* (passive CDS) provide justin-time access to external knowledge resources accessible by but not necessarily contained within the EHR. Based on the context of the interaction between the provider and the EHR, infobuttons (Cook et al., 2017) are found next to items in different sections of the EHR, such as problem list, medications, orders, and laboratory test results. Infobuttons have been a key strategy to present genetic information to clinicians as part of both the eMERGE and ClinGen and will be discussed below (Overby et al., 2014; Heale et al., 2016; Crump et al., 2018). Complementary technologies for passive CDS are being developed that enable the delivery of genomic results *via* mobile devices (Samwald and Freimuth, 2013). *Integrated information displays* provide intelligent visualization of patient data integrating multiple sources within and outside the EHR and can be used to display genetic data along with other relevant clinical data in the EHR.

While critical to help providers integrate clinical genomics in routine patient care, several challenges limit the implementation and adoption of CDS for clinical genomics. Overall, any basic CDS requires access to EHR data in a standard, structured, and computable format. However, as mentioned above, the absence of relevant vocabulary and messaging standards is a critical barrier. Even where these exist, there is low adoption of standard vocabularies for genetic tests and standards for the representation of genetic test reports in a computable format. Similarly, although standard representations of CDS logic have existed for decades (most notably Arden syntax (Hripcsak et al., 2018)), these have not seen widespread adoption in commercial EHRs. This means that institutions wishing to disseminate successful CDS implementations need to do so either using what the CDS Consortium (Middleton 2009) has termed Level 1 artifacts (Hongsermeier et al., 2011)—that is, narrative descriptions of CDS logic—or by distributing entire applications that implement the CDS, few of which exist for genomics. In an attempt to capture Level 1 artifacts, the eMERGE Network in conjunction with the NHGRI-funded Implementing Genomics in Practice (IGNITE) consortium developed the CDS Knowledge Base12 (CDSKB), which includes a dedicated library for the dissemination of genomic CDS. While primarily populated with Level 1 and Level 2 (flowcharts or wire frame) artifacts, as shown in **Figure 2**, it is capable of storing computable definitions (Levels 3 and 4), renderings of which have been explored for pharmacogenomics (Linan et al., 2015). To better understand the complexities of genomic CDS, eMERGE Network sites examined issues related to CDS implementation using pharmacogenomics as the use case (Herr et al., 2015). However, given the complexity of CDS logic in clinical genomics, it would be desirable for EHR systems to defer clinical genomic CDS to external web services. While not universal, the rapid adoption of emerging CDS standards such as CDS Hooks13, which takes advantage of Application Programming Interfaces (APIs) in the EHR, has the potential to enable a cloud-based ecosystem for clinical genomics as demonstrated in recent prototype work in pharmacogenomics (Dolin et al., 2018).

#### Infobuttons

Infobuttons are particularly appealing for clinical genomics because they are required for EHR certification in the United States Meaningful Use program (Federal Register, 2012); leverage external genomic resources; and can provide just-in-time access to relevant and up-to-date clinical genomics information under the clinician's control. This approach, which mapped infobuttons to existing publicly available resources, was successfully implemented in a non-commercial EHR system in 2006 (Del Fiol et al., 2006), well before the HL7 Infobutton Standard was developed. Once included in EHR certification, both eMERGE and ClinGen have studied the use of infobuttons for delivery of genomic knowledge at the point of care, as there are no other generalizable solutions in commercially available EHR systems.

There are barriers besides those discussed above that hinder the implementation of infobuttons for clinical genomics. First, with the exception of the Pharmacogenetic Knowledgebase 14(PharmGKB), clinical genomic resources are not compliant with the HL7 Infobutton Standard (Del Fiol et al., 2012; Strasberg et al., 2013), which is the mechanism used by EHR systems to communicate with external knowledge resources (Heale et al., 2016). Second, not all clinical genomic resources provide access to actionable recommendations in a format that can be readily accessed at the point of care. Last, EHR systems are unable to distinguish the context in which a clinical genomics resource might be useful, requiring the use of external web services such as OpenInfobutton (Del Fiol et al., 2013). The eMERGE and ClinGen EHR working groups have been working cooperatively to overcome these barriers.

<sup>12</sup>https://cdskb.org/.

<sup>13</sup>https://cds-hooks.org/.

<sup>14</sup>https://www.pharmgkb.org/.

registration is required.

adverse event related to the exposure. This reduces disruption of the clinician workflow. This artifact and many other examples are available at CDSKB.org. Free

Efforts are underway through the eMERGE Network and ClinGen to develop infobutton-compliant genomic resources to deliver targeted information to patients and providers (Overby et al., 2014). A survey of eMERGE and CSER consortia sites identified that existing resources contain the content that an institution would like to present at the point of care but may require some additional synthesis (selecting particular sections or paragraphs), localization (providing institution-specific information such as the contact information for genetic counselor referrals), and branding (institution logos for patient handouts) (Rasmussen et al., 2016). In addition, the adoption of a structured template would also benefit content authors to ensure that resources sufficiently answer anticipated questions for genomic medicine (Overby et al., 2014). More recently, the eMERGE Network has led the development of a tool called DocUBuild15,16, which is a freely accessible and open source platform to create information resources to support genomic medicine. DocUBuild supports features such as templating, content sharing, and localization (with linked provenance), as well as branding. While still in its infancy, DocUBuild is providing a testing ground to evaluate how genomic resources may be better optimized for patients and providers.

eMERGE and ClinGen are also working collaboratively with the ACMG on the ACT sheets (ACMG, 2001)17. ACT sheets were

17 https://www.ncbi.nlm.nih.gov/books/NBK55832/

initially developed to support clinician information needs related to newborn screening programs. They were designed to be used as point-of-care educational documents that provide clinicians with sufficient knowledge about a rare genetic condition they had likely not encountered previously and included recommendations on care needed to optimize patient outcomes. They were designed to include both a narrative summary (L1) and decision tree (L2) CDS artifacts. As genetic and genomic indications expanded, the content of the ACT sheets has extended to cover more indications. In particular, ACT sheets are under development to support the care of patients receiving a result from the ACMG secondary findings list (Kalia et al., 2017). The goal of this collaboration is to use these ACT sheets to develop computable CDS that can be distributed through EHR systems, lowering the burden of implementation for systems implementing genomic information into clinical care.

# Integrated Information Displays *Via* EHR Apps

An increasingly popular approach to integrating CDS capabilities into EHR systems is the Substitutable Medical Applications, Reusable Technologies (SMART) coupled with FHIR (SMART on FHIR) (Mandel et al., 2016). SMART enables applications to be integrated for interoperability across different EHR vendors, including single sign-on, end point for users to launch an app from within the EHR, and exchange of security token for apps to access the EHR's FHIR server. Examples of SMART on FHIR apps with integrated information displays for clinical genomics are available

<sup>15</sup>https://docubuild.fsm.northwestern.edu/

<sup>16</sup>https://www.genomeweb.com/informatics/researchers-develop-web-appimprove-curation-delivery-genomic-knowledge-point-care

(Alterovitz et al., 2015; Warner et al., 2016). In addition to including general patient genetic test result management functionality, apps such as these could be used to provide deep disease-specific functionality that combines genomics with other forms of relevant clinical data. Other solutions are also being explored. Partners HealthCare implemented an EHR integrated app, before the advent of the SMART on FHIR standard, to manage genetic results and associated knowledge (Aronson et al., 2012).

Building, clinically validating, integrating, and distributing these apps is resource intensive. In part, this is due to the complex nature of genomic data and the knowledge required to process the results into clinically actionable interpretations. Although many resources exist that contain this knowledge (e.g., CPIC Guidelines18, ACMG ACT sheets17), not all are currently available in a computable form. This is an additional challenge to the ones listed above regarding the representation of such knowledge. There are other issues with clinician adherence to guidelines that are not specific to genomics but must be recognized if guideline-based care is to be realized. Examples include inclusion of language that is not adequately explicit and therefore difficult to compute (e.g. "might consider" or "1 to 2 years"); the source of the guideline; differences in clinical workflow; clinician knowledge; and differences in management approaches by different specialties, among others (Cabana et al., 1999).

Having the data represented in a computable form will allow developers to more easily integrate these sources of information, reducing development time and duplication of the knowledge bases, as well as facilitating more rapid updates as knowledge changes. In addition, standards such as SMART and FHIR are not implemented equally across all EHR vendors and even across instances of the same vendor's EHR. As these standards continue to see adoption and maturation, ongoing validation and communication with vendors is needed to ensure that the implementations are delivering on the promise of the technology.

# Access to Genomic Knowledge

ClinGen's website, www.clinicalgenome.org, was established to support ClinGen's mission to "provide high quality, curated information on clinically relevant genes and variants" (Rehm et al., 2015) in a centralized way to the public. ClinGen's website was launched in 2014, and over the last 5 years, the website has undergone many improvements to enhance the ability to connect curations to the genomics community and the EHR.

In 2015, ClinGen provided access to ClinGen's curations and external genomic resources by releasing an infobutton-enabled search interface built into a section of the website. This update enabled ClinGen's website to utilize the HL7 Infobutton Standard (Del Fiol et al., 2012) to allow visitors to query a term related to other standard nomenclatures [OMIM, Human Genome Organization (HUGO) Human Gene Nomenclature Committee (HGNC), RxNorm] and have information from a variety of genomics resources to be presented to the user through the use of web standards and external links to resources.

Throughout 2016 and 2017, ClinGen improved the ability to query terms (OMIM, Orphanet, HGNC, RxNorm) and moved the search feature to ClinGen's home page. At this time, updates were made to allow ClinGen's website to support basic HL7 Infobutton-compliant requests and display curation knowledge generated by ClinGen's curation groups. In 2018 and 2019, ClinGen continued to make improvements by including support for multiple disease resources through the use of the Monarch Disease Ontology (MONDO), allowing ClinGen's curations to be directly published to the website from the curation interfaces after approval, and by investing resources to expand the depth of the curation knowledge available to the public.

As of June 3, 2019, ClinGen's website provided curated information on 747 Gene–Disease Clinical Validity Summary Curations, 102 Clinical Actionability Curations, and 1,475 Dosage Sensitivity Curations. ClinGen's Evidence Repository provides information about 684 Variant Pathogenicity Curations.

We have learned that the technical process for a website to implement basic support to become HL7 Infobutton compliant is straightforward and relatively easy to get started. The process to go further by providing a web resource that fully utilizes HL7 Infobutton and/or supports SMART on FHIR requires a commitment of resources and assessment to understand specific use cases. Resources should consider how their tools may be adopted and utilized within the EHR. This is an endeavor that each resource should undertake wisely, and resources should consider conducting usability studies to assess the user experience of the resource within the EHR.

Over the last 4 years, we have successfully been able to display ClinGen's curations and provide access to external genomics resources through the use of OpenInfobutton (Heale et al., 2016) by making our resource HL7 Infobutton compliant. We are continually working to improve the resource and information we offer, explain how genomic resources can become infobutton compliant, and promote the infobutton adoption in EHR platforms. Recognizing that infobuttons are not routinely "turned on" in most healthcare organizations, the ClinGen EHR WG has developed an implementation guide specific for OpenInfobutton access to the ClinGen resource that is freely available19.

# DISCUSSION

Integration of structured genomic information into the EHR to support patient care remains limited. Ongoing work at the national and international level is targeting the barriers described above. The HL7 FHIR specification is under active, collaborative development by a wide variety of stakeholders, including national initiatives. In particular, the Office of the National Coordinator's (ONC's) Sync for Genes20 precision medicine research program recently sponsored the pilot implementation of the FHIR Genomics specification, which will be used by the *All of Us* Precision Medicine Initiative. In another international effort to develop standards for genomics, the Global Alliance for Genomic Health (GA4GH)21 is

<sup>18</sup>https://cpicpgx.org/guidelines/

<sup>19</sup>http://www.openinfobutton.org/documentation

<sup>20</sup>https://www.healthit.gov/sites/default/files/sync\_for\_genes\_report\_ november\_2017.pdf

<sup>21</sup>https://www.ga4gh.org/

developing a suite of tools and specifications that enable genomic data sharing. The GA4GH is informed by FHIR but does not utilize FHIR. Representatives of eMERGE and ClinGen are working with HL7 and GA4GH leadership to keep the two projects aligned to reduce the risk of development of different standards that are incompatible. The standards developed by HL7 and GA4GH will require substantive changes in as well as enhancement to the currently available vendor-based EHR, laboratory, and ancillary genomic systems to achieve full integration. Projects focused on implementation of genomics in clinical care, such as eMERGE and ClinGen, provide a valuable test bed for the development, testing, optimization, and dissemination of best practices.

To date, most of the research has been focused on feasibility with relatively limited network-wide implementation. Future efforts must focus on the end user to measure the effectiveness of these modalities for education and support of clinicians and patients, and ultimately on the impact of genomic medicine. An early example of this is focused on the implementation of pharmacogenomics in eMERGE Phase 2 (Rohrer Vitek et al., 2017). The 10 sites implementing pharmacogenomics catalogued their strategies for clinician education. While not focused on the effectiveness of the educational interventions, this survey collects a broad range of approaches providing the basis for comparative testing of the effectiveness of the strategies. Another working group of ClinGen, Consent & Disclosure Recommendations (CADRe), is beginning to study this issue. CADRe has developed recommendations regarding consent and results disclosure for genomics focused on clinicians without training in genetics (Ormond et al., 2019). They are now working to develop educational materials to support the integration of CADRe recommendations into practice at the point of care. CADRe has initiated engagement with clinicians to guide development of the educational strategies, which will ultimately be included as part of the ClinGen resource.

Representatives of eMERGE and ClinGen are actively participating in various international standards development efforts, including those in HL7 (FHIR genomics) and GA4GH. The practical experience from early genomic medicine implementation efforts is critical to test the usefulness of existing and proposed standards. An example of this is the selection of ClinGen as a driver project for the GA4GH. The specific project is focused on the development of standards for data sharing (Dolman et al., 2018). These collaborations will accelerate the development and testing of standards necessary to overcome the barriers identified above.

One other consideration is the sustainability of the current efforts. eMERGE and ClinGen are funded research projects, raising the question of how such efforts can be sustained over time. This is particularly critical for the ClinGen resource, which is increasingly viewed as a foundational genomic knowledge resource essential for the clinical use of genomic information. Transition of the resource from a research project to some other sustainable model is essential, and discussion of alternative models has begun. Recognition of the value of the resource by a diverse set of stakeholders is essential to ensure investment and innovation to support sustainability.

In conclusion, eMERGE and ClinGen in conjunction with many other efforts in the US and internationally are working

to develop educational approaches within the EHR to support clinicians to integrate genomic information in clinical care. While much work remains, the lessons learned from these projects have provided rich information that can be used to advance the field. Efforts to engage with clinicians as end users to understand preferences and measure effectiveness are needed.

#### EXECUTIVE SUMMARY

	- Lack of standards to represent and communicate genomic information.
	- Inability to store genomic information in current EHR systems.
	- Translating genomic variants into clinical phenotypes that clinicians can recognize and use to manage patients.
	- Access to reliable genomic knowledge sources.
	- Existing efforts are largely supported by institutional and grant funding. Sustainable models are needed for further development.
	- Clinical decision support systems that can be modified to support some genomic medicine interventions.
	- Infobuttons (context-sensitive information retrieval tools) linked to genomic information resources.

# DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

# AUTHOR CONTRIBUTIONS

Conception of manuscript: MW. Conduct of research: MW, NW, SG, RF, LR, CT, SA, GF, EH, and GW. Writing: MW, NW, SG, RF, LR, CT, SA, and GF. Critical review of manuscript: MW, NW, SG, RF, LR, CT, SA, GF, CW, WC, GJ, JN, GW, EH, AF, CP, and RR.

#### FUNDING

Brigham and Women's Hospital: U01HG8685 (eMERGE); Cincinnati Children's Hospital Medical Center: U01HG8666 (eMERGE); Columbia University: U01HG008680 (eMERGE); Geisinger Clinic: U01HG8679 (eMERGE) and U41HG009650 (ClinGen); Group Health Cooperative/University of Washington: U01HG8657 (eMERGE); Mayo Clinic: U01HG063798 (eMERGE) and U41HG068345 (ClinGen); Northwestern University:

# REFERENCES


U01HG8673 (eMERGE); Partners HealthCare/Broad Institute: U01HG8676 (eMERGE); Vanderbilt University Medical Center: U01HG8672 (eMERGE).

# ACKNOWLEDGMENTS

The authors wish to recognize the contributions of other members of the eMERGE network who have contributed to work reported in this paper. Dr. Hakon Hakonarson (Children's Hospital of Philadelphia), Sheethal Jose (National Human Genome Research Institute), Dr. Josh Peterson and Dr. Ellen Wright Clayton (Vanderbilt University).


using structured knowledge representation. *AMIA Annu. Symp. Proc.* 2015, 1985–1994.


integration of pharmacogenomics in practice: the eMERGE Network experience. *Pharmacogenomics.* 18, 1013–1025. doi: 10.2217/pgs-2017-0038


**Conflict of Interest:** SA: Partners HealthCare receives royalties from sales of GeneInsight software. Group has received funding from Persistent Systems and Novartis to construct SMART on FHIR applications and infrastructure. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer YB declared a past co-authorship with one of the authors GPJ to the handling editor.

*Copyright © 2019 Williams, Taylor, Walton, Goehringer, Aronson, Freimuth, Rasmussen, Hall, Prows, Chung, Fedotov, Nestor, Weng, Rowley, Wiesner, Jarvik and Del Fiol. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

12

**35**

# Ensuring Best Practice in Genomic Education and Evaluation: A Program Logic Approach

*Amy Nisselle1,2,3\*, Melissa Martyn2,3,4, Helen Jordan5, Nadia Kaunein2,5, Alison McEwen6, Chirag Patel7, Bronwyn Terrill1,8,9, Michelle Bishop10, Sylvia Metcalfe1,2,3 and Clara Gaff1,2,3*

*1 Australian Genomics Health Alliance, Melbourne, VIC, Australia, 2 Genomics in Society, Murdoch Children's Research Institute, Melbourne, VIC, Australia, 3 Department of Paediatrics, The University of Melbourne, Melbourne, VIC, Australia, 4 Melbourne Genomics Health Alliance, Melbourne, VIC, Australia, 5 Melbourne School of Population and Global Health, The University of Melbourne, Melbourne, VIC, Australia, 6 Graduate School of Health, University of Technology Sydney, Sydney, NSW, Australia, 7 Genetic Health Queensland, Royal Brisbane and Women's Hospital, Brisbane, QLD, Australia, 8 Kinghorn Centre for Clinical Genomics, Garvan Institute of Medical Research, Sydney, NSW, Australia, 9 St Vincent's Clinical School, University of New South Wales Sydney, Sydney, NSW, Australia, 10 Genomics Education Program, Health Education England, Birmingham, United Kingdom*

#### *Edited by:*

*Go Yoshizawa, OsloMet – Oslo Metropolitan University, Norway*

#### *Reviewed by:*

*Karen Hanson, ICF, United States Chih-hsing Ho, Academia Sinica, Taiwan*

*\*Correspondence: Amy Nisselle amy.nisselle@mcri.edu.au*

#### *Specialty section:*

*This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics*

*Received: 01 August 2019 Accepted: 02 October 2019 Published: 08 November 2019*

#### *Citation:*

*Nisselle A, Martyn M, Jordan H, Kaunein N, McEwen A, Patel C, Terrill B, Bishop M, Metcalfe S and Gaff C (2019) Ensuring Best Practice in Genomic Education and Evaluation: A Program Logic Approach. Front. Genet. 10:1057. doi: 10.3389/fgene.2019.01057*

Targeted genomic education and training of professionals have been identified as core components of strategies and implementation plans for the use of genomics in health care systems. Education needs to be effective and support the sustained and appropriate use of genomics in health care. Evaluation of education programs to identify effectiveness is challenging. Furthermore, those responsible for development and delivery are not necessarily trained in education and/or evaluation. Program logic models have been used to support the development and evaluation of education programs by articulating a logical explanation as to how a program intends to produce the desired outcomes. These are highly relevant to genomic education programs, but do not appear to have been widely used to date. To assist those developing and evaluating genomic education programs, and as a first step towards enabling identification of effective genomic education approaches, we developed a consensus program logic model for genomic education. We drew on existing literature and a co-design process with 24 international genomic education and evaluation experts to develop the model. The general applicability of the model to the development of programs was tested by program convenors across four diverse settings. Conveners reported on the utility and relevance of the logic model across development, delivery and evaluation. As a whole, their feedback suggests that the model is flexible and adaptive across university award programs, competency development and continuing professional development activities. We discuss this program logic model as a potential best practice mechanism for developing genomic education, and to support development of an evaluation framework and consistent standards to evaluate and report genomic education program outcomes and impacts.

Keywords: workforce, genomic medicine, program logic, theory, education, evaluation

# INTRODUCTION

Genomic medicine is rapidly being incorporated into routine healthcare (Manolio et al., 2015) and, due to advances in technology, demand will grow as the time and cost of genetic/ genomic testing reduce (Stark et al., 2019b). There are longstanding concerns and evidence that health professionals not trained in genetics or genomics have rudimentary knowledge of these disciplines, and are neither equipped nor confident to adopt new genomic technologies into clinical care (Fuller et al., 2001; Carroll et al., 2011; Feero and Green, 2011; Houwink et al., 2011; Korf et al., 2014; Slade and Burton, 2016). The need for quality educational programs, activities, and resources (collectively referred to here as 'education interventions') to improve the knowledge of health professionals who are not trained in genomics is critical to the successful integration of genomics into routine healthcare (Carroll et al., 2011; Wildin et al., 2017).

We undertook a review of genomic education produced in Australia in 2016–17 (Janinski et al., 2018; McClaren et al., 2018) and found numerous genomic education interventions are developed and implemented across diverse contexts (for example, formal education or training versus continuing education), often in response to local healthcare system needs perceived by the educator. Interviews with program convenors (n = 32) revealed many interventions lacked clear learning objectives or evidencebased teaching and learning practices, and few convenors reported using needs assessments to inform programs or conducting evaluations of outcomes or development processes. Of the program convenors interviewed, only 13% had a tertiary qualification in education.

Program funders and stakeholders require evidence that education interventions have successfully met tangible outcomes (Gaff et al., 2007). If the pathway to achieving desired results is not clearly outlined prior to the implementation of an education intervention, it is difficult to deduce why, and how, the intervention produced the outcomes it did. Logic models delineate the key inputs, activities, and intended outcomes of programs. If presented with sufficient detail, program logic models can help to articulate a logical explanation as to how a program intends to produce the desired outcomes—its mechanism of action. Logic models can be used to describe whole programs, or parts of a program. For example, Horowitz and colleagues recently proposed the Genomic Medicine Integrative Research Framework as a "whole of system" logic model encompassing context, interventions, processes and outcomes to support those implementing genomic medicine (Horowitz et al., 2019), with genomic education defined as one type of intervention in their conceptual framework. Logic models not only provide an understanding of the reasoning underpinning a program, they can aid the planning of its evaluation (Horowitz et al., 2019).

Despite a need, there is little clarity on what defines quality genomic education interventions or successful strategies, and in which contexts (Wildin et al., 2017). Nor are there evidential standards around evaluating outcomes or reporting programs (Talwar et al., 2017). To begin to address this deficit, part of the Workforce & Education research program of the Australian Genomics Health Alliance (Australian Genomics; Stark et al., 2019a) aims to provide an evidence base for those developing genomic education (**Figure 1**). These include: 1) a program logic model to support design and development; 2) a framework for evaluation

spanning the education lifecycle; and 3) a minimum dataset to report program design, development, delivery and evaluation.

Here we describe the consultative process of engaging education and evaluation experts in developing and testing a program logic model for genomic education. We also illustrate the logic model's utility and flexibility across a variety of settings and contexts through narrative cases.

# METHODS

### Context

The scope of "education" considered for this program logic is education for any professional, with or without specialized genetic training, regarding the application of genomic medicine. This spans clinical and laboratory professionals and, depending on the local context, clinicians may be primary, secondary or tertiary healthcare providers. For example, they may be family physicians/general practitioners who refer patients to genetic services or hospital-based medical specialists/physicians who refer patients or directly order genomic tests. Here we use the term 'genetic specialists' to denote people with specialized genetic training (clinical and/or laboratory) and 'medical specialists' including primary care physicians (PCPs)—as medically qualified individuals specialized in a sub-discipline other than genetics, who may refer or order genomic tests.

### Developing the Logic Model

A Working Group (SM, CG, AN, MM, and HJ) developed a draft program logic model from June through to December 2017. This was based on theories of program logic, evaluation, and adult learning principles and drew on the collective knowledge and experience in developing and applying program logic models to genetic education interventions and research.

The draft program logic was reviewed and refined in a 2-day co-design workshop involving 24 Australian and international genetic education and evaluation experts (see Acknowledgements) held in February 2018. All attendees were experienced in developing genetic or genomic education interventions, program evaluation and/or implementation science. The workshop included didactic, self-directed, and group activities to: develop a shared understanding of program logic structure and language; discuss the application of program logic and the associated evaluation framework to genetic and genomic education interventions; and review and refine the draft model. The workshop also considered the process for testing the logic model.

# Testing the Logic Model

To test the connections between the key elements of the logic model, clarify the intended outcomes and test feasibility, we applied a clarificative evaluation approach using authentic case studies (Owen and Rogers, 1999). A sample of workshop participants tested the draft model in local contexts, both Australian and in the UK. The model was subsequently applied to three genomic education interventions in the conception, planning or development stage, and retrospectively to a recently completed intervention. A template was used to capture data relevant to the development, delivery, and evaluation of the education intervention in each setting. The dataset included personal educator and institutional characteristics; the description of the intervention, including components of the logic model relevant to each setting, and applicability and usefulness of the model; evaluations planned and/or undertaken (type, evaluation questions, study design, findings, etc.); and documentation collected (collaboration agreements, project plans, meeting minutes, etc.). Draft narratives were verified by the participants and quotes were extracted from the dataset, email correspondence or notes made during conversations. These four participants also provided feedback on relevance and utility of the model to their setting.

# RESULTS

#### Overview of the Program Logic Model

The logic model developed and refined by the working group and workshop participants captures four key components of the program cycle—planning, development, delivery, and outcomes—with goals, stakeholder engagement, and evaluation spanning all stages (**Figure 2**). Goals are the longer-term "ultimate" outcomes and, in the context of genomic education of health professionals, relate to improved patient outcomes. Stakeholders are people or organizations that are invested in the education intervention and evaluation. These can include funding agencies and sponsors, advocacy groups, learners, and those ultimately impacted by the intervention.

As the logic diagram depicts, the **planning** stage of the education intervention involves situation and opportunity analyses. A situation analysis considers numerous factors such as: stakeholders and partners, who may have mandates and competing priorities; project parameters (e.g., time, scope, budget); whether information exists, or can be gathered, around educational needs of target learners; and the target genomic workforce and level of genomic literacy. For example, depending on the context of the education provider (e.g., university lecturer versus clinician educator), if there is no current evidence on the genomic education needs of the target learner, the provider could conduct a needs assessment—if time and resources permit. At the minimum, this could involve assessing relevant stakeholder views on areas of genomic education that would better guide practice. An opportunity analysis may encompass potential partners (if not already identified), resources that can be repurposed or a literature review of, for example, competencies. The outputs of planning include a clearly-defined approach to stakeholder management (for example, frequency of meetings, reporting lines, etc.), goals, target groups, learning objectives, and a draft outline of the education intervention. Project management aspects overlap with components at all stages of the model. For example, at the end of the planning stage, it would be expected that education providers have all approvals and required resources in place. An evaluation plan should also be in place at this early stage—to foster transparency with stakeholders, identify questions, methods,

Planning stage deliverable. 2Testing also clarified that Project management aspects can span all stages so this component was removed.

and study design—before implementation and ensure sufficient resources for process and outcome evaluation are in place.

Activities in the **development** stage of the program logic model include evidence/theory-informed curriculum, content, and/or assessment design and development. There is growing evidence that education interventions that are based on clear theoretical foundations are more effective and have a greater impact on health professional educational outcomes than those without (Glanz et al., 2008; Bernstein, 2011). Adult learning theory is useful when considering strategies to cultivate the genomic medicine workforce, where skilled health professionals require continued education for immediate practical application (Gaff et al., 2007; Taylor and Hamdy, 2013). The development stage also includes activities related to project management to meet deadlines within budget and scope, then piloting the intervention (if appropriate) and also developing a promotion or marketing plan. The output of the development phase is an education intervention with a clear theoretical underpinning that has been planned, expertly reviewed and is ready to promote or market, with approvals and resources in place. Again, any decisions made during the development stage should be documented to allow later reflection and evaluation.

In the third stage, **delivery** of the education intervention, effective marketing is critical to success. Promotion is needed to ensure the target learners are aware of and use/attend/complete the education intervention. The education intervention is delivered or launched, including any assessment and/or immediate evaluation—such as pre-/post-workshop surveys or pop-up website user surveys—in addition to process documentation.

The fourth stage depicts the **immediate, intermediate, and long-term outcomes**. In the context of clinical genomic education, the immediate outcomes could relate to the learning objectives of the intervention, such as a change in knowledge, attitude, and skills. The intermediate outcome could relate to creating a competent genomic workforce, defined in relation to the aims of the education intervention (e.g., change in behavior). The long-term outcomes could include those related to the appropriate and timely use of genomic medicine, which then relates back to the overarching goal, which is improved patient outcomes. When constructing a program logic model to describe an education intervention, an education provider may use a series of "*if* … *then*…" statements (Owen and Rogers, 1999). For example, *if* learners complete this genomic education intervention and attain new skills (immediate outcome) *then* they become genomic-competent and practice accordingly (intermediate outcome), which *then* facilitates the appropriate use of genomic medicine (long-term outcome), which *then* improves patient care (ultimate goal).

**Evaluation** spans both the process of developing an education intervention and evaluating delivery (processes) and its impact here defined as immediate, intermediate, and long-term outcomes.

# Testing the Program Logic Model

Four workshop participants (CP, MB, BT, AM) tested the program logic model in local contexts. These included different countries (Australia and the UK) and different types of education intervention: workshops for pediatricians, a competency framework to support discussions around informed consent for genomic testing, online modules for medical specialists, and a university course. The four contexts involved varied outcomes, stakeholders and partners, and different organizations and resources. **Table 1** provides details of how each component of the program logic model was mapped to each context and the narratives below focus on different aspects of the model. Two narratives are described visually (CP and


#### TABLE 1 | Comparing components within the program logic model across four illustrative narratives.

(*Continued*)


(*Continued*)


*1www.aqf.edu.au/aqf-levels; 2(Herreid, 2005); 3(Campbell et al., 2010); 4(Schon, 1983); 5(Hase, 2009); 6(Taylor and Hamdy, 2013); 7(Beetham and Sharpe, 2013); 8(McEwen et al., 2019); 9(Herrington and Oliver, 2000); 10(McGee and Reis, 2012).*

BT) in the program logic model format (**Figures 3** and **4**), with detailed program logic models for all four narratives included as **Supplementary Materials**. Use of the program logic model for each of the four contexts are described below, followed by the changes proposed and made to the program logic model following testing.

#### Using the Model to Plan Workshops

Chirag is a medical geneticist with many years' experience developing and delivering genomic education to health professionals. He has no formal education qualification, is not supported by a university or education organization and offers occasional genomic education interventions in addition to his usual clinical workload. Chirag recently obtained funding to lead a research program exploring the use of whole genome sequencing (WGS) as a first-line investigation for pediatric patients across several tertiary specialties. For this research program to become clinically embedded and successful in the long term, hospital-based pediatricians will need to know how to order and interpret WGS tests.

TABLE 1 | Continued

Chirag used the program logic model to plan and develop face-to-face workshops to increase genomic literacy among hospital-based pediatricians in his region (**Figure 3**). Chirag's needs assessment (a survey of cross-disciplinary, tertiary hospital-based specialist pediatricians) revealed his target audience had limited genomic literacy and experience ordering and interpreting genomic tests. He used this information to help him define learning objectives and an evaluation plan to examine changes in confidence and practice. He obtained approval from the main pediatric tertiary hospital in his region to host the workshops, and a commitment to promote them to relevant staff. He also secured medical genetics colleagues to assist in teaching each workshop.

Chirag found the program logic model prompted him to include all the necessary considerations when planning genomic education for non-genetic health professionals.

*"It was great to have a formal document to use as a reference to consider all aspects of providing genomics education to nongenetics professionals. Many of the factors in the planning stage may not routinely be considered when planning smaller educational events (presentations to local departments), but clearly are essential in ensuring effectiveness and achievement of goals and outcomes for larger group educational activities like workshops. It allowed to me to ensure I had the correct resources and that I assessed the needs and current level of knowledge of my target audience, prior to designing the specific cases for the workshops."*

#### Using the Model to Aid Stakeholder Management and Reporting When Developing Competencies

Michelle has a postgraduate qualification in genetic counseling, a PhD in genetic education, and several years' international experience in genetic and genomic education. Michelle works for a national genomic education organization tasked with upskilling health professionals in genomic medicine. Previous work undertaken by Michelle's organization identified a need for competencies to support education and training of health professionals who will undertake the consent conversation for genomic testing with patients or their family members.

Michelle used the logic model to assist stakeholder identification and to develop a stakeholder management plan for developing, disseminating and evaluating the competencies.

"*We established a Working Group to oversee the development of the competencies, and I used the key points outlined in the program logic to structure the Working Group discussions. I also use these points as a checklist to structure the progress reports I submit to our Assurance Board*."

The initial stakeholder engagement activity included consultation with health service providers and laboratories, medical and nursing colleges and societies, and, at a separate event, consumer representative groups. All stakeholders agreed to the need for a set of competencies. The stakeholders wanted to *"outline the set of knowledge, skills and behaviors for 'doing the job' rather than what someone would achieve if they undertook a training session in this area"*.

Stakeholder consultation was also undertaken to develop a comprehensive mixed-methods evaluation plan that encompasses process and impact evaluation to both inform the refinement of the competence framework and answer the question: are health professionals who have genomic testing consent conversations competent in all areas of the competency framework? Michelle found the logic model helped stakeholders appreciate the importance of considering evaluation early in the planning and development phase.

"*Importantly, having the key points structured in a format that aligns to the resource development cycle means that key aspects such as defining the evaluation plan are considered throughout the development of the resource and not as an afterthought.*"

#### Using the Model for Reflection and Targeted Evaluation for Quality Improvement

Bronwyn has postgraduate qualifications in science communication and education, with many years' experience planning, developing, delivering, and evaluating genetic education interventions. She works for a medical research institute, which has a genomics-focused education and outreach team that aims to improve Australasian health professionals' understanding of, interest in, and use of genomics to facilitate its broader integration into healthcare. A previous needs assessment and opportunity analysis revealed a lack of short, accessible genomic education resources developed for Australasian medical specialists.

Planning for the educational intervention was already underway when Bronwyn participated in the program logic development workshop. Contributing to the development of the broader logic gave Bronwyn an opportunity to reflect on her own project, highlighting processes that she may have done differently or at a different stage (**Figure 4**). As current Australasian genomic competencies for medical specialists did not exist to benchmark the desired level of genomic literacy and guide stakeholder discussion on curriculum design, Bronwyn's team reviewed existing international competencies1 to synthesize 66 competencies relevant to the project. The program logic model approach prompted Bronwyn and her team to map the final content against the agreed competencies, as multiple rounds of drafting and expert review during development had resulted in changes to the original outline. They found that 56 of the 66 competencies were covered in the modules, five were deliberately removed to reduce length and complexity, and five were unintentional omissions. The reflective process using the program logic model identified areas for improvement and, if resourcing allows, these omissions will be remediated.

In response to stakeholder input, the course was deliberately designed in a modular fashion.

"*A key decision at the development stage was to have an open learning pathway so the modules could be completed as whole, or learners could select sections most relevant to them*."

However, learning analytics evaluation data reveal very few learners complete all modules. This has impacted the planned long-term evaluation of the modules, as recruitment information for the post-survey and 6-month follow-up interview was only included in the completion page of the modules, resulting in insufficient individuals being aware of the study.

Bronwyn reflected that the logic model is useful even for experienced educators:

"*As project planning was well underway by the time the program logic model was developed, it informed my input into the [international program logic model development] workshop.* 

<sup>1</sup> Inter-Society Coordinating Committee for Physician Education in Genomics (ISCC) Physician competencies; National Coalition for Health Professional Education in Genetics: Core Competencies for all Health Professionals added to by Callier et.al; European Society of Human Genetics: Core Competences in Genetics for Health Professionals in Europe.

*The aspects that extend across the model—stakeholder engagement and documentation/evaluation—are particularly valuable as reminders to review and assess the whole project at each stage. For example, a requirement to evaluate or document at both the planning and development stages may have meant we invested in a competency review at an earlier stage and/or broadened our stakeholder list.*"

#### Using the Model to Support a Cyclical Co-Design Approach When Developing a University Course

Alison is a clinician educator with an undergraduate education qualification and postgraduate genetic counseling and research qualifications. She is the Program Director for a new Master of Genetic Counseling course in Australia and used the program logic model to help monitor and manage a cyclical co-design process when developing, delivering and evaluating the course (McEwen et al., 2019). Alison's university perceived a need for the new course driven by the growing demand for genetic counselors in Australia, mirrored internationally (Slade et al., 2015; Stewart et al., 2015; Hoskovec et al., 2018). Building on early stakeholder activities undertaken by the university to scope new allied health postgraduate degrees for development, the planning stage began with extensive stakeholder engagement activities (McEwen et al., 2018). These activities revealed, for example, that existing courses were oversubscribed for limited places, and taught primarily on-campus in only two major Australian cities. Offering a more accessible course—blended learning through synchronous and asynchronous interactive online activities, on-campus intensives and clinical placements—would assist with a stated aim of the program to increase the diversity of students entering the profession.

Alison used co-design principles throughout the course development process, supported by the program logic model. The program was three months into the 15-month planning and development phase when Alison attended the program logic development workshop. Alison found the logic model aligned well with the co-design process and informed the ongoing development and delivery of curricula.

"*The inclusion of frequent check points and evaluation activities is of particular importance/relevance to ensure the program is meeting the needs of the learners, and of the practicing genetic counselors who interact with them while on clinical placement.*"

The program logic provided Alison with a framework for an in-depth evaluation that goes beyond her university's usual feedback processes. In addition to universitymandated student feedback surveys, students also provided evaluative feedback and staff completed a brief reflective survey for each subject, with the evaluations and feedback discussed at an 'end of semester' staff retreat. Survey data and feedback from class representatives further informed the co-design approach, with students providing ongoing feedback and suggestions to ensure the program is responsive to the experiences and insight of this core group of stakeholders.

Alison found the logic most helpful in illustrating the cyclical nature of planning, development, delivery and evaluation of the university award course, providing opportunities for ongoing improvement.

"*We use the program logic in a cyclic manner, to ensure we continue to reflect on the needs and goals of all the stakeholders involved, as we seek to deliver a robust and emergent genetic counselor education program.*"

# Refining the Model After Testing

Testing the model in local contexts revealed some tensions and areas for refinement. Three participants felt that 'Project management' components span all stages of the model, not just during the Development stage, as was shown in the draft version (**Figure 1**, footnote 2). Michelle noted, *"I've found the program logic incredibly helpful, not just to guide development of the competencies but also for the project management aspect, including reporting into our Delivery Board."* While we acknowledge our logic model may be used as a project management tool, this was not the primary aim and the component was therefore removed from the Development stage.

The narratives highlighted the importance of identifying and engaging stakeholders and partners as early as possible, as these groups may influence decisions made in the planning stage. Based on feedback, a 'Stakeholder management plan' was added as an output to the Planning stage, along with a 'Draft outline', to help develop the evaluation plan, seek approvals and gather resources. As Bronwyn noted,

"*Identifying partnership opportunities early in the process, even as part of stakeholder analysis, allows you to leverage their expertise from the beginning, and helps ensure that their perspectives, requirements and constraints are incorporated into your plans…… Our partnership was established once there was already a project plan, timeline and budget in place. If the partnership was established earlier, we could have avoided updating the [draft] materials. So possibly identifying partners would be best mentioned in 'Situation analysis'.*"

The feedback also confirmed the logic model can be used in a non-linear and/or iterative fashion. For example, Michelle and her team reviewed and refined the draft competencies though an iterative process using consensus methodologies with stakeholder representatives; health professionals from a range of disciplines then reviewed clinical scenarios at a workshop, mapping themes to the draft competencies and voting to highlight, and reduce, inconsistencies.2 These processes effectively combined the stage

<sup>2</sup> www.genomicseducation.hee.nhs.uk/images/pdf/Meeting%20for%20 Consent%20Competency%20Framework%2012-02-2019.pdf

of Development with the component 'Expert review'. After consideration, these were left as separate components of the model and potential overlap will be acknowledged in future companion documents.

Finally, additional tools were suggested during testing feedback that could further support genomic education development and evaluation. These included a list of organizations developing and evaluating genomic education interventions (to identify potential partners), a summary of the main education and evaluation theories, common learning designs and related assessments, expert review templates, and evaluation study designs. As Bronwyn noted, *"Even for experienced educators, I believe that a catalog of evaluation approaches and tools would be a valuable adjunct to this model."*

#### DISCUSSION

To support education providers to plan, develop and deliver genomic education interventions that achieve their goals and meet stakeholder needs, we have developed a 'generic' program logic model as part of a toolkit to support effective development, evaluation and reporting of genomic education. To optimize the model's relevance and usefulness, we used a structured, mixed-methods approach to develop a draft model, combining a literature review, expert input *via* iterative workshop activities to achieve consensus, then clarificative evaluation in local contexts to test the stages and activities within the program logic model against our aim (Owen and Rogers, 1999). The four narratives illustrated how the model can be helpful to a range of education providers (with or without education qualifications) across a range of contexts, spanning smaller, more *ad hoc* interventions to larger, clearly-structured, mandated and wellfunded interventions. While the model was not designed as a project management tool, several workshop participants were also project managers, so these aspects may have permeated the draft model as a result. Many people who develop and/or provide genomic education interventions may also be project managers and may use this model in a different way to someone who is using it to, for example, inform a theoretical framework.

The program logic developed in this paper is a versatile and useful tool for developing education interventions in different settings. Despite a "call to action" over a decade ago (Gaff et al., 2007), few papers published since have described use of program logic in their design or evaluation. This program logic model can be used to inform program development and redesign; it is not intended to be linear, but as with all program logic models, can be used through cycles, with the outputs and outcomes informing inputs and activities at different stages. As not all education providers will be familiar with a program logic model approach to developing interventions, we are developing a set of companion documents to support the use of the tool, including a "how to" guide, a glossary of terms, useful resources for both education and evaluation, and detailed definitions and examples throughout.

This model was developed with input from members of the Genomic Education and Evaluation Working Party (see Acknowledgements). These included education developers and providers from independent and government-funded organizations (e.g., Centre for Genetics Education, NSW Health; Health Education England) as well as research institutes (e.g., The Jackson Laboratory and Garvan Institute of Medical Research) and universities (e.g., University of Ottawa, University of Melbourne). While some members may have had past industry experience, none were able to provide current industry perspectives. The model may therefore be further strengthened by testing in industry and other contexts. Similarly, while deficits in the draft model were identified and addressed by expert consensus during development and testing, we expect that this will be the start of an iterative process as others use the model; we therefore encourage those who use the model to contact us to provide feedback.

Program funders typically require evidence of achieving genomic education intervention aims and objectives however, it consistently proves difficult to gather robust evaluation data for genetic education interventions, with even simple utilization statistics sometimes difficult to ascertain (Wildin et al., 2017). The program logic is a tool to support development of genomic education interventions; now the challenge is to evaluate these interventions using consistent approaches that reflect best practice in evaluation. This will also help to build an evidence base of "quality genomic education," to begin to define outcomes and impacts across different settings (Khoury et al., 2009; Wildin et al., 2017). These endeavors may be assisted by an evaluation framework for genomic education and standards for the description of genomic education interventions and evaluation outcomes.

Our proposed suite of tools to develop, evaluate and report genomic education interventions will enable education providers and researchers to begin to establish an evidence base of effective genomic education and evaluation practice. We are currently using the model to develop, deliver and evaluate continuing genomic education interventions from the ground up. Over time, we expect that other education providers will provide feedback on use of our program logic model in many different contexts. This relies on effective dissemination of iterations of the tool: effectively promoting and sharing tools and resources is a challenge generally. Reviews of genetic and genomic education interventions (see for example, (Haga, 2006; Talwar et al., 2017) quickly become outdated and are sporadic. Repositories created by specialist colleges or organizations may be helpful within disciplines but require funding for sustainability and maintenance and may be hidden behind membership firewalls, reducing accessibility. There are many high-quality genomic education repositories3 but to the best of our knowledge there are no open-access repositories for genomic education development, evaluation and reporting with international examples. For

<sup>3</sup> See, for example, Health Education England's Genomics Education Programme (www.genomicseducation.hee.nhs.uk/), Genetics Education Canada – Knowledge Organization (https://geneticseducation.ca/), the Genetics/Genomics Competency Centre (G2G2; https://genomicseducation.net) or The Jackson Laboratory (www.jax.org/education-and-learning).

example, a repository of program logic models describing genomic education interventions could be useful in showing how, over time, interventions change and adapt, and the reasons for these changes. This is the focus of ongoing research in our group but it is challenging to find. We have created a local network of genomic education and evaluation professionals (the Genomic Education Network of Australasia) to share research findings and exemplars of education and evaluation tools and networks will also be used for disseminating internationally. We recognize that sustainable, long-term hosting and dissemination of this model and body of work is necessary and continue to explore appropriate local and international options. Establishing and incorporating this evidence base is critical in the development of effective genomic education interventions that can be tailored to the needs of the audience.

#### DATA AVAILABILITY STATEMENT

All datasets generated for this study are included in the **Supplementary Files**.

#### AUTHOR CONTRIBUTIONS

CG and SM conceived the idea for the publication. AN, CG, SM, MM, HJ, AM, CP, BT and MB had intellectual input into the program logic model and preparation of the manuscript. NK provided intellectual input into preparation of the manuscript. All authors approved the final version and agree to be accountable for all aspects of the work.

#### FUNDING

This work was supported by the Victorian Government's Operational Infrastructure Support Program and a grant from

#### REFERENCES


the Australian National Health & Medical Research Council (GNT1113531).

# ACKNOWLEDGMENTS

The authors would like to thank all members of the Genomic Education and Evaluation Working Party who were involved in the development of the program logic model before, during, and after the workshop. In addition to the authors, these included: Agnes Dodds, University of Melbourne, VIC; Belinda McClaren, Murdoch Children's Research Institute, VIC; Brenda Wilson, University of Ottawa, Canada; Cathy Quinlan, Royal Children's Hospital, VIC; Fiona Cunningham, Melbourne Genomics, VIC; Jan Hodgson, The University of Melbourne, VIC; Jean Jenkins, Clinical Genomics Consultant, USA; June Carroll, Genetics Education Canada–Knowledge Organization, Canada; Kate Dunlop, Centre for Genetics Education, NSW Health, NSW; Kate Reed, The Jackson Laboratory, USA; Mike Dougherty, GenomEducation, USA; Stephanie Best, Macquarie University, NSW; Teri Manolio, National Human Genome Research Institute, USA. The authors would like to thank Krista Recsei, Pangolin Consulting, who provided helpful data and feedback on using the program logic model as the project manager and content development lead for Narrative 3. The authors also acknowledge the input and oversight provided by the Australian Genomics Workforce & Education Working Group, which includes Dr Debra Graves in addition to the authors of this manuscript.

#### SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fgene.2019.01057/ full#supplementary-material


**Conflict of Interest:** CG and SM, are co-editors of the special issue where this manuscript is featured; AN and MM are affiliated with CG and SM. CG and SM were not involved in the editorial or peer-review process. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Nisselle, Martyn, Jordan, Kaunein, McEwen, Patel, Terrill, Bishop, Metcalfe and Gaff. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Scoping the Scene: What Do Nurses, Midwives, and Allied Health Professionals Need and Want to Know About Genomics?

#### *Mona Saleh1\*, Romy Kerr1,2 and Kate Dunlop1*

1 Centre for Genetics Education, New South Wales Health, Sydney, NSW, Australia, 2 New Zealand Genetic Health Service (Northern), Auckland, New Zealand

#### Edited by:

Michael Dougherty, University of Colorado Denver, United States

#### Reviewed by:

Danya Vears, KU Leuven, Belgium Janet Karen Williams, The University of Iowa, United States

\*Correspondence: Mona Saleh mona.saleh@health.nsw.gov.au.

#### Specialty section:

This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics

Received: 26 June 2019 Accepted: 04 October 2019 Published: 12 November 2019

#### Citation:

Saleh M, Kerr R and Dunlop K (2019) Scoping the Scene: What Do Nurses, Midwives, and Allied Health Professionals Need and Want to Know About Genomics?. Front. Genet. 10:1066. doi: 10.3389/fgene.2019.01066

Introduction: Rapid changes in genomic technology are transforming healthcare delivery. Although it has been well established that many health professionals lack the adequate knowledge, skills, and confidence to adapt to these changes, the specific educational needs of Australian allied health professionals, nurses, and midwives are not well understood. This diverse group of health professionals is primarily involved in the management of symptoms and psychosocial care of patients with genetic conditions, rather than risk assessment and diagnosis. The relevance of genetics and genomics to their clinical practice may therefore differ from medical practitioners and specialists.

Materials and Methods: This paper reports on a study undertaken to identify the perceived genetic knowledge and education needs for this group of health professionals. Allied health professionals, nurses, and midwives were recruited from throughout New South Wales (NSW) and invited to participate in semi-structured telephone or face to face interviews.

Results: A total of 24 geographically and professionally diverse individuals (14 allied health, 6 nurses, and 4 midwives) were interviewed. Interview recordings were transcribed and using thematic qualitative analysis recurring themes were identified. The results show that this is a diverse group that is keen to know more about genomics and genetic services but unsure of reliable sources.

Discussion: The need for a generic update from a trustworthy source was identified and suggested topics to be covered included genetic fundamentals, recognizing common genetic conditions, and psychosocial/ethical aspects of genetics/testing including informed consent. In addition, the challenge of incorporating education into highly clinical roles was identified as a key barrier and having a readily accessible, accredited learning resource would help overcome this. Findings from this study are informing the development of a targeted, interactive e-learning resource for allied health professionals, nurses, and midwives.

Keywords: genetics, genomics, allied health, education, nurse, Australia, midwife, knowledge

1 **49**

# INTRODUCTION

The advances in genomic technology and the advent of genomic medicine are changing healthcare delivery and the educational requirements of health professionals. Where previously genetic testing was most often limited to single gene tests for conditions with a clear phenotype (Bowdin et al., 2016), non-targeted, highresolution next-generation sequencing technologies are now able to detect disease-causing changes in uncharacterized genes; identify an increased risk for complex conditions; predict disease development in the absence of symptoms; determine individual drug metabolism and efficacy; and identify personalized targeted therapy approaches (Mattick et al., 2014). Clinical genomics is moving beyond clinical genetics services to care management and treatment decisions in general medicine. This increase in utility and accessibility of genomic technology has resulted in an increased use of genomics by non-genetic healthcare providers and a change in their required knowledge and skillset (Campion et al., 2019).

Allied health professionals (university qualified health professionals with a non-medical, dental, or nursing qualification such as physiotherapists and pharmacists), nurses, and midwives are a diverse group of health professionals and as such their use of relevant genetic knowledge and skills varies. Nonetheless, many will be involved in both independent therapies or multidisciplinary work where they will encounter genetics in their clinical practice (Calzone et al., 2010; Crane et al., 2012). A survey of 3,600 American allied health professionals found that 70% of respondents reported discussing the genetic basis of health concerns with their clients and 30% reported providing counseling for genetic concerns (Lapham et al., 2000). Moreover, Barnoy et al. (2010) demonstrated that patients regarded advice about genetic testing from expert nurses and expert physicians as equally valuable, indicating a high level of trust between patients and nurses and the value of nurses with good genetics knowledge in healthcare.

The effective implementation of genomic medicine in the health system relies upon non-genetic health professionals remaining abreast with current genomic knowledge and confidently applying genetic skills in their practice. This requires maintaining a good understanding of basic genetic concepts; the current capabilities and limitations of genomic technology; the social, ethical, and psychological implications of genetic testing; the relevance of genomic medicine to clinical practice; and an awareness of available services and the confident use of skills such as family history taking and result interpretation (Bowdin et al., 2016; Tonkin et al., 2018; Wynn et al., 2018). In Australia, the National Health Genomics Policy Framework 2018–2021 focuses on integrating genomics into the healthcare sector through five main strategies, including ensuring a healthcare workforce that is literate in genomics as a priority (Council, 2017).

Despite this evidence to support the need for allied health professionals, nurses, and midwives to be equipped with genetics and genomics knowledge and skills, fewer than 30% of allied health professionals report a high level of confidence in carrying out tasks relating to genetics (Lapham et al., 2000). Over 80% of registered nurses and midwives who participated in a 2016 Australian study indicated the perception that their knowledge of genetics was poor to average (Wright et al., 2019). A systematic review of published studies reporting nurses' competence in genetics found that nurses in the United Kingdom, Europe, and the United States of America lacked the required genetics knowledge and skill to meet their national core competencies (Godino and Skirton, 2012).

Much of the existing research focus has been on the educational needs of doctors (Lapham et al., 2000; Houwink et al., 2011; Nair et al., 2018; Rubanovich et al., 2018). Some research has focused on the educational needs of nurses and midwives particularly around confidence levels (Maradiegue et al., 2008; Calzone et al., 2010; Crane et al., 2012; Godino and Skirton, 2012; Skirton et al., 2012; Calzone et al., 2013; Wright et al., 2019), with limited understanding of the educational needs of allied health professionals (Neils-Strunjas et al., 2004; Christianson et al., 2005; Zant et al., 2015; Brown et al., 2019). Furthermore, only more recently has research focused specifically on the genomics education needs of non-genetic health professionals. There remains, therefore, a gap in understanding how allied health professionals, nurses, and midwives perceive the impact of genomic medicine on their clinical practice or what their educational needs are.

Importantly, this is a clear gap for those health professionals practicing in Australia. Much of the research addressing their genetic and genomic educational needs originates from the United Kingdom or the United States of America and Canada. The Australian Genomics Health Alliance has undertaken comprehensive needs assessment of medical specialists and general practitioners in genomics education but has yet to target this group (see https:// www.australiangenomics.org.au/resources/publications/reports/).

This study aims to explore this gap in the understanding of the genomic educational needs of allied health professionals, nurses, and midwives working in Australia through a qualitative exploration of allied health professionals', nurses', and midwives' perceptions of their knowledge of genetics and genomics and its relevance for their clinical practice.

The findings of this study will be used to inform an educational strategy and resources for allied health professionals, nurses, and midwives aimed at addressing the identified educational needs.

# MATERIALS AND METHODS

# Participants

The study was approved by the Human Research Ethics Committee Review Board of Northern Sydney Local Health District. Allied health practitioners, nurses, and midwives in New South Wales (NSW), Australia were recruited using a number of targeted strategies. A letter of invitation and information flyer was sent to previous professionals in this group who had contacted The Centre for Genetics Education (CGE) for professional development in genomics over the past 2 years and also to relevant health service managers and department heads throughout NSW. The net was cast as widely as possible in order to recruit from a broad geographical area and a range of clinical specialties. Recruitment materials were sent to the NSW Ministry of Health Chief Nursing and Midwifery Officer and Committee and the Chief Allied Health Officer and Committee, as well as through local health networks including the NSW employee mailing lists through appropriate channels and with appropriate permissions. Included in the invitation to participate was a request to share the project invitation and flyer to colleagues. The promotional flyer was also placed in local health district newsletters and on staff notice boards. Flyers were also distributed to NSW clinical genetics services and genetic outreach genetic counselors (see www.genetics.edu.au).

Contact details for the researchers were included on the invitation to participate and the promotional flyer. Those who wished to participate were required to contact the researchers to indicate their interest. Interested health professionals were then sent a recruitment pack containing a participant information statement, consent form, and reply-paid envelope (if necessary). Interested participants who had not returned their consent forms 2 weeks after the initial contact were followed up by phone or email to remind them of the study and to request they return their signed consent forms if they still wished to participate. Health professionals who consented to participate were contacted to arrange a mutually agreeable time and location for a telephone or face-to-face interview.

Recruitment continued until data saturation was reached.

#### Instrumentation

An interview guide adapted in part from Reed et al. (2015) was developed and conducted with participants either face to face or over the telephone. It consisted of demographic questions followed by semi-structured and open-ended questions about participants' understanding and training in genetics and genomics; their experience of genetics in their practice; their confidence using genetic knowledge and skills; and their perceived genetic and genomic educational needs. Probes were used to encourage thorough exploration of the participant's experiences and opinions. The interviews were carried out by either MS or RK (supervised by MS). Recruitment was ceased once there was no new information or themes being observed in the interviews. Interviews were digitally recorded and transcribed verbatim.

# Data Analysis

Using QSR International's NVivo 11 qualitative data analysis software and using thematic qualitative analysis, index themes and categories were identified within the textual data. Categories were verified by at least two of the authors to maintain interrater reliability and increase validity (Miles and Huberman, 1994; Krueger and Casey, 2000; Pope et al., 2000). All the data relevant to each category were then identified, contextually defined (by referring back to the audio and/or transcripts), and coded manually. Themes recognized through this process were documented including illustrative verbatim comments from participants. RK identified the initial themes and categories and coded all transcripts. Five of these were then coded independently, using the developed categories, by MS. There was 100% consensus between both coders with regards to the main themes identified. Where small discrepancies occurred with respect to specific categories, discussions were held until consensus was reached.

TABLE 1 | Participant demographics.


Main group totals are in bold.

# RESULTS

# Participant Characteristics

A total of 24 interviews were carried out with participant characteristics shown in **Table 1**. The majority of participants were female and the mean age of participants was 48 years with an average years of practice being 18.7 years.

# Qualitative Findings

All participants acknowledged the importance of up-skilling in genomics. The extent and focus of these skills, however, and where to find appropriate education were not clear to most of those interviewed. The challenge in recognizing the relevance of genomics information was also reflected in many interviews with one participant summing this up by stating:

"I think that you don't know what you don't know until someone tells you. It's [genetics] often discussed at a higher level rather than actually explaining things properly so people don't recognize that it would be of value to your work"—Nurse (P05)

Overall, four distinct themes arose from the qualitative data: 1) existing genomics knowledge or exposure in practice; 2) relevance of genomic knowledge/skills to profession; 3) education and other challenges of incorporating genomics into practice; and 4) potential genomics topics to be incorporated into training.

Below is a summary of these themes and subcategories with evidence from transcripts to illustrate the issue.

#### Existing Genomics Knowledge or Exposure in Practice

The majority of participants felt that their graduate qualifications contained little if any genetics. If there was some genetics, it was very basic and therefore any relevant and applicable genomic education was sought out as an additional qualification or individual training.

#### **During Undergraduate Degree**

"In terms of training, basically no. I have a Bachelor of Applied Science in Physiotherapy and naturally there's physiology, DNA and a certain amount of understanding of genetics from that, but it's basic."— Physiotherapist (P01)

"We did touch on it but it wasn't as deep as what I expected it to be and I just feel the average nurse would like to know more because you open up a Pandora's Box we get told how important it is but unless you do a degree in medicine, I suppose you wouldn't know."— Nurse (P05)

#### **On-the-Job Training**

Recognizing an interest and need for improved genomic knowledge, some participants revealed how and where they had sought out further education either formally or through interactions with peers.

"Last year I organized for a geneticist and genetic counselor to come and speak to our team and give us an update. It's tricky to organize with everyone's schedules but it's worth it."—Occupational Therapist (P18)

"I've just learnt through osmosis. It's not a taught thing, just more working with the consultants and watching them take histories and things."—Nurse (P16)

There were no participants who had undertaken any formal genetics training.

#### **Interactions With Genetic Professionals and Services**

Participants had variable interactions with genetic services. They felt that doctors, rather than allied health practitioners, nurses, and midwives, would be more likely to have direct interactions with genetic services. Others, however, who worked closely with or were linked to a genetic service appeared to possess some confidence/insight into genomic knowledge and referral pathways for patients. While this was a positive finding, unfortunately, there were others with limited contact and had little awareness of what genetic services were available, what they offered, and how to contact them.

"Yes definitely yes I would just ring the [Geneticist] on call they are very approachable. Often they can answer queries on the phone but if not they will address the issue another day or they will come and see the patient. They're very good."—Midwife (P09)

"I wouldn't know where to refer them but I think I would probably get on the internet and search through a website and possibly do a preliminary phone call to make sure that was the correct service for that person to be referred to and then refer them on to that."— Social Worker (P04)

#### Relevance of Genomic Knowledge/Skills to Profession

Midwives and nurses were more likely to feel that genomics and rapidly changing screening and testing options meant that they needed to keep abreast of current practices. They tended to rely on

their professional societies and colleges to ensure they remained up to date. Attending relevant conferences or individual reading was mentioned as a way of staying informed. Some even learnt from their patients.

"A lot of women have the nuchal translucency and the [brand name] test and something else too, something 'NIP'… I'm not sure because that's all moved very quickly and because we don't deal with these things it's the women telling me what they've had rather than me understanding what they've had as such."—Midwife (P22)

"We have in-services occasionally from our genetics team here, but I've had no training."—Midwife (P09)

#### **Family Health History as a Practice Tool**

With regards to taking a family health history (FHH) and its relevance to their practice, once again it was nurses and midwives who expressed their opinion that this was relevant and in fact some responses showed a good understanding of the principal of taking a family health history. Others revealed a lack of technical understanding, feeling only maternal history was relevant.

"Yes, we take an obstetric and health history and family history, medical history; Gynecological history; Consanguinity; Standard questions."—Midwife (P08)

"Family health history yes we do basically looking at maternal family history we do ask about deafness, blindness any Down syndrome and any genetical or hereditary abnormalities in the family. Just maternal only."—Midwife (P09)

Responses from allied health professionals showed that they generally played a symptom focused role with each individual patient and therefore family health history was not seen as a priority. For specific symptomatic issues, however, family history was seen as relevant.

"I work from very much a functional point of view so if there's a functional problem then I deal with that. I mean I could get carried away with the genetics and things like that but don't, but sometimes it would be interesting to have a bit of an understanding of that."— Occupational Therapist (P18)

"No we don't do that. I mean I take a general family history especially with stuttering. I would just ask more general questions does anyone in the family have any speech language or learning delay or issues."— Speech Pathologist (P11)

#### Education and Other Challenges of Incorporating Genomics Into Practice

Participants expressed the difficulty of incorporating continuing education into their work day. A lack of time as well as difficulty finding relevant and appropriate education were given as the most common reason for this. Having education provided and supported by the employer and also earning professional/ ongoing education points for the professional were seen as the best possible way to incorporate genomic education. Also having targeted learning for specific health areas was seen as something more attractive, particularly to allied health specialists.

"I'm dubious about a lot of people going around with their shingles (office/business) providing professional development, I'm aware there are a lot of fad treatments out there and that sort of thing and I think that I would probably look at ones that has been around for a little bit longer and have research to back them up."—Speech Pathologist (P11)

"I think pharmacists probably will only be particularly interested in medication effects so you'd have to tailor it that way for it to be relevant."—Pharmacist (P17)

"Whether it's about raising awareness at the management level that can then be filtered down through allied health departments, greater availability of Continuing Professional Development (CPD) events, they [employers] might support that, and CPD events that are perhaps targeted to allied health so that we see them advertized and think oh yeah it probably is worth my while going to that, whereas at the moment if I see a genetics talk advertized I would be likely to just dismiss it as something that's more for the doctors than for me."—Occupational Therapist (P20)

#### Potential Genomics Topics That Should Be Incorporated Into Training

Participants were given an opportunity to express their perceived topics of interest and those that should be made a priority in any future genomics education packages for allied health practitioners, nurses, and midwives. Participants were prompted by being asked, "What do you feel the genetic and genomic educational needs are for your profession and what suggestions do you have for incorporating genetics and genomics education into your training/professional development?"

These are listed in **Table 2** with genetic fundamentals and genetic conditions specific to professional roles mentioned as most relevant.

TABLE 2 | Genetic and genomic topics preferred by participants.


# DISCUSSION

This qualitative study is one of the first to explore the educational needs of allied health professionals, nurses, and midwives in Australia, and includes their experience of genetics and genomics to inform education. Most other studies for this group of health professionals have focused on confidence and relevance of genomics for their practice and the level of genomic literacy. Wright et al. (2019) reported a high perceived relevance or importance of genomics to practice among Australian nurses and midwives but a low level of genomics knowledge. A recent US study of audiologists (2019) and speech pathologists reported low confidence in their ability to implement principles of genetics, but over two-thirds agreed genetics was relevant for their field (Peter et al., 2019).

We found overall that Australian allied health professionals, nurses, and midwives are aware of the importance of up-skilling in genomics but remain unclear about how it applies to their practice. We did not find that genomics was necessarily seen as relevant to their practice and that some felt genomics primarily belonged with the medical profession. Genetics and genomics have not traditionally been central to the practice of most allied health professionals and nurses. Midwives, due to awareness of prenatal testing, reported far greater exposure and were the most familiar with genetic services and understanding and recording of Family Health History. Despite this, genomic literacy has been reported in Australia as generally low in midwives (Wright et al., 2019). Allied health professionals in our study felt that their limited exposure to genomics may be related to their specific roles, which often focus on functional problems rather than diagnosis. Zant et al. reported that physical therapist educators didn't recognize the need for education due to the lack of perceived clinical applicability despite practicing physical therapists in this US study agreeing to the importance of increased genetic-related knowledge.

Participants in our study felt that reliable and relevant genomic education was not visible, and there was a lack of awareness about the role and existence of genetic services. All groups reported challenges in incorporating continuing education in their practice and highlighted the value of having education provided and supported by management and authority. Similarly, Campion et al. (2019) recommend the importance of service and educational activities of health professionals to be valued by genetics chairs and chiefs in the US. Genomics has a low profile in nursing in Australia at present (Wright et al., 2019). A mapping exercise of genomics education and training by the Australian Genomics Health Alliance Program 4 in 2018 did not identify any substantive Australian education programs for allied health professionals, nurses, and midwives except in the area of nutrigenomics for dieticians (McClaren et al., 2018). Internationally there have been significant efforts to provide accessible genomics education in particular the Health Education England's Genomics Education Programme Nursing and Midwifery Transformational Strategy, which includes postgraduate training programs and genomic competencies for nurses (Tonkin et al., 2018). Also, in the United States and Canada, a number of organizations provide accessible genomics continuing education and resources such as the NIH National Human Genome Research Institute, Jackson Laboratory, and the American Medical Association (Campion et al., 2019). However, the impact on knowledge and practice of nurses, midwives, and nursing and allied health professionals has not been reported.

Our findings indicate that genetics fundamentals as a topic were the highest priority for this group when asked about their topics of interest, followed closely by genetic conditions and genetic testing. Selecting such broad topics may reflect their lack of confidence in knowledge and the lack of genetics and genomics in their undergraduate training. While this provides a good starting point for education resource development, it is interesting that participants acknowledged and volunteered the need to learn this content, but did not demonstrate interest to seek out opportunities independently. Some allied health professionals requested targeted education reporting that generic genomics education may not necessarily be seen as relevant or a priority learning area in the clinical setting. Stevens et al. found that most nurses were aware of the importance of genetics in relation to a specific disease highlighting this need for a connection to practice. While up-skilling is seen as important, it does not necessarily equate to interest (Wright et al., 2019). Therefore overcoming this mismatch may be complex and require in these early efforts well-targeted programs to reach and engage particular groups.

### LIMITATIONS

The participants were recruited from NSW only, were selfselected, and just over a quarter had a previous connection with the researchers, which may have led to a more informed group of participants than the workforce generally. In addition, the recruitment invitations included the words "genetics" and "genomics," which in retrospect may have deterred those with no prior knowledge. However, due to the wide-reaching recruitment process, we were able to recruit a cross section of health professionals to represent the target group. Genomics knowledge was not assessed and therefore the study has no measure of what participants understood to be a satisfactory level of understanding. A limitation of the study was that allied health professionals have distinctly different roles from nurse and midwives but also among the different specialties, so it may be hard to generalize detailed findings for allied health professionals. However, themes were easy to identify and were consistent among researchers, and there was general consensus among all participants for the main themes.

# FUTURE DIRECTIONS

To adequately up-skill a workforce who lack understanding of the fundamentals of genomics and who struggle to see the relevance to their own clinical practice demands much more than incidental on-the-job training. A comprehensive and concerted approach to engaging this group in education that is targeted and relevant is required along with ongoing conversations among educators and healthcare managers to raise the profile of the importance of this education. Further research could explore the needs of specific groups of allied health professionals in genomics education and the impact of genomics education programs on knowledge and practice of nurses, midwives, and allied health professionals to further inform educational approaches.

In conclusion, our results suggest that allied health professionals, nurses, and midwives are aware of the importance of up-skilling in genomics and the need for educational resources particularly in the fundamentals of genomics. However, with few Australian education programs available, the inability to find relevance in genomics and the challenges in accessing education, nurses, midwives, and allied health professionals may fail to engage. Findings from this study will inform the development of an online genomics module and resources to be located on a state-wide education site that can be used as the foundation for targeted programs. Developing a workforce that is literate in genomics will require the development of accessible and innovative targeted education programs with support at policy and clinical level to reach and engage this group.

# DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

# ETHICS STATEMENT

The studies involving human participants were reviewed and approved by Northern Sydney Local Health District HREC. The patients/participants provided their written informed consent to participate in this study.

# AUTHOR CONTRIBUTIONS

MS and RK designed the study with advice from KD. MS and RK carried out the interviews and analyzed the data. Writing of this manuscript was managed by MS. However, there was input from all authors.

# ACKNOWLEDGMENTS

The authors wish to acknowledge the participants who gave their time freely to contribute to this study.

# REFERENCES


**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Saleh, Kerr and Dunlop. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Genomic Education at Scale: The Benefits of Massive Open Online Courses for the Healthcare Workforce

*Michelle Bishop1\*, Edward Miller1, Amelia McPherson1, Siobhan Simpson1, Stuart Sutherland2 and Anneke Seller1*

1 Genomics Education Programme, Health Education England, Birmingham, United Kingdom, 2 National School of Healthcare Science, Health Education England, Birmingham, United Kingdom

#### Edited by:

Michael Dougherty, University of Colorado Denver, United States

#### Reviewed by:

Isa Houwink, Leiden University Medical Center, Netherlands W. Andrew Faucett, Geisinger Health System, United States

> \*Correspondence: Michelle Bishop Michelle.Bishop@hee.nhs.uk

#### Specialty section:

This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics

Received: 03 July 2019 Accepted: 11 October 2019 Published: 13 November 2019

#### Citation:

Bishop M, Miller E, McPherson A, Simpson S, Sutherland S and Seller A (2019) Genomic Education at Scale: The Benefits of Massive Open Online Courses for the Healthcare Workforce. Front. Genet. 10:1094. doi: 10.3389/fgene.2019.01094

To support the delivery of the UK's 100,000 Genomes Project, Health Education England's Genomics Education Programme developed a suite of resources, including a 3-week Massive Open Online Course (MOOC) on whole genome sequencing via the FutureLearn platform. This MOOC is a synchronous learning event, with course educators and mentors (NHS healthcare science trainees in genomics) facilitating the experience in real time. Crucially, the platform allows participants to interact and learn from each other's experiences. The evaluation of the course was considered from the learners' and mentors' perspectives. Perceptions of course relevance were examined through analysis of learner comments made throughout the course and responses to an end-of-course survey. Evaluation of mentors' experiences focused on how prepared they felt to undertake their role and the value and benefit of their experience. Data was collected through a mixed methods study after the first two runs of the course. Here we present findings from 440 learners who provided end-of-course reflections, 360 learners who completed the postcourse survey and 14 mentors who facilitated the course. The course met learners' needs by providing a greater understanding of whole genome sequencing and the application of this technology in healthcare. Learners also highly valued the engagement with mentors. Mentors appreciated the experience and identified areas of professional development gained through the mentoring experience. Our findings show that a team of specialist healthcare course mentors engaging with a range of different healthcare professional MOOC learners in online conversation can enhance the learners' experiences and provide a beneficial continuing professional development opportunity for mentors.

Keywords: workforce development, genomic medicine, Massive Open Online Course, evaluation, genomic education, multi-disciplinary education, online learning

# INTRODUCTION

With the establishment of genomic medicine initiatives around the world the use of genomic information is increasingly being used as part of routine clinical practice (Stark et al., 2019). There are many challenges to successfully integrating genomics into healthcare systems, one of which is workforce capacity and capability (Manolio et al., 2015). In establishing the 100,000 Genomes Project, England became one of the first countries to introduce whole genome sequencing (WGS) into an established health system (NHS England 2019b). Alongside the scientific and clinical discoveries, this multifaceted project provided a unique opportunity to implement a co-ordinated approach to workforce education and development. Health Education England's Genomics Education Programme (GEP) was established to provide the educational support to staff delivering the project (www.genomicseducation.hee.nhs.uk).

To prioritize the education and training needs of the workforce, the GEP and key stakeholders undertook an exercise to identify the resources required to support the clinical and scientific activities across the 100,000 Genomes Project pipeline. While most of the resources supported areas specific to the project protocol, others had wider clinical applicability. One of these was a Massive Open Online Course (MOOC) 'Whole Genome Sequencing: Decoding the Language of Life and Health' (https:// www.futurelearn.com/courses/whole-genome-sequencing).

MOOCs are defined as open access courses for unlimited numbers of learners (Yousef et al., 2014). While MOOCs have been in existence for over a decade, the modern MOOC movement, characterized by the development of dedicated platforms and providers delivering online courses to large numbers of learners, began in 2012 (Pappano, 2012). MOOCs offer open access learning irrespective of geographical, professional or educational settings compared to other types of online learning (see **Table 1**).



By 2018 the total number of learners signed up to at least one MOOC had surpassed 100 million (Shah, 2018). MOOCs with a healthcare focus have seen rapid growth internationally in both the number of courses available and the number of registered learners (Liyanagunawardena and Williams 2014, Shah, 2018). MOOCs have enabled healthcare professionals to learn at scale and pace across professional and geographical boundaries (Wewer Albrechtsen et al., 2017, Liyanagunawardena and Aboshady 2018, Sneddon et al., 2018).

Another area of rapid evolution is the accreditation/ credentialing of MOOCs. As MOOC providers have increasingly focused on supporting professional development, they have developed an array of paid-for offerings on top of free, open courses so that learners can earn certificates of completion, credentials, professional body CPD points, and academic credit (Brown, 2018).

The GEP chose to partner with FutureLearn (www.futurelearn. com), a UK-based MOOC platform launched in 2013 that, at the time of writing had 18 courses that focused on genomics from a healthcare perspective. FutureLearn courses are delivered synchronously, with specific start and end dates for each run. Courses are structured into weeks, with each week containing a number of 'steps'. The content in each step can be delivered *via* different formats, the most common being text and video. Additionally, courses can contain test steps (for formative and summative testing) and poll steps for learners to vote on key topics. A defining feature of the FutureLearn platform is that learners can comment throughout the course and 'like' and comment on each other's comments. Course designers can also include specific discussion steps, actively encouraging reflection and communication amongst learners. The platform also allows course providers to allocate the role of mentor to specific members of the course delivery team who are tasked with supporting and responding to learners.

Mentors have an essential role to play throughout a MOOC run, fostering a social and connected learning experience (Leon Urrutia et al., 2015). As learners come from very diverse backgrounds, mentors can support the course delivery team in handling the variety and quantity of comments and questions raised. Furthermore, mentors act as mediators to facilitate learning and encourage learners' engagement with the course (Kop, 2011, Watolla, 2016). The open accessibility of MOOCs mean they can suffer from learner attrition with data from one of the biggest providers, edX, showing 52% of registered learners never start the course (Reich and Ruipérez-Valiente, 2019). Some commentators suggest the inability to facilitate and support such a varied cohort of learners may be one explanation for the loss of engagement (Hone and El Said, 2016).

This paper summarizes the process the GEP followed to develop the MOOC, including the recruitment and training of mentors. It reports on the short-term outcomes from the evaluation plan and comments on how MOOCs could be used to support healthcare workers' ongoing professional development.

# METHODS

#### Development of the Course

**Figure 1** outlines the steps taken to develop the MOOC, including the recruitment and training of mentors and the communication

strategy. The aim of this MOOC was to increase understanding of WGS technology and its application in healthcare, to a broad range of NHS professionals who had limited understanding and/or exposure to genomic testing. The development of the course was overseen by a course delivery team which included the authors.

# Recruitment and Training of Mentors

A decision was made by the course team to recruit healthcare professionals who are specialists in genomics to support the facilitation of the MOOC. The National School of Healthcare Science (www.nshcs.hee.nhs.uk/) invited individuals who were enrolled in NHS healthcare science training programs in genomics. These included individuals on the Higher Specialist Scientist Training (HSST) program, a five-year doctoral level program, and individuals in their final years of the Scientist Training Program (STP), a three-year master's-level program. Recruitment was targeted to trainees rather than practising healthcare professionals as trainees could use this experience as evidence for competencies in their curriculum. In this paper we present the evaluation data of the mentor model from runs 1 and 2. Sixteen individuals (12 HSST trainees, three STP trainees, one course educator) were recruited to mentor runs 1 and 2, with five mentors involved in both runs. Prior to each run, new mentors participated in an induction webinar, and were provided with guidance documents to support their mentoring activities. Our mentor model (shown in **Figure 2**) was based on that described by Leon Urrutia et al. (2015), where university graduate students (Leon Urrutia et al., 2016) and faculty (Leon Urrutia et al., 2015) were used as mentors for non-healthcare related FutureLearn MOOCs.

# Evaluation of Learners' Experiences and the Mentor Model

To evaluate learners' experiences and identify any changes to learners' knowledge, we analyzed two sources of secondary

data provided by FutureLearn. To evaluate the effectiveness of the mentor model, we adopted a mixed methods approach to data collection and analysis: stage 1 following run 1, and stage 2 following run 2. This work has been categorized as service evaluation, and as such did not require NHS ethics approval. The GEP worked within the data governance framework of Health Education England.

#### Analysis of Learners' Comments Posted During the Course

A step was created at the beginning of the course (Step 1.1) for learners to introduce themselves and outline what they hoped to gain from the course. At the end of the course (Step 3.17) learners were given the opportunity to reflect on what they had learnt and their overall experience. Comments from both steps from all six runs were downloaded from the FutureLearn platform, anonymised and uploaded to NVivo 10 for data management. Content analysis was used to categorize the comments from Step 1.1 into professional groups, then thematic analysis was undertaken using the constant comparison approach as first described by Glaser and Strauss (1967). Thematic analysis of the comments from Step 3.17 was also conducted using the constant comparison approach. Quotes presented in this paper are from all six runs and have been de-identified to remove any reference to the learner or their profession.

#### FutureLearn Post-Course Survey

At the end of each run FutureLearn administers a post-course survey. This paper presents findings from the post-course survey on learners' views of the platform and how they rated the content of the course. All survey responses were anonymous. The authors requested the data from the post-course survey, and this was provided for runs 1 and 3–6. Data from the post-course survey for run 2 was not available for analysis. The questions included in the survey were decided by FutureLearn, and not all questions were present in each survey. Learner's satisfaction with the FutureLearn platform, interactions with other learners and mentors, and course content (including complexity) was assessed through numerous statements, with learners asked to rate each statement from 1 (strongly liked/very satisfied) to 5 (strongly disliked/very dissatisfied). For this paper we use the term 'satisfied/very satisfied' for simplicity in reporting. An additional question asking if learners worked for the NHS or Public Health England (Yes/No response) was also included in the post-course survey for runs 1, 3 and 4 at the authors' request. Descriptive statistics were used to describe the findings.

#### Evaluation of the Effectiveness of the Mentor Model

Stage 1: A survey was administered to the mentors who facilitated run 1 *via* an anonymous questionnaire. Consent was implied by return of the questionnaire. Mentors were asked to provide their own description of their role and their activity as a mentor. Mentors' preparedness and feeling supported in their role, their experience of being a mentor, and the challenges they faced, were assessed through a series of statements where respondents were asked to rate each statement from 1 (strongly disagreed) to 5 (strongly agreed). Throughout the questionnaire mentors were asked open-ended questions to comment or expand on their responses. Descriptive statistics were used to describe the findings. Thematic analysis of the solicited comments was undertaken using a constant comparison approach.

Stage 2: Recruitment letters were emailed to the mentors who facilitated run 2. Participants who expressed an interest contacted MB to schedule a telephone interview. Mentors who expressed an interest but were unable to attend an interview were sent a questionnaire *via* email. Verbal consent was obtained prior to beginning the phone interviews. For mentors who received the questionnaire, consent was implied by return of the completed questionnaire. A semi-structured interview guide was used in the interviews, which were approximately 20 min per participant. The interview guide was informed from the findings of stage 1 and explored mentors' experiences, their perceptions of learners, and their impressions of the mentoring experience. The questionnaire was based on the topic guide and covered the same key areas. All interviews were recorded and transcribed verbatim. Thematic analysis of the interview transcripts and the responses to the questionnaire were conducted using a constant comparison approach. Analysis of the transcripts and questionnaire responses was undertaken by MB and checked by StS for consistency. NVivo 11 was used for data management.

# RESULTS

The MOOC has run six times between September 2016 and October 2018. Over these runs 19,683 individuals have enrolled on the course, of which 45.2% have entered the course and viewed at least one step (n = 8,894). Of these learners 28.9% (n = 2,573) were also 'social learners'—defined as posting at least one comment on any step.

#### Learners Come From Five Different Sectors

Analysis of the comments from Step 1.1 showed learners came from five different sectors (See **Table 2**). Each group had specific motivations for undertaking the course, with healthcare professionals primarily wanting to improve their knowledge of genomics and/or WGS and increase their awareness of the clinical utility of WGS. Data from post-course surveys showed that of the 316 people who completed the questionnaire (runs 1, 3 and 4), 32% worked within the NHS (n = 101).

### The Course Met Learners' Needs and Provided a Strong Foundation in Genomic Knowledge

From the 440 comments analyzed from Step 3.17 the course appeared to meet the needs and expectations of learners. After completing the MOOC, learners stated that they had:



TABLE 2 | High level overview of learners and their motivations for participating in the MOOC\*.

\*Please note that additional demographic information such as age and number of years of experience was not available for analysis.


For most learners, the course was pitched at the right level. However, a small minority (n = 3) felt the course content and discussion was too simple with one commenting the content was presented using an "unscientific narrative" (Psychiatrist). These findings corresponded with the results from the post-course survey, where 343 of the 360 respondents (93.3%) stated they were satisfied or very satisfied with the course content. In runs 4 and 5 respondents were also asked to rate the level of complexity of the course, with 91.8% of the 98 respondents stating they were satisfied or very satisfied. For those who were not satisfied, it was either because the information was considered "very difficult" (n = 2) or "way too low" (n = 6).

#### Healthcare Professionals Intended to Apply Their New Knowledge in Their Practice

As demonstrated by the comments from step 3.17, completing this course increased learners' genomic knowledge and, as one learner commented, this "helped secure a lot of terms and processes by putting them in context" (Biomedical Scientist). Those who were healthcare professionals stated they would be more confident in engaging in conversations with colleagues and having informed discussions with patients. As well as increasing their knowledge, some learners mentioned that they would take away examples of how to explain genomic concepts in an understandable way as they found they had "the tools to explain this to others" (Pharmacist).

A minority of learners stated they would use this knowledge to evaluate the benefit of using WGS within their own specialism. While it is not clear which health setting these professionals worked in, the small number who identified as working within the NHS stated that following completion of the course, they looked forward to discussing how WGS could be applied in their area of practice.

#### The FutureLearn Platform Enhanced Learners' Experiences

Learners who commented on Step 3.17 were complementary of the FutureLearn platform. The three most common reasons related to: learning as a cohort, the ability to comment on each step, and the flexible nature of the course as they could complete their learning at a time and a pace that suited them.

Learners commented on the positive experience of learning as a cohort, with one likening it to "being in a class" (Biomedical Scientist). Other learners referred to each other within the comments as "class-mates" and "fellow learners". Many of the 440 learners who provided their final reflections stated that they enjoyed reading the comments and looking at the diversity of views amongst the cohort, particularly those from patients and their families. One learner even stated that contributing to discussions was a great way to check their own understanding and to reflect on what was covered in the course.

These findings corresponded with the results from the postcourse survey, where 77.6% of the respondents to the postcourse surveys (n = 357) were satisfied or very satisfied with the discussions with other learners.

Learners also valued the mentors' contributions to the course discussion and enjoyed engaging in dialogue with individuals who worked at the "coalface" of genomic medicine. Learners appreciated the disciplined nature in which mentors responded to learners' questions which, according to some learners, "isn't the case on all courses" (Secondary school teacher).

A question about interacting with the course team was only asked in the post-course survey for runs 3–5 (n = 129). However, 93.0% of these respondents were satisfied or very satisfied with their interactions with the course team and reading comments posted by the educators or mentors.

# Mentors' Experience of Facilitating a MOOC

Nine out of the 11 mentors involved in the first run completed the stage 1 survey. All 10 mentors who facilitated the second run of the MOOC participated in stage 2 of the evaluation: six participated in a telephone interview and four responded to a structured questionnaire. The results from both stages 1 and 2 are presented together.

#### Mentors Felt Well Prepared and Enjoyed Facilitating the MOOC

All nine respondents of the stage 1 survey agreed or strongly agreed that the introductory webinar prepared them for the role. One respondent stated:

*"I think everyone was a little nervous before doing it as it was quite new, but after having a go for half an hour or so it soon became clear and quite enjoyable" (Survey Responce)*

In addition, all stage 1 respondents agreed or strongly agreed that the ongoing guidance and support from the GEP helped them perform their role and focus their activities during the MOOC run.

When asked to provide their own description of their role and activity as a mentor, the most common themes from the stage 1 survey were those of being helpful and supporting learners. In some cases, this involved answering learners' questions; for others it involved signposting to additional resources and references. For some, responding to individual learners' needs was the most enjoyable aspect of the role:

*"Just seeing people go from not understanding to understanding a topic because of my help"*

Similar themes were identified in stage 2, with an additional role described: that of clarifying misconceptions from learners who professed to be somewhat knowledgeable about the subject:

*"Presumed knowledge was a little dangerous as partially informed learners were posting well-intentioned but incorrect information in threads that needed intervention for clarification. They posted these (comments) with presumed authority which mislead the learners at times" (Mentor 8, HSST)*

When presented with a series of statements in the stage 1 survey about potential challenges, the responses show that none of these were commonly encountered by the nine respondents. However, another challenge was raised which related to the level and depth in which to respond to a question.

*"There was always a bit of a worry that my answers to questions would not be at the right level for the learner. Either too complex or too simple and therefore potentially patronizing" (Survey Responce)*

#### Mentors Found Mentoring an Unexpected Learning Opportunity

The findings from both stages 1 and 2 showed that many of the mentors had, on reflection, understood more about how different people view genomics and were more understanding of the patient perspective after their mentoring experience. Some mentors also identified new skills they had developed as a result of mentoring.

Most mentors are involved in training junior colleagues as part of their every-day role. Many mentioned how they would take the skills that they had learnt through this experience and apply them in their own practice.

*"The process of being a mentor itself allowed me to reflect on how I can improve my own training skills—ways of using open questions to stimulate independent thought" (Mentor 10, HSST)*

Mentors also reflected on how they learnt from the learners. Many mentors mentioned how this experience increased their awareness of the diversity of views about genomics.

*"I really enjoyed this and liked how it promoted questions and ideas of my own as a result of seeing such a wide range of posts made by many different people. I felt like I was educated too!" (Survey Responce)*

*"… (to see) the different types of discussions from different users and different backgrounds. I think it is quite eye opening" (Mentor 1, HSST)*

Another common theme was the value that patients (and their families) brought to the course. Like the comments posted by learners, comments from patients enhanced the mentors' experiences.

*"They really helped me to reflect on the significance of my own work, so it was interesting in putting the whole field into perspective, a different perspective. I guess it's a bit like sitting in a clinic, how people went through it, questions they have, the uncertainties and the fear, anxieties, the whole human dimension." (Mentor 2, HSST)*

Given feedback from learners, mentors also found themselves reflecting on perceptions of their own professional role:

*"Realizing I'm part of a group of genomics professionals involved in work that other health professionals and the public/patients view in wonderment and amazement." (Survey Responce)*

*"Genomics is not just in my office it's everywhere, people are interested." (Mentor 2, HSST)*

# DISCUSSION

Our course 'Whole Genome Sequencing: Decoding the Language of Life and Health' was well-received by learners, including healthcare professionals. While recognizing the biased sample of the learners who provided comments and completed the postcourse survey, those that did complete the final reflection step felt they had an improved understanding of WGS and greater awareness of the current applications and limitations of this technology in healthcare after completing the course.

The FutureLearn platform, and course structure, was wellreceived by learners. Preferred features included the availability of content in different formats, the flexible nature of the course, learning as a cohort and social learning. The FutureLearn platform encourages learners to engage with the course and expand the discussion by drawing on individuals' different perspectives (Sharples et al., 2014), and is built on the foundations of Conversation Theory (Pask, 1976) and Conversational Framework (Laurillard, 2002). Central to this framework is the continual dialogue between learner and teacher (or in our case, mentor), as well as between learners, which extends the learning experience. While FutureLearn encourages the peer-to-peer learning model, the results from our evaluation demonstrate the critical role mentors play in overseeing these conversations to ensure any misconceptions raised by well-intentioned learners are not perpetuated as 'scientific fact'.

### The Mentoring Model Can Act as a Template for Other MOOCS

One of the most successful features from the learners' perspective was the mentors. The model we used, based on Leon Urrutia et al. (2015), ensured mentors were well prepared for the role, understood the expected duties and responsibilities, and were supported throughout their experience. This validates the system reported by Leon Urrutia et al. (2015) demonstrating the effectiveness of establishing a virtual reporting system in order to ensure a connected mentoring team. We have used the mentors' evaluation data to refine the induction sessions (as shown in **Figure 2**), and to monitor mentors' activities in subsequent runs. Sustainability of mentoring MOOCs has been raised in the literature, with studies identifying workrelated implications for mentors due to unrealistic expectations of workload (Sinclair et al., 2015, Leon Urrutia et al., 2016, Watolla, 2016). This was not an issue raised in our evaluation, likely due to two factors: the use of rotas to organize mentor shifts to fit in with their day-to-day workload; and the fact that mentors could use their activity as evidence for their training program portfolios. In addition, our model is potentially more sustainable since it draws on mentors from large populations of trainees rather than a much smaller academic team (Leon Urrutia et al., 2016, Watolla, 2016).

This evaluation also highlighted unexpected benefits for the mentors. Some of these benefits, such as learning new skills that can be applied in their own training practice and appreciating the diversity of views about the topic area, have previously been reported by Leon Urrutia et al. (2016) who explored the experiences of PhD students as MOOC mentors. Our mentors also identified additional benefits such as gaining a new perspective on their own role in the clinical pathway and hearing from patients about the impact of genomic testing. Although not patient facing, as they are based in clinical laboratories, acting as a mentor has provided them with an opportunity to interact with and hear from patients firsthand, which they do not usually experience as part of their day-to-day practice. Additionally, this experience could also be used as evidence of patient interaction for their training program (The Royal College of Pathologists, 2015, National School of Healthcare Science, 2016).

# Taking This Work Forward

The need to support NHS staff in education, training and professional development in all specialties has been highlighted in recent NHS policy documents (NHS England, 2019b, NHS England, 2019a). Investment in continuing professional development (CPD) for NHS staff has decreased over recent years, and there has been a recent call for this to be reversed (NHS England, 2019a). The findings from this study suggest using mentors to facilitate MOOCs may be one avenue to explore as a sustainable approach to the provision of high-quality healthcare CPD opportunities, especially where scale is required. Although the student cohort who undertake MOOCs can be quite diverse, the learning can be personalized, as mentors can intervene and elevate the learning by engaging in discussion and signposting to additional resources (Leon Urrutia et al., 2015). The investment required for the development and sustainability of MOOCs could be offset by savings from the costs of releasing staff to attend face-to-face events.

We have shown this model of learning is acceptable to healthcare staff with the added benefit of providing professional development for mentors. During more recent runs of this MOOC recruitment of mentors has expanded to other genomic professions (genetic counsellors, bioinformaticians) with similar benefits seen (unpublished data). As genomic medicine becomes embedded in mainstream care recruitment could be upscaled to include healthcare professionals not typically associated with genomics. Just like other online courses, MOOCs can be used as stand-alone educational resource, as seen here, or part of a structured course (Yousef et al., 2014, Cornelius et al., 2019). As with all educational material, investment will still be required to support the up-frontcosts of course development, and mechanisms will need to be in place to keep course content current. While we have shown the benefit of using frontline healthcare professionals as mentors for our MOOC, more research will be needed to see if this mentor model can be replicated for other healthcare professional groups and in other healthcare settings.

# Conclusion

MOOCs are an excellent vehicle for reaching large numbers of learners from across healthcare professions. The use of frontline practitioners as course mentors was successful in this setting: these mentors enhanced the learning experience, while the model itself developed frontline staff as educators. Further research is needed to see if this model, which may offer a sustainable way to deliver healthcare MOOCs, can be replicated, both in terms of using different professional groups as mentors and in healthcare settings outside of the NHS.

# DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

# ETHICS STATEMENT

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.

# AUTHOR CONTRIBUTIONS

MB conceived the idea for publication. MB and StS had intellectual input into the evaluation design. MB, SS, and StS contributed to data collection and analysis. MB, EM, AM, SS, StS, and AS provided intellectual input into preparation of the

# REFERENCES


manuscript. All authors approved the final version and agree to be accountable for all aspects of the work.

# FUNDING

This work was supported by Health Education England's Genomics Education Program.

# ACKNOWLEDGMENTS

We gratefully acknowledge Tim Packwood, Samantha Butler, and Steve Scott for their contribution to the course development and the mentors who volunteered their time to facilitate the MOOC.


**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Bishop, Miller, McPherson, Simpson, Sutherland and Seller. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Informing Integration of Genomic Medicine Into Primary Care: An Assessment of Current Practice, Attitudes, and Desired Resources

*June C. Carroll1\*, Judith Allanson2, Shawna Morrison2, Fiona A. Miller3, Brenda J. Wilson4, Joanne A. Permaul5 and Deanna Telner6*

1 Sinai Health System, Department of Family and Community Medicine, University of Toronto, Toronto, ON, Canada, 2 Department of Genetics, Children's Hospital of Eastern Ontario, Ottawa, ON, Canada, 3 Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, ON, Canada, 4 Division of Community Health and Humanities, Memorial University of Newfoundland, St. John's, NL, Canada, 5 Sinai Health System, Ray D Wolfe Department of Family Medicine, Toronto, ON, Canada, 6 South East Toronto Family Health Team, Department of Family and Community Medicine, University of Toronto, Toronto, ON, Canada

#### Edited by:

Michael Dougherty, University of Colorado Denver, United States

#### Reviewed by:

Jason L. Vassy, Harvard Medical School, United States W. Gregory Feero, Dartmouth College, United States

\*Correspondence:

June C. Carroll June.Carroll@sinaihealthsystem.ca

#### Specialty section:

This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics

Received: 27 June 2019 Accepted: 28 October 2019 Published: 21 November 2019

#### Citation:

Carroll JC, Allanson J, Morrison S, Miller FA, Wilson BJ, Permaul JA and Telner D (2019) Informing Integration of Genomic Medicine Into Primary Care: An Assessment of Current Practice, Attitudes, and Desired Resources. Front. Genet. 10:1189. doi: 10.3389/fgene.2019.01189

Introduction: Preparing primary care providers for genomic medicine (GM) first requires assessment of their educational needs in order to provide clear, purposeful direction and justify educational activities. More understanding is needed about primary care providers' perspectives on their role in newer areas of GM and what resources would be helpful in practice. Our objective was to determine family physicians' (FP) current involvement and confidence in GM, attitudes regarding its clinical value, suggestions for integration of GM into practice, and resources and education required.

Methods: A self-complete anonymous questionnaire was mailed to a random sample of 2,000 FPs in Ontario, Canada in September 2012.

Results: Adjusted response rate was 26% (361/1,365), mean age was 51, and 53% were male. FPs reported many aspects of traditional GM as part of current practice (eliciting family history: 93%; deciding who to refer to genetics: 94%; but few reported confidence (44%, 32% respectively). Newer areas of GM were not part of most FPs' current practice and confidence was low (pharmacogenetics: 28% part of practice, 5% confident; directto-consumer genetic testing: 14%/2%; whole genome sequencing: 8%/2%). Attitudes were mixed with 59% agreeing that GM would improve patient health outcomes, 41% seeing benefits to genetic testing, but only 36% agreeing it was their responsibility to incorporate GM into practice. Few could identify useful sources of genetic information (22%) or find information about genetic tests (21%). Educational resources participants anticipated would be useful included contact information for local genetics clinics (89%), summaries of genetic disorders (86%), and genetic referral (85%) and testing (86%) criteria. About 58% were interested in learning about new genetic technologies. Most (76%) wanted to learn through in-person teaching (lectures, seminars etc.), 66% wanted contact with a local genetic counselor to answer questions, and 59% were interested in a genetics education website.

1 **65** Conclusion: FPs lack confidence in GM skills needed for practice, particularly in emerging areas of GM. They see their role as making appropriate referrals, are somewhat optimistic about the contribution GM may make to patient care, but express caution about its current clinical benefits. There is a need for evidence-based educational resources integrated into primary care and improved communication with genetic specialists.

Keywords: primary health care, genomics, genetic services, health services needs, questionnaire

# INTRODUCTION

Genomic medicine (GM) is anticipated to profoundly affect medical practice. Primary care providers (PCPs), as first contact with the health care system and key to continuous and coordinated care, will be critical to the effective and appropriate implementation of GM. In studies over a decade ago, PCPs described how they would play an increasing role in GM. Essential skills identified by PCPs at that time included taking a family history, assessing genetic risk, providing a gatekeeping function by deciding who is appropriate for referral to genetics, providing patient support and coordinating surveillance and management.(Emery et al., 1999; Carroll et al., 2003) Over the subsequent years, integration of GM into clinical practice, including primary care, has been slow. A key reason for this is the lack of evidence of clinical utility of many genetic tests, but barriers and challenges to primary care implementation also include concern about the ethical, legal, and social implications of genetic testing, lack of PCP knowledge and skills, systems issues (e.g. time), and lack of awareness of genetic services. (Delikurt et al., 2015; Mikat-Stevens et al., 2015) PCPs and genetics experts acknowledge that PCPs need more knowledge in the area of genomics.(Emery et al., 1999; Carroll et al., 2003; Skirton et al., 2010; Houwink et al., 2011; Carroll et al., 2016a). Recognizing that a disease might be hereditary, indications for genetics referral and benefits and limitations of genetic tests ranked highest in a study of educational needs for general practitioners by a heterogeneous panel of experts.(Houwink et al., 2012) Core competencies in GM for health professionals have been developed.(Skirton et al., 2010; Korf et al., 2014) There is agreement that strategies to enable the appropriate integration of GM into primary care require more than merely addressing a knowledge deficit, but must also address attitudes and propose new systems of care to facilitate practice. These proposed "roadmaps" include training and education but also innovative systemic changes such as integration of genomic results into the electronic health record (EHR) with clinical decision support, and new models of delivering genetic services such as genetic counselors or nurses embedded in primary care clinics or made available through telephone helplines, etc. (Battista et al., 2012; Manolio et al., 2013; Houwink et al., 2013; David et al., 2015)

Preparing PCPs for GM first requires an assessment of their educational needs, in order to provide clear and purposeful direction and to justify educational activities. Little is known about what role PCPs see for themselves in the rapidly changing landscape of GM including pharmacogenomics, direct-to-consumer genetic testing and whole genome sequencing, or what system changes they think might be helpful and would be willing to incorporate in their practices. Our objectives were to determine family physicians' (FP) current involvement in GM, confidence in GM primary care competencies, attitudes regarding the clinical importance of GM, awareness of genetic services, resources required, and suggestions for changes that would enable integration of GM into practice.

# MATERIALS AND METHODS

### Questionnaire Design and Administration

This study used a self-complete, anonymous questionnaire which was developed by a multidisciplinary team. Where possible, questions were derived from the literature or previous questionnaires. (Carroll et al., 2009; Carroll et al., 2011) The questionnaire was divided into eight sections: current role and confidence in the tasks of each role providing genetic services in their practices (14 questions), completion of family history (2 questions), attitudes toward GM (11 questions), awareness of and experience with genetic services (12 questions), knowledge (18 questions), education and resources required (37 questions), and demographics (18 questions). Answers were a mixture of 3–5 point Likert scales (confidence, attitudes, awareness, resources), yes/no (experience), and multiple choice (knowledge). The knowledge component of the questionnaire consisted of 10 clinical vignettes with an accompanying question (4 cancer; 1 inheritance; 2 prenatal; 1 pediatric; 1 consanguinity; 1 adult onset disorder). One question asked "What would help you integrate genomic medicine into your practice in the future?" Several options were listed that were derived from the literature (Battista et al., 2012) as well as the research team, with a box to add "other" suggestions. Questions were pilot tested for face and content validity with 20 FPs from three practices.

In the body of the questionnaire we defined genomics as "the study of genes, their function and their interaction with all the other genes in the genome and the environment." GM was defined as medicine that "uses genomic information and technologies (e.g. DNA sequencing) to determine an individual's risk, predisposition, diagnosis and prognosis, and the selection and prioritization of therapeutic options (e.g. pharmacogenetic testing prior to administration of certain medications)."

The study was conducted from September 2012 to April 2013. Questionnaires were mailed to a random sample of 2,000 Ontario FPs taken from Scott's Directory of Canadian physicians. A modified Dillman Method was employed (Dillman et al., 2009) including an introductory letter, questionnaire package 1 week later with instructions for a web link if preferred for questionnaire completion, a postcard reminder/thank you 2 weeks following the questionnaire, a second questionnaire package to non-responders 4 weeks following the postcard, and final mailed reminder 8 weeks later. As a token of appreciation, once a completed questionnaire was received, the respondent was entered into a draw to win one of twenty \$150 Amazon Canada gift cards. FPs were considered eligible if they were in active fulltime or part-time practice of family medicine in Ontario, Canada. Ethics approval was obtained from the Children's Hospital of Eastern Ontario Research Ethics Board.

# Statistical Analysis

Completed questionnaires were coded, data were entered into an Excel spreadsheet, and analyzed using IBM SPSS, version 23 (IBM Corporation, Armonk, New York, USA, 2015).

Five-point Likert scales were collapsed into binary data by combining levels 4 and 5 for confidence variables as "confident" in skills, for attitudes and awareness variables as "agree/strongly agree," for interest in education variables as "moderate/high," for genetics resources as "useful/very useful." A confidence score was created from items 1–10 of **Table 2**. These items were chosen as they were considered current core GM skills. We did not include newer skills related to pharmacogenomics and direct-toconsumer testing. One point was given for a rating of 4 or 5 on a confidence item, with a total score of ≥5/10 items indicating a "high" confidence score. A knowledge score that was greater than 7/10 correct was categorized as "high."

Frequency distributions provided a descriptive analysis of the data. Correlation analysis was used to establish if there was an association between high knowledge and high confidence. Chisquared analyses were conducted to look for associations between demographic variables and outcomes. Variables with significant associations were entered into binary logistic regression models to determine if they were predictors of confidence, attitudes, awareness, knowledge, and education and resources regarding GM. Covariates included in the model were older age (≤50/ > 50 years), younger age (≤40/> 40 years), sex (male/female), years in practice (<15/≥15 years), practice location (urban – population ≥ 500K/rural – population < 500K), practice type (solo/group or other), focused practice (yes/no), involved in teaching (yes/no), use electronic medical record (EMR) (yes/no), formal education in genetics (yes/no), continuing medical education in genetics in the last 5 years (yes/no), special interest in genetics (yes/no), and genetic condition in a close family member (yes/no).

# RESULTS

### Demographics

In total, 2,000 surveys were mailed, of which 159 were ineligible: wrong address, not in active practice or deceased, not practicing in Ontario, or belonged to excluded specialties. Of the remaining 1841 questionnaires, 361 were returned completed, giving a raw response rate of 19.6%. A random sample of 100 of the 1,442 non-responders was contacted by the project manager (SM) to determine if they met the eligibility criteria. Of those, 33 of the 100 contacted were not eligible for the reasons listed above. We then assumed that approximately 33% of the total non-responder group would also be ineligible, giving an adjusted response rate of 26.4% (361/1,365 eligible FPs) (**Figure 1**).

Demographics of respondents are shown in **Table 1**. Mean age was 51 years, with 53.2% male. Most (72.5%) had no formal education in genetics, but a small proportion indicated a special interest in genetics (18.3%), presence of a genetic condition in a close family member (20.7%) or had personally seen a genetic counselor or geneticist (10.6%).

# Current Role in Genomic Medicine

Participating FPs reported high involvement in some aspects of traditional GM (eliciting FH (93.3%)), identifying individuals with genetic conditions (89.5%), deciding who should be offered genetic referral (93.8%), knowing where to refer for genetic counseling (91.9%), and providing support to a patient coping with a genetic test result (82.8%) (**Table 2**). Most respondents (69.2%) reported completing a family history on 100% of new patients, with 72.6% reporting they routinely updated the family history yearly or at the periodic health exam.

However, reported involvement in some GM tasks was more limited with fewer than two-thirds saying that evaluating the use of a genetic test, discussing benefits, risks, and limitations of genetic testing with patients, describing what to expect at a genetic counseling session, and obtaining credible, current information about genetics were part of their current practice. Finally, involvement in emerging genomics practices such as pharmacogenetics (28.0%), direct-to-consumer genetic tests (discussing risks/benefits/limitations 16.7%, interpretation 14.1%), and whole genome sequencing (7.6%) was even more limited.

# Confidence in GM Skills

Self-reported confidence for these same GM skills was generally low (**Table 2**). Even for high involvement skills, confidence was moderate (ranging from 21.3% to 55.3%), while fewer than 5% agreed/strongly agreed they were confident in the emerging genomic practices listed above.

# Attitudes Toward GM

More than half (203/342, 59.4%) agreed/strongly agreed that they expected advances in GM to improve patients' health outcomes and that they needed to keep up to date with advances in GM (179/343, 52.2%) and 43.1% (148/343) agreed it was important to learn about personalized patient care based on targeted or whole genome sequencing (**Table 3**). Fewer than half (124/342, 36.3%) agreed it was their responsibility to incorporate GM into practice or saw sufficient benefits to warrant testing for inherited adult onset disease (140/342, 40.9%). Only 15.2% (52/341) agreed or strongly agreed that genomics is an exciting part of practice. However, the majority agreed/strongly agreed that GM

is going to make important contributions to the diagnosis and management of prenatal (269/342, 78.7%), pediatric (259/342, 75.7%), and adult onset conditions (215/341, 63.0%).

# Awareness of Genetic Services

Very few agreed/strongly agreed that they could identify useful sources of information regarding genetics for their practice (78/349, 22.3%) or could find information about genetic tests available within the health care system (74/348, 21.3%) (**Table 4**). The majority however, knew where to refer for various genetic disorders (prenatal 240/255, 94.1%; newborn screening 173/216, 80.1%; pediatric 241/294, 82.0%; adult onset 247/328, 75.3%), with most having referred for prenatal genetic issues or adult onset genetic disorders (prenatal 177/253, 70.0%; newborn screening 69/210, 32.9%; pediatric 106/282, 37.6%; adult onset 236/327, 72.2%).

# Knowledge Regarding Clinical Genetic Disorders

The median knowledge score on the 10 clinical vignettes was 6/10 with a range from 0 to 10 (**Table 5**). On average, 31.0% indicated they were unsure of the answer.

# Genetics Resources

Resources "usually used" for information about genetics included Up to Date® or similar internet sources, Google or Wikipedia (**Table 6**). Fewer than half used their local genetics clinic or local

#### Carroll et al. Preparing Primary Care for Genomics

#### TABLE 1 | Participant demographics (n = 361)\*.


\*Includes all respondents including family physicians (FPs) in focused practice.

specialists. Resources that respondents indicated would be useful included local genetics clinic contact information (308/347, 88.8%), genetic referral (293/343, 85.4%), and testing (296/344, 86.0%) guidelines, information summaries for patients about genetic disorders (246/344, 71.5%) and disease-specific risk assessment tools (279/343, 81.3%). Over half (193/342, 56.4%) thought a genetics education website would be useful (results not shown). Respondents indicated their level of interest in a menu of education topics in GM listed in **Table 7**. More than half (205/355, 57.7%) expressed moderate to high interest in learning about new advances in genomic technologies.

#### TABLE 3 | Attitudes toward genomic medicine.


Contact with a local genetic counselor by telephone/fax or email (225/339, 66.4%) or a buddy system with a geneticist being available for questions (172/339, 50.7%) were the most popular suggestions for how to integrate GM into primary care practice. Less than half wanted a visiting genetic counselor providing educational sessions (118/339, 34.8%), a FP in their clinic with a special interest in genetics (73/339, 21.5%), or a genetic counselor in the clinic seeing patients (65/339, 19.2%) (results not shown).

#### TABLE 2 | Current role in delivering genomic medicine and confidence with each task\*.


\*Includes only respondents who indicated they were not in focused practice, i.e. provided full scope family medicine.

#### TABLE 4 | Awareness of genetic services.


\*Includes only respondents who provide care in specified areas.

There was a weak positive correlation between high knowledge and high confidence (Pearson correlation coefficient r = 0.227, p < 0.001). No demographic variables were associated with high confidence. Being age 50 or under (40.7% ≤50 vs 21.5% > 50, p < 0.001), female (38.2% vs 23.2% male, p = 0.005), in group practice (35.2% group vs 14.3% solo, p = 0.001), involved in teaching (36.7% teaching vs 21.7% not, p = 0.005), using an EMR (34.4% using EMR vs 16.0% not p = 0.002), having some formal genetics education (41.4% education vs 26.0% not, p = 0.009), and indicating interest in genetics (42.9% interest vs 27.7% not indicating interest, p = 0.036) were significantly associated with higher knowledge. Respondents who were involved in teaching (43.4% vs 28.1% not in teaching, p = 0.004), indicated interest in genetics (50.0% vs 33.6% not interested, p = 0.024), or had high confidence in the GM skills specified (50.9% vs 30.2% low confidence, p = 0.004), were more likely to agree/strongly agree that it was their responsibility to incorporate GM into their practices.

**Table 8** indicates predictors of high reported confidence in various clinical skills in GM. Participants who indicated they had an interest in genetics were twice as likely to have a high confidence score (≥5/10) (OR 2.17 95% CI 1.00–4.70, p = 0.05). Individuals who indicated an interest in genetics were also more likely to agree or strongly agree that advances in GM will improve patients' health outcomes (OR 3.18, 95% CI 1.50–6.71, p = 0.002) and that it is their responsibility to incorporate GM into practice (OR 1.93, 95% CI 1.03–3.63, p = 0.042). (**Table 8**) Female FPs (OR 1.90, 95% CI 1.05–3.41, p = 0.033) and those indicating an interest in genetics (OR 2.01, 95% CI 1.01–3.98, p = 0.046) were also significantly more likely to have a high knowledge score (≥7/10) (**Table 8**).

TABLE 5 | Clinical vignettes/knowledge questions regarding clinical genetic disorders.


(Continued)

#### TABLE 5 | Continued


Those who indicated an interest in genetics were significantly more likely to indicate moderate or high interest in almost every type of education offered (**Table 9**). Those who use an EMR were more likely to find various guidelines, apps, and tools useful (**Table 9**). We compared demographic variables of those who indicated a special interest in genetics with those who did not. The only significant difference was that 32% of those with a special interest in genetics indicated they had a genetic condition in the family compared with 18% of those with no special interest (p = 0.15).



\*Includes all respondents including FPs in focused practice.

#### TABLE 7 | Genomics topics of interest to family physicians\*.


\*Includes all respondents including FPs in focused practice.

#### DISCUSSION

This study offers a comprehensive view of FPs' involvement, confidence, attitudes, and resources needed in GM. The vast majority of participating FPs reported that key tasks in the delivery of traditional GM (eliciting family history, identifying patients with a genetic condition, deciding who should be offered genetic referral, knowing where to refer) were part of their current practice. The concern is that their confidence in these tasks was low. Fewer than half were confident in eliciting FH and knowing who to refer. There was a weak positive correlation between knowledge and confidence. Those who indicated they had continuing education in genetics in the past 5 years had significantly increased confidence in a number of GM skills. This lack of confidence has been shown in many studies spanning almost two decades (Suchard et al., 1999; Greendale and Pyeritz, 2001; Burke, 2004; McCahon et al., 2009; Carroll et al., 2011; Mainous et al., 2013; Rinke et al., 2014; Chambers et al., 2015) Fewer than 2/3 of participants in our study reported that evaluating or discussing genetic tests was part of their current practice. This is similar to a recent US study of PCPs where only 19% had ordered genetic testing, and 18% had consulted with a genetic counselor in the past 6 months, most frequently for cancer risk testing and prenatal testing. (Chambers et al., 2015) Many genetic tests are already in the primary care domain and with new advances in GM, it is likely more will be available to PCPs. It is also likely that limited genetics resources (e.g. genetics clinics with long wait times), and few genetic specialists and counselors, will push more genetic testing into PC practice and that genetics specialists will be looking to their PCP colleagues to take a bigger role in pre-test counseling and assessment.

Attitudes regarding GM were mixed. Over half the respondents agreed that GM is going to make important contributions to diagnosis and management and will improve health outcomes. However fewer than half (41%) of the responding FPs agreed TABLE 8 | Confidence, attitudes, awareness, and knowledge regarding genomic medicine: significant results from binary logistic regression analysis.


CE, continuing education in genetics in last 5 years. Genetics education, some formal education in genetics.

CI, confidence interval; EMR, electronic medical record.

there are sufficient benefits to warrant testing for inherited adult onset diseases, and were even less convinced that it was their responsibility to incorporate genomics into practice (26%). The literature is mixed in this regard with some reporting cautiously optimistic attitudes about genetic testing, citing its value for risk stratification, and that testing is likely to have impact on clinical practice in the future, (Mainous et al., 2013; Manolio et al., 2013; Chambers et al., 2015) and others expressing caution about the role of FPs in clinical genetics (Mathers et al., 2010) and wanting more evidence of clinical utility (Mainous et al., 2013). It is interesting that an interest in genetics was predictive of "positive" attitudes to GM, needing to keep up to date and incorporate GM into practice.

Our findings regarding some of the newer areas of GM are similar to those found in the literature. Not surprisingly, emerging areas such as pharmacogenetics, direct-to-consumer genetic testing, and whole genome sequencing were less likely to be part of current practice and confidence in these areas was low. Haga's study of PCPs showed that most (73%) had heard of pharmacogenomics and anticipated its value in informing drug response (65%) (Haga et al., 2012), however only 13% felt well-informed and 67% were uncomfortable ordering a pharmacogenetic test. This study concluded that "primary care practitioners envision a major role for themselves in the delivery of pharmacogenomic testing but recognize their lack of adequate knowledge and experience about these tests," (Haga et al., 2012) very similar to how providers see GM generally. A similar situation exists for direct-to-consumer genetic testing. Health care providers report low awareness and experience of direct-to-consumer genetic testing (Bernhardt et al., 2012; Ram et al., 2012; Goldsmith et al., 2013; Carroll et al., 2016a; Carroll et al., 2016b), however, many believe it will be helpful in patient management (Bernhardt et al., 2012; Powell et al., 2012a; Powell et al., 2012b). In Powell's survey of PCPs, of 39% who were aware of direct-to-consumer genetic testing, 43% thought it was clinically useful. The majority (85%) were unprepared to answer patient questions and 74% wanted to learn more. (Powell et al., 2012a; Powell et al., 2012b) This is in contrast to a study of academic FPs who were concerned that direct-to-consumer genetic tests might cause more harm than benefit. (Mainous et al., 2013) Many patients however, plan to share their personalized genomic test results with their PCP (Van der Wouden et al., 2016) and report satisfaction with that encounter if they perceive that the PCP



CE: continuing education in genetics. Table 9. Genomic Medicine Education and Resources: Significant Results from Binary Logistic Regression Analysis

understands genetics and is willing to discuss test results. (Van der Wouden et al., 2016)

Addressing system issues has been highlighted as important to successful integration of genomics into primary care practice. (Mathers et al., 2010; Manolio et al., 2013; David et al., 2015) Less than a quarter of participating FPs indicated they could find information about genetics and available genetic testing, although encouragingly, most knew where to refer for genetic disorders. Fewer than half contacted their local genetics clinic for information, the majority used various internet resources. These findings speak to the challenge of educational initiatives, the need to enable providers to assess when genomic testing offers added value and will change patient outcomes (Manolio et al., 2013; David et al., 2015), and the need to strengthen the relationships between genetic centers and the PC community in order to make GM services more accessible.

Increasing skills and confidence in taking a FH should be a key priority for medical education at all levels. Family history is still relevant in the genomic era as it is key to risk assessment, informing appropriate screening, and identifying those who may benefit from genetics consultation. (Skirton et al., 2010; Doerr and Teng, 2012; Pyeritz, 2012; Korf et al., 2014) Opportunities should be sought to build on existing knowledge and skills in eliciting FH, to frame GM as part of ongoing skill development, not a specialized area of medicine dealing with "rare" diseases. (Botkin et al., 2015) Development of FH tools suitable for primary care, that are integrated into the EHR with clinical decision support, may facilitate this.

More efforts are needed to develop both effective education and practice strategies to enable PCPs to integrate GM into primary care. This needs assessment builds on existing literature to provide direction to educational initiatives. Core competencies in genetics for non-genetics health professionals have been proposed (Burke et al., 2009; Skirton et al., 2010; Houwink et al., 2013; Manolio et al., 2013; Korf et al., 2014) including taking a FH, risk assessment, when and how to order genetic tests, interpretation, pharmacogenetics, ethical dilemmas and psychosocial effects related to genetics, and insight into the organization and role of clinical genetics services (Houwink et al., 2011). Clearly the FPs in our study identified taking FH, knowing who to refer and supporting patients who received genetic results as their current role, suggesting that educational and practice strategies should focus in these areas. Our results would suggest that newer educational methods such as podcasts and web-based tools may be more appealing to younger physicians. There are limited studies of educational interventions in GM showing mixed effectiveness. (Rubanovich et al., 2018) They include studies of interactive webbased curricula and educational modules (Blazer et al., 2005; Blazer et al., 2011; Houwink et al., 2013; Bell et al., 2014; Houwink et al., 2014; Orlando et al., 2014; Reed et al., 2016; Paneque et al., 2017), FH and clinical support programs (Jackson et al., 2018), point-ofcare tools and decision support (Carroll et al., 2011; Carroll et al., 2014), and push reflective e-learning (Carroll et al., 2016b). Several websites exist with genomics information and on-line educational programs for PCPs (GECKO www.geneticseducation.ca; Genetics in Primary Care Institute https://www.aap.org/en-us/advocacyand-policy/aap-health-initiatives/Pages/Genetics-in-Primary-Care-Institute.aspx; Genomics Education Programme, www. genomicseducation.hee.nhs.uk; The Jackson Laboratory, https:// www.jax.org/education-and-learning/clinical-and-continuingeducation; Genetics/Genomics Competency Centre, www.g-2-c-2. org; Gen-Equip programme, www.primarycaregenetics.org). A recent systematic review of interventions providing genetics education for PCPs highlights some of the challenges in this area and the need for evaluation of educational initiatives to include changes in practice to see if they are effective in improving patient management. (Paneque et al., 2016) Generally, initiatives using effective continuing education strategies (interactive, case-based, skill focused, sequential reinforced learning) have been most successful. (Paneque et al., 2017)

The abundance of studies over the past decade demonstrating a continued lack of knowledge and confidence in GM among PCPs shows that education alone is not sufficient. As Feero says "Available studies suggest that development and maintenance of freely available high-quality genomics reference and educational materials is likely insufficient to ensure a meaningful increase in genomics competency among non-geneticist health providers." (Feero et al., 2014) Among the cultural and infrastructure changes he recommends are efforts to address the usability of EHR to manage and interpret genomic information and the time/cost burden in practice. Burke has also addressed the slow introduction of personal genomics into practice. (Burke and Korngiebel, 2015) She describes several factors that contribute to "this translational gap between knowledge and clinical application" including an evidence deficit to support the use of some genetic tests, lack of clinical education and decision support for health care providers, and inflated expectations of the clinical benefit of GM, particularly in managing chronic complex diseases. She suggests using the principles of implementation science "which focuses on identifying and overcoming barriers associated with deploying and tailoring new interventions" as a means to address the gap between testing capability and practice, in those cases where evidence of utility is clear. (Burke and Korngiebel, 2015)

Our findings suggest that PCPs are open to changes in practice to facilitate GM. Over half our respondents thought that a telephone/ fax/email helpline to a local genetic counselor or a "buddy system" where a designated geneticist was available to answer questions, would help them integrate GM into their practices. There is an emerging literature exploring how this might happen. (Battista et al., 2012; Houwink et al., 2013; Manolio et al., 2013; David et al., 2015) One such model used tailored genetics education outreach delivered by a genetic counselor to general practices over 1 year, including genetic update sessions, a responsive advice service, and referral guidelines. This service was evaluated positively by participants with continued utilization of the genetic counselor for advice following completion. (Drury et al., 2007) This type of model requires clinician acceptance and "reconfiguration of professional roles and responsibilities." (Battista et al., 2012) Interestingly, the idea of a FP or nurse with a special interest in genetics in the clinic or a visiting genetic counselor to consult in the practice was less popular among our respondents. This may be due to the relative rarity of genetic conditions in primary care. Access to a genetics specialist has been positively associated with use of genetic testing for disease diagnosis or susceptibility, however many PCPs report they do not have access to genetics expertise. (Haga et al., 2013) It may be as Haga postulates that "access for some PCPs may be effectively limited if they are unfamiliar with these experts or have not had any clinical occasion to consult them." Perhaps there is a role for counseling by phone, telemedicine or electronic consultation to enhance communication and contact. (Haga et al., 2013) As a result of this study, we developed a website containing evidence-based resources, including point-ofcare tools, on GM for PCPs with clear information about how to access local genetic services (www.geneticseducation.ca). We are also exploring electronic consultation, questions directed to clinical geneticists by PCPs over a secure electronic platform, with response within 7–10 days, as a means for seeking clarification or guidance regarding clinical care in GM.

# LIMITATIONS

The main limitation to this study was the low response rate, bringing into question the generalizability of the results. Compared to the 2013 National Physician Survey in Canada (closest in time to the study), our study respondents were of similar age (median age 51 this study, 52 National Physician Survey), higher proportion female (47%/40%), slightly lower EMR use (74%/78%), and similar likelihood to be paid through an alternative funding arrangement rather than fee for service (49%/51%). (College of Family Physicians of Canada, 2013) This implies some similarity of our sample to Canadian FPs. Study respondents were very similar in age distribution to non-respondents. This study had more female respondents than non-respondents (respondents 47% female, non-respondents 40% female). The random sample of 100 nonrespondents that we contacted in order to adjust our response rate was 39% female, similar to our overall non-responder rate. The age distribution of the sample of 100 non-respondents was similar to the overall non-respondents. The low response rate may have been due to the length of the survey, possibly suggesting that those with more interest or knowledge of GM completed the survey. If this is the case, our results raise even more questions regarding FPs' assessment of the clinical value of genetic tests and their readiness to incorporate GM into busy primary care practices. This study was conducted in one province in Canada, so its generalizability to PCPs in other countries is unknown.

# CONCLUSIONS

This study shows that FPs see a role for themselves in taking FH, identifying individuals with a genetic condition, making appropriate referrals and supporting patients following genetic test results. They continue to lack the knowledge and confidence in GM skills needed for practice, particularly in the emerging areas of GM. They are somewhat optimistic about the contribution GM may make to patient care, but express caution about its current clinical benefits. Our study suggests that there is a need for more evidence of clinical utility of genetic tests, educational resources which can be integrated into primary care practice, clinical decision supports, and improved communication with genetic specialists. Resources need to include the basic skills for delivering GM (e.g. referral guidelines and testing criteria) as well as the advancing areas of pharmacogenetics, direct-to-consumer genetic testing, and whole genome sequencing.

# DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

#### ETHICS STATEMENT

The studies involving human participants were reviewed and approved by Children's Hospital of Eastern Ontario Research Ethics Board. Written informed consent for participation was not

## REFERENCES


required for this study in accordance with the national legislation and the institutional requirements.

## AUTHOR CONTRIBUTIONS

JC, JA, SM, FM, BW, JP, and DT substantially contributed to conception and design, analysis and interpretation of data, and drafting the article and gave final approval to the version to be published. SM and JP contributed to acquisition of data.

# FUNDING

This study was funded by Genetics Education Canada— Knowledge Organization (GEC-KO) which is supported by funding from the Children's Hospital of Eastern Ontario (CHEO).

and management of colorectal cancer. *J. Community Genet.* 5, 303–311. doi: 10.1007/s12687-014-0185-1


**Conflict of Interest:** DT is a consultant for mdbriefcase.com and for the Center for Effective Practice.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Carroll, Allanson, Morrison, Miller, Wilson, Permaul and Telner. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Using the Findings of a National Survey to Inform the Work of England's Genomics Education Programme

Siobhan Simpson, Anneke Seller and Michelle Bishop\*

Genomics Education Programme, Health Education England, Birmingham, United Kingdom

A national coordinated approach to workforce education and training in genomics is essential for the successful implementation of whole genome sequencing and, more broadly, genomic medicine within the National Health Service (NHS) in England. However, there have been no workforce wide assessments of genomics education and training needs that can be used to inform the strategic approach to be taken. In order to assess these needs the Genomics Education Programme (GEP) undertook a cross-professional training needs analysis. Responses from 2,814 individuals allowed the identification of four themes related to NHS staff's perceived education and training needs in genomics, those who: a) have a role in genomics and are competent; b) have a role in genomics but identified a specific learning need; c) could not identify whether genomics is relevant, but want to know more, and; d) do not see genomics as relevant to their role and do not believe they need to learn about it. Individuals are motivated to undertake training for their own continuing professional development and if they perceive training to have a direct impact on patient care. Overall, online learning is the preferred mode of delivery, but there are still many individuals who value face-to-face teaching. This paper demonstrates how the GEP has used these findings to provide an evidence base to inform the ongoing strategy for genomics education and training in the NHS, including the development of competency frameworks and a range of resources to address the diverse genomics learning needs of the healthcare workforce.

Keywords: training needs assessment, genomics education, genomic medicine, survey, education strategy, multidisciplinary

# INTRODUCTION

Genomics has been a focus within England's National Health Service (NHS) since the launch of the landmark 100,000 Genomes Project in 2012 (Couzin-Frankel, 2012). The information learned from this project is now informing the development and implementation of an England-wide NHS Genomic Medicine Service (NHS England, 2019b). This service will increase access to genomic testing across different specialties so clinicians can use this technology as part of the patient diagnosis, treatment, and management pathway.

#### Edited by:

Michael Dougherty, University of Colorado Denver, United States

#### Reviewed by:

Karen Hanson, ICF, United States Katherine Johansen Taber, Counsyl, United States

\*Correspondence: Michelle Bishop Michelle.Bishop@hee.nhs.uk

#### Specialty section:

This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics

Received: 03 July 2019 Accepted: 18 November 2019 Published: 17 December 2019

#### Citation:

Simpson S, Seller A and Bishop M (2019) Using the Findings of a National Survey to Inform the Work of England's Genomics Education Programme. Front. Genet. 10:1265. doi: 10.3389/fgene.2019.01265

**77**

Organizations responsible for the training and regulation of healthcare professionals in the UK recognize the impact of genomics in healthcare, and therefore the importance of genomics education and training for future healthcare staff. This is evidenced by the embedding of genomics into relevant professional standards and training programs (General Medical Council, 2018; Nursing and Midwifery Council, 2018; National School of Healthcare Science, 2019). However, with genomic testing now entering mainstream care (NHS England, 2019b), an understanding of the technology and the information these tests provide is needed by many of the NHS's current 1.4 million staff. The level of understanding required will differ depending on the role undertaken by the individual. This may range from an awareness of genomics and how genomics is used in their area of practice, through to specialist knowledge on testing, interpretation of results, and how genomics influences patient care and management. This poses a challenge: to provide appropriate education and training for the existing NHS workforce, across multiple professional groups in a rapidly changing field.

Health Education England (HEE) is responsible for improving the quality of patient care in the NHS through education, training, and development of NHS staff in England (www.hee.nhs.uk). The Genomics Education Programme (GEP), which sits within HEE, is the NHS in England's method of ensuring its staff have the knowledge, skills, and experience to ensure that the health service remains a world leader in genomic and precision medicine (www.genomicseducation.hee.nhs.uk).

To effectively provide education and training for the workforce, an understanding of the areas in which the workforce requires development is needed. Previous studies of genetics and genomics training needs of the healthcare workforce have identified gaps in knowledge and training but have tended to focus on a single workforce group rather than a whole healthcare system. Workforce groups such as physicians and nurses have been assessed, but often in specific areas—for example, with regards to direct to consumer testing, pharmacogenetics, or whole genome sequencing (Powell et al., 2012; Selkirk et al., 2013; Christensen et al., 2016), or particular specialisms—for example, obstetrics and gynecology, or dermatopathology (Adjei et al., 2017; Torre et al., 2018). There is acknowledgement, however, from those within other healthcare professions of a need for genomics education, but little direct evaluation of their training needs has been undertaken (Cornwall et al., 2018). Most studies have been conducted outside the UK or if done within the UK they have focused on a specific workforce group (Godino and Skirton, 2012).

While it is likely that these results are applicable across the NHS workforce it cannot be assumed they reflect the wider situation within England. As a first step in establishing a strategic approach to ensure all NHS staff can access genomic education and training that meets their learning and professional needs, the GEP undertook a cross-professional training needs analysis to identify genomic learning requirements across this large and diverse group.

#### METHODS

#### NHS Workforce and Genomic Medicine Centres

#### NHS Workforce

The NHS workforce is large and diverse, with 1,390,849 people employed by NHS England (NHS Digital, 2018). Of these 11.5% are doctors (n=160,135), 23.0% have a nursing qualification (n = 320,324), 1.9% are midwives (n = 25,866), and 1.6% are ambulance staff (n = 22,245). In addition, 11.5% are classified as scientific, therapeutic, and technical staff (n = 159,674). The remaining 50.5% of NHS staff have roles supporting clinical staff and in auxiliary services such as the operational and infrastructure side of the NHS (NHS Digital, 2018).

#### Genomic Medicine Centres

Thirteen Genomic Medicine Centres (GMCs) were established by NHS England between 2014 and 2015 to support the delivery of the 100,000 Genomes Project. These Centres covered all geographical areas of England (see Supplementary Material) to ensure equitable access to the project for eligible NHS patients (Genomics England, 2018). Within each of the GMCs, an Education and Training Lead was appointed to facilitate local workforce development in genomics, both within and outside of the GMC. The GEP provided financial support and oversight of the education and training activities within each GMC.

#### Data Collection

To inform regional and national strategies for NHS workforce development in genomics, the Education and Training Lead in each GMC was tasked to develop a questionnaire to identify local requirements. The GEP was informed that NHS ethics approval was not required as the purpose of these surveys was service evaluation. Handling of data was carried out within the governance framework of each organization. General guidance on the purpose and structure of the questionnaire was provided by the GEP, but the GEP did not directly design or deploy any questionnaires. Thus, each of the GMC regions developed their own questionnaire enabling different service requirements to be addressed within the surveys. Questionnaires were entered into either Survey Monkey or Bristol Online Survey system. Electronic links to the surveys were deployed through different communication networks available to the Education and Training leads within their regions, such as hospital trust intranets and mailing lists. Where possible, reminders were sent. Due to the different methods in which the surveys were deployed, it is not possible to determine the number of NHS staff who received the link to the online surveys. Data collection occurred between July 2016 and April 2017. Two questionnaires targeted specific workforce groups (West Midlands and Yorkshire and Humber) as these were considered workforce development priority areas for these regions, while the other questionnaires were aimed at the NHS workforce more generally. An exemplar questionnaire is available from the authors on request.

#### Measures

Each questionnaire had between 9 and 20 questions. Here we present the findings related to questions asking about perceived education and training needs, training delivery preferences, and motivations to undertake training.

Demographics, including involvement in genomics, were collected using closed questions [for example: "are you involved in the 100,000 Genomes Project" (Yes/No/Don't know), "apart from the 100,000 Genomes Project, do you currently have a role in delivering any genetics/genomic services (Yes/No)"]. Two questionnaires asked about use of genomics in current practice by asking a series of statements: "are you currently using genomics in your clinical practice for prevention/diagnosis/treatment/No/Not applicable for my role." Another questionnaire asked more specific questions around involvement with "genetic testing" (Yes/No), "discussion of genomics or molecular diagnostics at MDT" (Yes/No), and "processing samples for 100,000 Genomes Project" (Yes/No). Previous training in genetics and genomics was asked by four of the questionnaires by asking "Have you had any previous training in genetics and/or genomics?" followed by a list where respondents could tick as many as applied.

Education and training needs: Perceived knowledge and skill gaps were asked in three different ways: "Do you feel you have sufficient knowledge and the skills to perform your current role in genetics/genomics?" (Yes/No/My role does not involve genetics/genomics); "Do you feel you have sufficient knowledge in genomics to allow you to do your job effectively?" (Yes/No); "Do you feel that you need further training in genomics?" (Yes/No).

Training delivery preferences: Five of the questionnaires asked, "How would you like training to be delivered?" followed by a list of options, with respondents able to tick as many as applied. Three questionnaires asked follow-on questions to the primary question about perceived education and training needs, to ask respondents to specify how they would like education and training to be delivered with a list of options provided.

Training motivation: Four questionnaires asked respondents "what motivates you to undertake education and training" with a list of options. Another questionnaire asked the same question but left this as a free-text response.

All questionnaires provided the option for free-text responses throughout to clarify or comment on their responses. In addition, four questionnaires also provided the opportunity for respondents to provide any closing remarks before exiting the questionnaire.

#### Data Analysis

Data from the questionnaires were downloaded, anonymized, and sent to the GEP in an Excel format. Quantitative data from each questionnaire were analyzed separately. Descriptive statistics were used to describe the sample in terms of their professional workforce group, previous genomics education, and their perceived education and training needs in genomics. The responses to the question asking about education and training delivery methods were coded as "face-to-face," "online," or "both." For statistical analysis only individuals who expressed a preference for one or the other (as opposed to "both") were analyzed. For four of the five questionnaires that asked about training motivations, descriptive statistics were used to describe the sample. For the fifth questionnaire, free text responses were coded to the categories used in the other questionnaires. Where possible, Kruskal-Wallis tests, with appropriate post hoc testing, were performed to determine an association between professional workforce groups and education and training needs, preferred education and training delivery methods, as well as motivation to participate in education and training. Thematic analysis of the free-text comments made throughout the questionnaires was conducted using a constant comparison approach as first described by Glaser and Strauss (1967).

# RESULTS

A total of 2,814 responses were received from eight questionnaires (covering nine GMCs), representing 10 workforce groups (see Table 1 for a description of the workforce groups). These workforce groups included clinical and non-clinical roles, as well as "other" individuals such as hospital chaplains, housekeepers, and librarians. Most responses were received from medical professionals (34.4%), with the least (less than 1%) from the public health workforce. Overall 880 (31.3%) respondents indicated they were currently involved in the delivery of genetic and/or genomic services, including the 100,000 Genomes Project (Table 2). Of those respondents asked about their previous education and training in genomics (n = 1625), 322 (19.8%) had no previous genomics education and training, 674 (41.5%) had undertaken CPD, 474 (29.2%) had genomics education as part of a non-specialized degree (e.g. undergraduate medical degree), and 155 (9.5%) had obtained a specialized genomics degree.

# Identifying Learning Needs

Not all respondents who competed the questionnaires stated that they needed genomics education and training. Table 3 outlines the results for each questionnaire. For the questionnaires that asked if respondents had sufficient knowledge in order to perform their role, between 5.1% and 40.8% replied no, indicating they needed further training. Conversely in those questionnaires that asked if they felt they needed further training in genomics, between 75.9% and 85.7% responded yes, they did need further training.

There were no significant differences in perceived need for further training between the workforce groups within each questionnaire with two exceptions: Oxford (Kruskal-Wallis p < 0.01) and Greater Manchester (Kruskal-Wallis p < 0.001). For the respondents from Oxford the significant test result is due to the difference between the nurses, midwives, and associated roles

#### TABLE 1 | Definitions of workforce groups.


group (41.2% state sufficient knowledge) and the Healthcare scientists group (78.6% state sufficient knowledge) (Dunn's pairwise tests p < 0.001, adjusted using the Bonferroni correction). In the respondents from Greater Manchester the difference (Dunn's pairwise tests p < 0.01, adjusted using the Bonferroni correction) is between the Administration and clerical group and all other groups. Only 54.2% of the Administration and clerical group indicated that they would like more training, while the other groups were all over 82.8%. Neither involvement in delivering genetic/genomic services or the level of previous education and training were significantly associated with reported education and training need across the questionnaires.

Analysis of the free-text comments in each of the questionnaires identified four themes relating to NHS staff's education and training needs.

A. Individuals have a role in genomics and are competent. These individuals felt they had enough knowledge and the right skills to perform their current role; however, respondents were cognizant that genomic knowledge constantly evolves, and, as stated by one respondent:

#### "There's always so much to learn" (Nurse, Pediatrics).

There was also the recognition from some of these respondents that they were making a self-assessment of their competence and, as such, may not have all the knowledge and skills they need. As one medical professional commented:

#### "But I might be unconsciously incompetent" (Medical consultant, Immunology).

B. Individuals have a role in genomics and identified a specific learning need. While many of the learning needs quoted by respondents related to very niche areas of knowledge and specific skills, three common areas were identified:


C. Individuals could not identify whether genomics is relevant to their practice but want to know how genomics may impact on their clinical role. Some of these respondents were aware that genomics would be relevant to their professional group, whereas others were not sure. However, both groups still wanted to find out more about the application of genomics to healthcare. In general, these respondents requested introductory level resources, primarily related to their professional group such as "genomics for nurses" and the "application with respect to radiology."

D. Individuals do not see genomics as relevant to their role and do not believe there is a need to learn about it. These NHS staff were not interested in knowing more about genomics, as they could not see how it would change their every-day practice.

"Do I need to know more? I can do my job without having any knowledge in genomics" (Nurse, Intensive care)

However, it is likely that some of these responders will need some level of genomics knowledge, as genomics is being used in the clinical area in which they work (e.g., maternity, cardiology, pediatrics, etc.).

As the free-text questions were optional, counting the responses would not have provided a reliable indication of the proportion of healthcare professionals within each category.

#### Challenges to Identifying a Learning Need

Analysis of the questionnaire comments also highlighted elements that made identifying genomic learning needs challenging. For some respondents, their lack of knowledge about genomics itself meant that they did not know if this was a topic they should know more about.

"I honestly don't know, I have no idea what it is" (Nurse, Anesthetics)

"Not familiar with the term Genomics" (Medical Consultant, Gynecology)


Others were quite skeptical on the impact of genomics, so questioned the relevance and the need for education and training in this area.

"If the outcome is to tell patients to do anything other than lose weight, exercise and stop smoking and drinking, I will be astonished" (General Practitioner)

For others, in particular those who responded to surveys where the question about education and training was directly linked to their current practice, a lack of clarity about their role made answering this question difficult.

#### Training Delivery Approaches

All surveys (n = 2,814 respondents) asked a question around preferred method of learning. There were respondents in all workforce groups who were receptive to both online and face-toface modes of delivery. Of those who indicated a preference, there was a significant preference (Kruskal-Wallis p < 0.001) for online learning (n = 861) over face-to-face learning (n = 653). The remaining respondents (n = 1025) indicated that they were receptive to both types of learning. There were no significant differences between workforce groups in preferred training delivery methods.

Several respondents provided comments in the questionnaire about barriers to accessing continuing professional development (CPD) opportunities. The most common theme was a lack of protected time to participate in CPD.

"I am using my annual leave to do my further training in genomics as the (hospital) does not provide any training or allow study leave for this reason" (Junior Doctor, Foundation year training)

"If spaces are made available … there is no capacity within the (hospital) to allow time to train—understaffing, under resourced, plus not enough study days" (Healthcare Scientist, Genomics).

In some cases, this appeared to pertain to accessing protected time to access online courses.

"Can't get study leave for online learning" (Medical Consultant, Pediatrics)

A number of respondents also raised the issue of a lack of funding to pay for the education or training session.

"I like the idea of learning more, but I don't have the time, energy or funds" (Clinical Researcher)

Five surveys (n = 1,786 respondents) also provided a list of reasons that may motivate individuals to undertake training: continuing professional development (84.6%, n = 1,511) and direct impact on patient care (71.8%, n = 1,283) were the reasons most often cited. There were no significant differences between workforce groups.



#### DISCUSSION

This paper reports the perceived education and training needs in genomics of England's NHS staff, the largest assessment of this workforce to date. The aim of this work was to collect data from NHS staff that could be used to direct the work of local education and training initiatives and that of the GEP. As with all surveys there is the potential for response bias. Due to the nature of how these surveys were deployed there will be a level of response bias, with people with a vested interest in the subject more likely to respond (Duda and Nobile, 2010). However, views have been collected from a diverse group of staff, not all of whom were familiar with genomics or used genomics within their current role.

Not all respondents identified a need for genomics education and training, but the proportion who expressed a need differed depending on how the question was asked. When asked if they have sufficient genomics knowledge and skills to perform their current role, the proportion of respondents who responded "no," therefore indicating a need for education and training, was much lower than when a general question was asked about engaging in genomics education and training activities. These responses suggest that there is an appetite for genomics education and training initiatives within the NHS, even if this knowledge and/or the skills are not yet required by an individual to undertake their job role. In most cases there was no significant difference observed between the different workforce groups and their perceived education and training needs, and perceived need was not significantly influenced by previous education and training.

The identification of the four different types of genomic education and training needs provides a framework in which to segment the NHS workforce on their learning requirements rather than their workforce group. Each segment of the workforce has differing requirements.


information about genomics so they can identify how this technology impacts on their role, and the patients that they care for.

• Those who do not see genomics as being relevant to their role, and so do not think there is a need to learn about it. While some NHS staff in this group may not require an understanding of genomics to perform their role, others will. This second group is likely to be the most challenging group to reach as they will need persuading of the relevance of genomics to their work before they will engage in any relevant learning.

#### Informing Genomic Education and Training Resource Development

For those NHS staff that need to understand genomics and apply this to their practice, our findings suggest there are two levels of education and training resources required. The first is general information targeted to professional groups and the second is cross-professional resources on specific areas or activities that form part of the clinical pathway. However, the results from these surveys also emphasize the need for ongoing awareness raising about genomics in general, as there are still healthcare professionals, as well non-clinical NHS staff, who do not know what genomics is, let alone how it can be applied to healthcare.

These findings have influenced the development of GEP resources, addressing both levels of education and training requirements, as well as general awareness, with innovative ways to engage and inform our audiences, ranging from videos and animations to formal qualifications (for example, Master's level). Figure 1 demonstrates how key messages from each of the themes have guided GEP activities and outputs. Resources targeting specific professional groups highlighting where and how genomics is relevant in these clinical areas have been produced (www.genomicseducation.hee.nhs.uk/genomics-inhealthcare/). Cross-professional education and training resources corresponding to clinical activities across the patient pathways in the new Genomic Medicine Service are also in development. In addition to delivering education and training resources, the GEP has initiated the development of crossprofessional competencies. Work has commenced on defining these competencies for the clinical activities of the consent conversation and feedback of genomic test results. These

competencies can direct future work of the GEP, by prioritizing resource development, and they can support individual NHS staff by providing a framework that they can use to identify learning or training gaps (Hepp et al., 2015).

The importance of providing the NHS workforce with these two levels of education and training resources has also been recognized at a policy level. The Interim NHS People Plan, which sets out how people working in the NHS will deliver the ambitious 10-year vision for healthcare in England, signals the need for a NHS workforce that has education and training "tailored to the needs of the individual" and with a balance of general knowledge and specialist skills depending on the clinical role (NHS England, 2019a).

# Supporting NHS Staff to Engage in Education and Training

NHS staff overall showed a significant preference for online delivery; however, it is important to note that many respondents still preferred face-to-face education and training. It is unclear from our results if individual's preference is due to personal learning styles or more pragmatic reasons such as their ability to access to learning. It is recognized that some training, such as learning practical skills, including laboratory science, may be best delivered face-to-face (Jaggars, 2014). However, there are times when online learning is equally or more effective than face-toface delivery and often has the added advantage of being flexible, allowing learners to access learning opportunities at a time and place that suits them (Maloney et al., 2015; Brady et al., 2018).

Providing different modes of delivery allows individuals to choose the method that best serves them, either in terms of learning style or time and convenience, but this may not always be possible. In the case of genomics education within the NHS, the scale and pace at which education and training needs to occur often makes online learning the most practical choice for those developing resources. While there is recognition at a national level that access to continuing professional development is a priority for the NHS (NHS England, 2019a), our findings suggest people are becoming less willing to do CPD in their own time. Concessions will therefore be needed to be made to ensure the same consideration for protected learning time is given for those wanting to participate in online learning rather than face-toface sessions.

#### Understanding Motivations to Engage in Learning and Applying This to Resource Design

Understanding training motivations can help ensure education and training courses and resources are appropriately marketed to the audience. However, an individual's education or training motivation can also influence the depth to which they will learn. Training motivators can be considered intrinsic or extrinsic, but these are not mutually exclusive. Individuals primarily motivated by intrinsic factors are likely to be deep learners, while individuals motivated by extrinsic factors are typically surface learners (Baeten et al., 2010). As an educator, understanding target audience's motivations can help tailor content to maximize learning. For example, individuals who are undertaking training purely to meet CPD requirements are likely to be, at least initially, less engaged surface learners, learning what they need to pass, compared to individuals who are undertaking training because they are motivated by intrinsic factors such as "direct impact on patient care".

While meeting CPD requirements was the main motivator of our respondents, there were many NHS staff who identified "direct impact on patient care," an intrinsic factor, as a primary motivation to engage in learning. This suggests this proportion of the workforce will be deep learners if they can see how learning will benefit their patients. Understanding these two factors has influenced the way in which the GEP develops its resources. Where relevant, education and training activities are accredited with relevant bodies as recognized CPD activities. In addition, the GEP ensures that the link between the learning activity and patient care is a central component in resource development.

#### CONCLUSIONS

The findings from these surveys have provided an evidence base that informs the ongoing strategy for the GEP. This study demonstrates how a questionnaire-based needs assessment can provide information to direct the development of relevant resources to meet the education and training needs of a diverse health professional workforce.

The development of evidence-based competency frameworks and educational resources by the GEP to support all NHS staff who will use genomics as part of their role in the patient pathway will result in a workforce better placed to take advantage of advances in genomic medicine.

# DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

# ETHICS STATEMENT

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and

### REFERENCES


institutional requirements. The patients/participants provided their written informed consent to participate in this study.

# AUTHOR CONTRIBUTIONS

MB conceived the idea for publication. MB and SS had intellectual input into the study design. MB and SS contributed to data analysis. MB, SS, and AS provided intellectual input into preparation of the manuscript. All authors approved the final version and agree to be accountable for all aspects of the work.

#### FUNDING

This work was supported by Health Education England's Genomics Education Programme.

#### ACKNOWLEDGEMENTS

We gratefully acknowledge the following Education and Training Leads for access to their raw data: Gemma Chandratillake, East of England GMC; Jenny Hiley, West Midlands GMC; Jennifer Whitfield, Oxford GMC; Mel Watson, West of England GMC; Leigh Jackson, South West GMC; Glenda Beaman, Greater Manchester GMC; Lisa Dowell, North West Coast GMC; Kate Tatton-Brown, South London GMC; and Julie Atkey, Yorkshire and Humber GMC.

# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fgene. 2019.01265/full#supplementary-material

SUPPLEMENTARY FIGURE 1 | Map showing approximate geographical regions covered by each GMC.


Conflict of Interest: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2019 Simpson, Seller and Bishop. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# The African Genomic Medicine Training Initiative (AGMT): Showcasing a Community and Framework Driven Genomic Medicine Training for Nurses in Africa

#### Edited by:

Sylvia Ann Metcalfe, Murdoch Childrens Research Institute (MCRI), Australia

#### Reviewed by:

Christine Patch, Genomics England, United Kingdom Ignacio Macpherson, Universitat Internacional de Catalunya, Spain Vajira Harshadeva Weerabaddana Dissanayake, University of Colombo, Sri Lanka

#### \*Correspondence:

Nicola Mulder nicola.mulder@uct.ac.za Victoria Nembaware vnembaware@gmail.com

†Full list of contributing members is available in Appendix

#### Specialty section:

This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics

Received: 07 August 2019 Accepted: 01 November 2019 Published: 20 December 2019

#### Citation:

Nembaware V, African Genomic Medicine Training Initiative and Mulder N (2019) The African Genomic Medicine Training Initiative (AGMT): Showcasing a Community and Framework Driven Genomic Medicine Training for Nurses in Africa. Front. Genet. 10:1209. doi: 10.3389/fgene.2019.01209

*Victoria Nembaware1\*, African Genomic Medicine Training Initiative† and Nicola Mulder2\**

1 Division of Human Genetics, Faculty of Health Sciences, University of Cape Town, Cape Town, South Africa, 2 Computational Biology Division, Department of Integrative Biomedical Sciences, IDM, Faculty of Health Sciences, University of Cape Town, Cape Town, South Africa

The potential of genomic medicine in improving the quality of healthcare both at population and individual-level is well-recognized globally. However, successful adoption of genetic and genomic evidence into clinical practice depends on training the healthcare workforce and clinical researchers in genomic medicine. Due to limited expertise in the medical genetics and genomics field, widespread uptake largely depends on task-shifting for the implementation of genomic medicine implementation to key healthcare professionals such as nurses. Their knowledge would be developed through courses aimed at professional development. Globally, trainers, and training initiatives in genomic medicine are in early stages of development, but resource limited settings such as the African continent face additional logistical and institutional challenges. The African Genomic Medicine Training (AGMT) Initiative was conceived during a combined conference of the African Society of Human Genetics (AfSHG) and the Human Heredity and Health in Africa Consortium (H3Africa) in 2016, Senegal, in response to the needs for developing knowledge and skills in genomic medicine. AGMT was established to implement a sustainable genomic medicine training initiative primarily for healthcare professionals who are not geneticists but are nurses, doctors, and pharmacists in Africa. This paper reports on the establishment of the AGMT initiative and the strategies developed and piloted by this initiative in designing and implementing an accredited frame-work and community-based blended learning course for nurses across 11 African countries. The global implementation experiences, outcomes and lessons learnt are highlighted. The AGMT initiative strategy takes advantage of existing research consortia and networks to train and create a pool of trainers and has adopted evidence-based approaches to guide curriculum and content development/adaptation. This initiative established the first Africa-wide online blended learning genomic medicine course which forms the basis from which to develop courses for other healthcare professionals and the wider public.

Keywords: genomic medicine, Africa, precision medicine, training, nurses, competencies, Kern's six step model

# BACKGROUND

Historically, knowledge translation of genomic knowledge for healthcare in Africa has been challenged by the dearth of genomic data from people of recent African origin (Popejoy and Fullerton, 2016). However, recent initiatives such as the Human, Heredity, and Health in Africa Consortium (H3Africa) (Dandara et al., 2014a; Rotimi et al., 2014), H3ABioNet (Mulder et al., 2016), and MalariaGen (Achidi et al., 2008) aim to build capacity for genomics research in Africa, and are challenging the existing norms (Gurdasani et al., 2015). Cumulating results from genomic projects of human health and disease projects are helping explain African-specific susceptibility and variability in disease severity to conditions such as kidney-related diseases (Cooper et al., 2017) and sickle cell disease (SCD) (Pule et al., 2015). The application of genomics information in optimizing treatment has given rise to development of pharmacogenomics-based dosing of antiretroviral therapy (ART) (Dandara et al., 2014b; Skelton et al., 2014). Large-scale genomic characterization of African populations holds great promise for identification of additional health-linked genetic variants relevant to the understanding of possible genomic drivers of the high burden of infectious diseases and the growing prevalence of noncommunicable diseases in Africa (Wonkam and Mayosi, 2014; Tekola-Ayele and Rotimi, 2015). Therefore, in anticipation of the changing African genomic landscape, there is an urgent need for strategies to translate this genomic knowledge into clinical practice and augment clinical decisions.

Efforts to translate genomics into clinical practice face a number of barriers which include limited resources to sequence and characterize human genomes from African populations. In addition, there is lack of access to next generation technologies and analytical capabilities in Africa and, policies that are silent on genetics and genomics and medical curricula that are not adequate in teaching genetics/genomics concepts, leading to limited appreciation of the utility of genomic knowledge in healthcare (Wonkam et al., 2006; Muzoriana et al., 2017). In addition, African countries lack the critical mass of experts in genetics, genomics, data science, and bioinformatics required to implement country-specific genomic medicine driven healthcare. For example, South Africa is the only African country with a critical mass of skilled genetic counsellors and medical geneticists, key personnel for the implementation of genomic medicine (Abacan et al., 2019).Therefore, cost-effective strategies are urgently required to promote incorporation of genomics research and findings in healthcare in Africa for quality health outcomes. Most of the developed world is moving to adopting genomics in its health care programs, however, Africa is still lagging behind, a trend that will continue to widen existing health disparities between developed and developing countries.

Globally, several introductory genomic medicine courses have been developed and implemented, tailored for specific healthcare workers such as nurses (Nembaware et al., 2016), and other healthcare professionals (https://www.genomicseducation.hee. nhs.uk). However, several of the genomic medicine curricula are characterized by numerous shortcomings, which include limited development of competencies in a systematic manner. Mapping and alignment of curricula and competencies are slowly being integrated into genomics curricula development (Jenkins et al., 2015). Competencies in genomics and genetics for nurses are publicly available online (Jenkins and Calzone, 2007; Kirk et al., 2014), a noteworthy competency resource was developed by the Inter-Society Coordinating Committee for Physician Education in Genomics (ISCC) based on five "Entrustable Professional Activities" EPAs (Korf et al., 2014). Another short-coming of most existing genomic medicine curricula is the limited application of well-established curriculum development frameworks such as the Kern's six step model (Kern and Thomas, 2009). This model is a widely used systematic curriculum development approach which links curricula to healthcare needs and promotes continual curriculum monitoring and evaluation (Khamis et al., 2016). This model has the added advantage of being adaptable to suit the needs of the implementers (Khamis et al., 2016).

The existing publicly available genomic medicine curricula require tailoring of competencies and content for the African context due to the continent's diverse cultures, disease burdens, and healthcare facilities and resources. In addition, reported challenges from an informal online survey in training genetics and genomics highlight lack of expertise, and lack of resources and funds (https:// training.h3abionet.org/AGMC\_2016/outputs/). To address the highlighted training needs and establish a foundation for genomic medicine in the region, the African Genomic Medicine Training Initiative (AGMT) was initiated to pool expertise and resources from across Africa to develop a training program for African healthcare professionals, which could be further tailored across the diverse countries. The goals of the AGMT initiative are to:


This article focuses only on the curriculum development objective of the AGMT and outlines the steps taken in the development and implementation of the genomic medicine curriculum, firstly targeted at nurses in Africa. The article demonstrates how the AGMT initiative adapted the Kern's sixstep model for the development of a medical curriculum. In addition, the Kern's six-step model was modified to incorporate a competency mapping approach developed by the International Society of Computational Bioinformatics (ISCB) (Mulder et al., 2018). Formal medical educational programs have aims and goals that are often not clearly articulated and, in some instances, poorly understood by key constituents inside and outside of the formal education system (Kern and Thomas, 2009). Using a model/framework to develop the curriculum helps clarify aims and objectives around which the curriculum is structured (Kern and Thomas, 2009). The curriculum becomes the official documentation that includes the goals of teaching and learning; the instructional methods and materials as well as the assessment. The curriculum reflects the envisaged aspirations of society as well as the curriculum that is ultimately implemented. This helps the newly trained medical educators meet the needs of their students, patients, and other key stakeholders. The use of a framework to guide the development of genomic medicine training for Africa also presents an opportunity to implement formal evaluations and studies to share lessons and to learn from.

# DESCRIPTION

A workshop aimed at establishing AGMT initiative was conducted during a combined conference of the African Society for Human Genetics and H3Africa Consortium in Senegal, May 2016, and was attended by over 80 participants. The workshop was used to plan and initiate the Kern's six-step approach for designing medical education curricula, which guided development of the AGMT nurse curriculum. Steps included conducting a general needs assessment, followed by specific needs of targeted learners, defining goals and objectives, determining the educational strategies, planning the implementation, and developing an evaluation plan (Kern and Thomas, 2009). The specific needs of targeted learners and defining goals and objectives of the training was guided by the competency mapping strategy from the ISCB (Mulder et al., 2018). Key results from each step are outlined below:

#### Step 1: General Needs Assessment

During the AGMT establishment workshop, general needs for new approaches which promote community-based genomic medicine training in Africa were solicited and deliberated by the members present. Data and information generated from this workshop provided the foundation of a survey which was conducted online and advertised through various mailing lists. The survey was conducted to solicit gaps and needs in genomic medicine training from a broad representation of 33 stakeholders and from 19 African countries (https://training. h3abionet.org/AGMC\_2016/wp-content/uploads/2017/01/ TrainingSurveyAfrica-Upload.pdf). In addition, monthly planning meetings were held to refine the training strategy, develop the curriculum and map competencies, and plan and implement the pilot.

From this multiapproach general needs assessment, several gaps were identified which included: limited and not up-to-date curriculum content, lack of expertise in training in genomic medicine relevant fields such as genomics and genetics; and lack of training resources at the various institutes and limited funding. To address these challenges and gaps, the strategy included short courses, which may be developed further into diploma level content, graduate and postgraduate programs for healthcare and research professionals. Training would also target patients, especially those who plan on being advocates in the genetics and genomics fields. Public engagement activities could also be implemented to align with the training developed for healthcare professionals and patients. **Figure 1** illustrates some of the key trainees AGMT could target in the near future. In addition to healthcare workers, it is important to engage/train patients and the general public in genomic medicine. A website (https://training. h3abionet.org/AGMC\_2016/) and mailing list were created to facilitate seamless communication. This was made possible

through support from the H3Africa Consortium's H3ABioNet, a pan-African bioinformatics network which focuses on genomics capacity development (http://h3abionet.org).

# Step 2: Needs Assessment of Targeted Learners

This step aims to embed the specific training needs of targeted learners and identify specific stakeholders for the curriculum development, implementation, and evaluation. AGMT engaged with the nurse professors/lecturers, recent graduates, and the Global Research Nurses forum to identify specific needs of the nursing community. An article was published on the Global Health Nurses portal (https://globalresearchnurses.tghn.org/ articles/preparing-genomic-medicine-nurse-training-africa/) which summarized existing nurse training and highlighted the role of nurses in Africa. In a similar fashion to the ISCB strategy developed for a Bioinformatics curriculum, four nurse personas were created to make explicit the nurses' current roles and their training needs and targeted outcomes in genomic medicine (**Table 1**). Nurses at different professional levels developed the initial personas and their roles based on the common nurse specializations in Africa. We had a professor in nursing who has worked and trained nurses from several different African countries. The team agreed to work with four personas as they were a convenient number that was perceived to be sufficient to capture common nurse roles in different contexts across the continent. Although the effectiveness of the four personas in informing the curriculum might need to be probed the future, it is believed that convenient sampling used, that is small in scale and purposively selected on the basis of crucial criteria, was deemed appropriate by the team and allowed us to focus on our purpose (Punch, 2005). Based on roles of nurses highlighted by the nurse personas and feedback from nurses it became clear that the course needed to emphasize practical application of content into students' current settings and roles using problembased learning with clinical case studies relevant to Africa. This strategy was critical to highlight the relevance of the course to current clinical practice and increase uptake. In addition, skills in genetic counseling, community engagement, ethical conduct in research, inclusion of genetics and genomics in patient care, and development of health promotion material which included relevant genetics/genomic material were also found to be required. There was also a need to address stigmas and misconceptions of genetics and genomics commonly found in African communities.

# Step 3: Goals and Specific Objectives for the Training Course

In general, the course aims to support improved genetics and genomics knowledge, attitudes and skills for: research nurses in the biomedical field or those aspiring to be research nurses; specialist nurses working in the genomics/genetics field and general nurse practitioners in their day to day duties, or recent graduates. The overall objectives for the nurse personas were to develop and implement a plan of care for patients that incorporates genetic and genomics knowledge and is sensitive

#### TABLE 1 | Personas used to create and map competencies.

#### Getrude – Research Nurse at the University of Malawi

Getrude is 29 years old. She holds a Diploma in Nursing and has 5 years' experience. She is registered with the Malawi Nursing Council. She was recently recruited as a research nurse in a clinical study being conducted at the Malawi Medical College of Medicine and Nursing. Her current duties include:


#### Melody – Senior Midwife HIV Specialist: Malawi

Melody is 40 years old and holds a Senior Nurse Position in a district hospital. She holds a Bachelor degree in Nursing with two postgraduate diplomas in midwifery and HIV & AIDS Care. She is registered with the South African Nursing Council (SANC). She manages the day-to-day nursing operations within the midwifery department. She has 8 years experience in nursing. Her role covers the areas below:


#### Duties include:

Coordination of patient care.

Patient consultation, counselling and recommendation of treatment plans this includes the following clients:

	- pregnant women
	- those with severe drug responses
	- Providing consultation and advice to other nurses as a specialist practitioner
	- Individual and team supervision.
	- Ensure adherence of the unit to hospital and government policies and guidelines as they relate to nursing procedures, standards and practices, administrative and budgetary management.
	- Working in collaboration with other healthcare professionals when they are available.

#### Douglas - Community Health Nurse: Nigeria


(Continued)

#### TABLE 1 | Continued


#### Erensia - General Nurse, Stellenbosch, South Africa

She holds a Bachelor degree in Nursing. She is registered with the South African Nursing Council as a Nurse (general, community, psychiatry) and midwife. She is currently working in an adult medical ward.

	- Conducts individualized patient assessment, prioritizing data collection based on the adult or elderly patient's immediate condition or needs within time frame specified by governing policies, procedures or protocols.
	- Develops individualized plans of care patients reflecting on collaborations with other members of the healthcare team.
	- Performs appropriate treatments as ordered by physicians in an accurate and timely manner.
	- Performs therapeutic nursing interventions as established by individualized plan of care for the adult or elderly patient and his/her family, taking into account the patient's family history.
	- Provides individualized patient/family education customized to the adult or elderly patient and his/her family.
	- Documents patient assessment findings, physical/psychosocial responses to nursing intervention and progress towards problem resolution.
	- Initiates emergency resuscitative measures according to adult resuscitation protocols.
	- Maintains confidentiality in matters related to patient, family and healthcare staff.
	- Provides care in a non-judgmental, non-discriminatory manner that is sensitive to the adult or elderly patient's and family's diversity, preserving their autonomy, dignity and rights.
	- Reports patient condition to the multidisciplinary team during each shift.
	- Maintains current competency in General Nursing
	- Keeps up to date with current research evidence in order to change policies and procedures to improve healthcare outcomes

to individual and cultural preferences, practices and norms by offering basic genetic counseling to patients and families, and conducting genomics research that is ethical and appropriate to the nurses' context.

Competencies were adapted from the ISCC competency portal (https://genomicseducation.net/competency) to suit the needs of the African continent and these were mapped to the nurse personas in one face to face workshop, Google documents, and several online meetings. The mapping of ISCC to the AGMT nurse competencies were not retained due to numerous rounds of editing. The AGMT competency mapping team was split into three groups to review the personas and map competencies. Two personas were reviewed by two groups only instead of three. Consensus was agreed during monthly meetings, after face to face discussions and *via* Google docs. Once the targeted competencies had been established, the Bloom's taxonomy was used to determine the most appropriate level for a specific nurse by several competency mapping teams (see **Table 2**) for each persona. The final recommended competency to target is indicated in the last column in **Table 2** and was arrived upon after the three competency mapping teams (each team's competency level mapping is colour coded in **Table 2**) had reached a consensus.

# Step 4: Educational Strategies

This step involves planning the content to be taught and the educational methods to be used. Content was mainly adapted from a genomic medicine curriculum developed by Health Education England to upskill United Kingdom's National Health Service healthcare professionals, in readiness for the implementation of genomic approaches through the 100k Genomes project (https://www.genomicseducation.hee.nhs.uk/; https://www.genomicsengland.co.uk/about-genomics-england/ the-100000-genomes-project/). Assessments were adapted to align to the specific competencies identified in step 3. **Table 3** provides a brief description of the final course modules, full details are available on the AGMT website (https://training. h3abionet.org/AGMC\_2016/). The four modules varied in length depending on the number of classes they had. Each class was allocated 1 week with contact sessions which lasted around 2 h. Student centered approaches that encourages integration of prior and current experiences were deemed most suitable to facilitate learning for working adults. By selectively drawing on elements of problem-based and project-based learning this enabled the use of real-life questions, a challenge or problem as educational strategies to facilitate the development of knowledge (Lennon et al., 2019). Therefore, several case studies relevant to African health were sourced from the various working group members and embedded in the course material and class assessments. These types of teaching methods are often used for training of health-care professionals as they get students to engage with self-directed learning and offer opportunities for facilitation by the instructor (Kaufman and Holmes, 1996; Kaufman and Holmes 1998).

Learning activities such as quizzes, online discussions on the University of Cape Town's learning management system Vula (powered by Sakai), preclass exercises, and postclass assignments were developed in alignment with indicative content and competencies for each lesson. In addition, the range of learning activities was structured to enable students to actively engage and apply their knowledge and thus promote student-centered learning and flipped class learning (Goh and Ong, 2019). Furthermore, exercises and assessments were made relevant to the participants' context as it required participants to produce resources such as generate a list of genetics and genomics resources and services available at their institutes to make it easier for them to refer their patients.

Classes were required to submit a collaborative research project at the end of the course. which aimed to promote collaborative development of publishable research and reviews, and the assignments could be submitted to a special collection in a specific journal. However, classes were also free to choose not to publish their work or publish in a separate journal. The initial plan was for the course to run over six months, however the formatting of class projects into manuscripts continued after the course had concluded and this exercise varied across the different classes.

# Step 5: Implementation

Each classroom is managed by a facilitator who ensures the lectures are played, the class is linked up to the live sessions, and facilitates the interactive activities. We had a brief threeweek training for facilitators in three areas; online facilitation;

#### TABLE 2 | Suggested nurse competencies mapped to nurse personas.


Key: No competency – 0; Awareness – 1: Bloom (Knowlegde and Comprehension); Working knowledge – 2: Bloom (Application and Analysis); Specialist knowledge – 3;Bloom (Synthesis and Evaluation). There are multiple competence levels shown as between 2 and 3 different groups mapped the competencies. The last column in bold was the final competence level selected.

face to face group facilitation; and facilitation of role plays required for genetic counseling. Trainers were asked to develop slides based on the indicative content provided and the aligned competencies and assigned levels. This was a negotiated process.

An electronic advertisement was circulated *via* several mailing lists to advertise the course, see **Figure 2**. Almost 30 different sites applied to host a classroom for the course, however 19 classes were chosen based on their meeting the requirements to provide stable internet connectivity, a qualified Epidemiology

#### TABLE 3 | Summary of modules included in the training.

good enough, the study is published


facilitator in genetics and hardware that could handle hosting webinars. The coordinator advertised the training extensively *via* social media, the chosen class facilitators were also required to recruit participants in their areas/regions. The course was open to qualified and practising African-based nurses and it ran from April to August in 2017 with a weekly contact session every Wednesday.

A distributed blended online classroom approach, similar in structure to the Structured Training for African Researchers (STARS) Career development course which was developed through the Association of Commonwealth Universities (https://www.acu.ac.uk/focus-areas/early-careers/structuredtraining-for-african-researchers/) and was recently adopted and adapted by the H3ABioNet Introduction to Bioinformatics Training course (Gurwitz et al., 2017), was used for the training. For the virtual classroom approach, trainers were required to prerecord their material, which was then loaded onto a learning management system, in this case – Vula (Sakai based). Facilitators of the various classes were then required to download the course material onto local storage devices such as hard drives to avoid relying on internet connectivity during the live weekly contact sessions. The facilitators and learners watched the videos within the physical classrooms distributed across Africa before the 1-h long online contact session with the trainers. The online live sessions were made possible *via* a webinar platform. During these live sessions, only one connection was allowed from each class *via* the facilitator. The class could then pose questions *via* the facilitators. **Figure 3** summarizes the distributed classroom approach.

All facilitators were encouraged to obtain accreditation from relevant bodies across Africa for continued professional development points (CPD). While the initial plan was to obtain accreditation from all affiliated universities for this course as a short course, this was not feasible and therefore the short course accreditation was only obtained from the University of Cape Town. During the first implementation of the project, facilitators were also tasked with marking the qualitative and face to face assessments such as role plays.

#### Step 6: Evaluations and Assessments

Two types of assessments were designed for the participants, namely summative and formative analysis (Taras, 2005). Formative assessments included the prelesson exercises which were posted on the Vula discussion forum for all to comment, assess, and give feedback. These prelesson exercises aimed to facilitate the students to acquire key skills or understand concepts that the lesson was targeting to address as required for flipped classes (Riddell et al., 2017). Facilitators also used these preclass exercises to gauge general understanding of the concepts by the participants. In addition, feedback forms were sent to participants after each lesson for the participants to rate content, trainers, and logistics. Classes were also required to submit their research proposals for review by the coordinator and the AGMT working group after lessons on research proposal development. AGMT working group members volunteered to assist/review class proposals that aligned with their own research interests. The summative assessment included the participants completing a knowledge, attitude, perceptions, and practices survey during registration to gauge knowledge at baseline. The same survey was administered at the end of the course and will be administered again 24 months after completion of the course. The summative assessment also included assignments and quizzes given to participants after each lesson. Once the research proposals had been approved by the AGMT working group members, as part of the summative assessment each class was required to write a research report which made up 30% of the students' final marks.

In addition to the participant focused assessments, the overall course's implementation process was also evaluated in order to:


to face sessions, participants can download videos from Vula and watch before class and do preclass exercises. During class (face to face sessions), participants engage with learning material (videos, lectures, etc) during face to face classrooms predownloaded by a facilitator. The class can then connect with the trainer virtually for a question and answer session. Participants also submit a quiz and rate the class. After class participants submit an assignment.

The data collection surveys were adapted from literature and used to collect data for each of the assessment strategies highlighted above (McCann et al., 2007; Muzoriana et al., 2017). The participant's assessment strategy and overall implementation evaluation of the project was submitted for review by the Ethics Approval Committee University of Cape Town, Faculty of Health Sciences Ethics Review board.

# PERSPECTIVE

To facilitate a rapid and informed adoption of genomic medicine into routine clinical care in Africa, Continuous Professional Development (CPD) training and formal higher education for healthcare professionals requires radical transformation suitable for low resource settings. Lack of or inappropriate training could delay the translation of the emerging information from several capacity building efforts in genomics and genetics into quality healthcare (Wonkam and Mayosi, 2014; Weitzel et al., 2016). However, instead of being deterred by the challenges and gaps rampant in Africa, the AGMT course was created to pool resources and expertise across the continent and beyond to provide training for healthcare professionals in genomic medicine which would not have been possible through a single institute or initiative.

While a more thorough evaluation of this program is currently ongoing, preliminary results suggest that this is a feasible model. During the first iteration of the course, 368 applications were received, and 225 participants enrolled into the course from 19 Classrooms in 11 Countries. 35% of the participants completed the course and obtained a certificate of completion of the short course from the University of Cape Town. A special collection was set up by the AGMT to which classes submitted their class projects as manuscripts in this peer-reviewed journal http://gheg-journal. co.uk/2018/05/advancing-genomic-medicine-globally/. So far, one, one class published their class project the special collection. The second iteration of the course is still ongoing.

Nurses are frontline workers in most healthcare facilities in Africa and have access to in-depth knowledge of the patients, families, and communities (Prows et al., 2005). There has been an ongoing debate on whether genetic counselling should only be done by professionally trained genetic counselors or if nurses can receive extra training to enable them to provide basic genetic counseling as a component of their current role (Barr et al., 2018). Based on recent review of 10 articles, Barr et al. (2018) confirmed that nurses already provide genetic counseling, as highlighted by the nurse personas development in this study. However, the provision of genetic counseling by nurses is not standardized. There are calls for formal recognition of the nurses' counseling role and the provision of training to support this task. Because of the lack of genetic counselors on the continent (Abacan et al., 2019) and limited resources to train and employ genetic counselors widely in addition to low job creation for genetic counselors low (Kromberg et al., 2013), providing training in genomics and genetics to nurses who can provide basic genetic counseling seems like a feasible strategy to increase the availability of genetic counseling in Africa.

The importance of establishing a set of core competencies to guide the development of skills, knowledge and attitudes required to deliver safe and effective healthcare is well established (Korf et al., 2014). Competencies in genomics and genetics for nurses have been developed and are publicly available online. However, the alignment of existing curricula to such competencies remains limited or is probably reported poorly. The slow rate of curricula modifications in Europe and America has been largely attributed to lack of implementing personnel and difficulties in operationalization of long and complicated competency lists (Jenkins and Calzone, 2007). Noteworthy, is that developing continents including Africa have been largely underrepresented in such competency development initiatives or curricula development initiatives (Jenkins and Calzone, 2007; Korf et al., 2014). This may be partly due to outdated and static curricula which make the alignment with competencies very difficult because it cannot respond appropriately to societal challenges and needs (Gonzalo et al., 2017).

The draft curriculum and competency map provided from this work are likely to promote increased adoption and adaptation of the genomic medicine training into existing nursing curricula across nurse training colleges and centers across Africa. The embedding of online/distance learning modules into formal university/college training has been demonstrated for various massive online open courses (MOOCs). Although MOOCS were originally developed as stand-alone training to be accessed by university students outside of regular curriculum (Swinnerton et al., 2017), when embedded in university medical curriculum, participants have reported high satisfaction on MOOC sourced-course material (Aboshady et al., 2015). Guidelines would need to be developed to facilitate the inclusion of the AGMT modules into existing university curriculum as done by de Jong et al. (de Jong et al., 2019) for the MOOCs.

Another immediate goal of AGMT is to design training for other healthcare workers such as doctors, pharmacists, clinical scientists, patients, and the general public (e.g., patient support groups). The pilot training program and experiences of the process provides a foundation for the group to develop a toolkit for designing and implementing training for other healthcare workers and possibly offering tailored modules across different professions to reflect the multidisciplinary approach in healthcare systems. Unlike other continents/ countries such as Europe (Paneque et al., 2016), Australia (McEwen et al., 2013), and Canada (Ferrier et al., 2013) where bodies have been established to standardize the genetic counseling competencies, where genetic counselors and genetic nurses can register/be certified, most countries in Africa do not yet have certification or registration systems or guidelines to advise on training and practice standards for genomic medicine. The AGMT initiative provides a unique opportunity to be a springboard for development by partnering with existing professional bodies, and by extending training activities to other healthcare professionals.

To our knowledge, this is the first large-scale communitybased training initiative for genomic medicine that has been conducted across Africa. This study highlights the importance of societies and consortia in developing a rigorous training program and a pool of trainers and resources for emerging areas such as genomic medicine.

# DATA AVAILABILITY STATEMENT

All datasets generated for this study are included in the article/ supplementary material.

# ETHICS STATEMENT

Approval was not required according to the study format and local legislation. The paper reports on the implementation of a training program and does not contain personal information.

# AUTHOR CONTRIBUTIONS

VN and NM developed the first manuscript draft. AGMT members who edited or contributed to the manuscript are listed in alphabetical order in the Appendix. Their role in the project is noted in the first column. Members of the planning team (Kuda Majada, Minnet Cotzee, Faisal Fadlelmola, Pedro Fernandes,

# REFERENCES


Samar Kamal Kassim, Cordelia Leisegang, Ebony Madden, Alice Matimba, Oyekanmi Nash, Michael Pepper, Fouzia Radouani, Raj Ramesar, Michelle Skelton) were responsible for the course development and organization.

# FUNDING

NM is supported by the National Human Genome Research Institute (NHGRI) and the Office of The Director (OD), the National Institutes of Health under award numbers U41HG006941 and U24HG006941. VN is supported by NIH/ NHLBI U24HL135600. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The funders had no role in study design, data collection, and analysis, decision to publish, or preparation of the manuscript.

# ACKNOWLEDGMENT

We thank Dr Maria Giovanni for reviewing the article and Jennifer Troyer for helping with strategy.


**Conflict of Interest:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2019 Nembaware, African Genomic Medicine Training Initiative and Mulder. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# APPENDIX AUTHOR DETAILS -MEMBERS OF THE AFRICAN GENOMIC MEDICINE TRAINING INITIATIVE


Mulder et al.

#### Continued


(Continued)

African Genomic Medicine Training Initiative

#### Continued


# Preparing Medical Specialists for Genomic Medicine: Continuing Education Should Include Opportunities for Experiential Learning

Belinda J. McClaren1,2,3\*, Erin Crellin1,2,3, Monika Janinski 1,2,3, Amy E. Nisselle1,2,3, Larissa Ng1,2,3, Sylvia A. Metcalfe1,2,3 and Clara L. Gaff 1,2,3 on behalf of the Australian Genomics Workforce & Education Working Group

<sup>1</sup> Australian Genomics Health Alliance, Melbourne, VIC, Australia, <sup>2</sup> Genomics in Society, Murdoch Children's Research Institute, Melbourne, VIC, Australia, <sup>3</sup> Department of Paediatrics, The University of Melbourne, VIC, Australia

#### Edited by:

Georgia Theocharopoulou, Ionian University, Greece

#### Reviewed by:

Vasiliki Nataly Rahimzadeh, Stanford University, United States Siobhan Simpson, Health Education England, United Kingdom

\*Correspondence: Belinda J. McClaren belinda.mcclaren@mcri.edu.au

#### Specialty section:

This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics

Received: 07 June 2019 Accepted: 10 February 2020 Published: 03 March 2020

#### Citation:

McClaren BJ, Crellin E, Janinski M, Nisselle AE, Ng L, Metcalfe SA and Gaff CL (2020) Preparing Medical Specialists for Genomic Medicine: Continuing Education Should Include Opportunities for Experiential Learning. Front. Genet. 11:151. doi: 10.3389/fgene.2020.00151 With the demand for genomic investigations increasing, medical specialists will need to, and are beginning to, practice genomic medicine. The need for medical specialists from diverse specialties to be ready to appropriately practice genomic medicine is widely recognised, but existing studies focus on single specialties or clinical settings. We explored continuing education needs in genomic medicine of a wide range of medical specialists (excluding genetic specialists) from across Australia. Interviews were conducted with 86 medical specialists in Australia from diverse medical specialties. Inductive content analysis categorized participants by career stage and genomics experience. Themes related to education needs were identified through constant comparison and discussion between authors of emerging concepts. Our findings show that participants believe that experiential learning in genomic medicine is necessary to develop the confidence and skills needed for clinical care. The main themes reported are: tailoring of education to the specialty and the individual; peer interactions contextualizes knowledge; experience will aid in developing confidence and skills. In fact, avenues of gaining experience may result in increased engagement with continuing education in genomic medicine as specialists are exposed to relevant applications in their clinical practice. Participants affirmed the need for continuing education in genomic medicine but identified that it would need to be tailored to the specialty and the individual: one size does not fit all, so a multifaceted approached is needed. Participants infrequently attended formal continuing education in genomic medicine. More commonly, they reported experiential learning by observation, casereview or interacting with a "genomics champion" in their specialty, which contextualized their knowledge. Medical specialists anticipate that genomic medicine will become part of their practice which could lessen demand on the specialist genetic workforce. They expect to look to experts within their own medical specialty who have gained genomics expertise for specific and contextualized support as they develop the skills and confidence to practice genomic medicine. These findings highlight the need to include opportunities for experiential learning in continuing education. Concepts identified in these interviews can be tested with a larger sample of medical specialists to ascertain representativeness.

Keywords: genomic education, genomic medicine, medical specialist, workforce, qualitative needs assessment, experiential learning

#### INTRODUCTION

The emerging practice of genomic medicine, the use of genomic information to guide diagnostic and treatment decisions, promises to transform the way medicine is practiced (Collins and McKusick, 2001; Williams, 2019). Yet challenges remain in maximizing the potential benefits within healthcare settings and beyond specialist genetic services (Ginsburg, 2014; Gaff et al., 2017). Zebrowski et al. recently evaluated perspectives on implementing genomic medicine within the IGNITE network (Implementing GeNomics In pracTiCe). While participants identified clinician engagement as essential for genomic medicine implementation, researchers actually observed a lack of clinician engagement among participants studied (Zebrowski et al., 2019). Medical specialists who are not already engaged in providing genetic services will need to "develop and expand" their expertise in inherited diseases and the use of new genomic technologies in their clinical practice (Burton, 2011; Burton et al., 2017; Gaff et al., 2017).

Changes to medical education and training curricula will address this gap over time, but there is a pressing need for those already in practice to be ready to integrate testing and application of test results into medical care. The challenges for medical specialists to integrate genomic medicine into their clinical practice have only been investigated in a piecemeal approach so far, with most studies involving hospital-based specialists from the same specialty. For example, in studies involving oncologists, clinicians reported feeling underprepared to comprehend and communicate genomic test results despite practicing in areas in which the clinical utility of genomic investigations for some conditions or some patients was established and testing was available (Chow-White et al., 2017; Johnson et al., 2017; Weipert et al., 2018). While expressing familiarity with discussing genetic information, cardiologists in the MedSeq study similarly felt underprepared to navigate complex genomic test results, particularly those that lay outside their specialty (Christensen et al., 2016).

In the U.S.A., a nation-wide study of pharmacogenomics has been conducted (Stanek et al., 2012), but we found no other, nation-wide studies that include a broad range of medical specialties to explore readiness to practice genomic medicine and the role continuing education plays. Yet, the need for education to support the implementation of genomic medicine has been recognised internationally by policy makers (Manolio et al., 2013; Bowdin et al., 2016). For instance, the Australian Government recently released a National Health Genomics Policy Framework1 which identified "building a skilled workforce that is literate in genomics" (page 3) as a key strategic priority (Australian Government Department of Health, 2017). However, policy statements such as these need education plans to prepare clinicians to practice and explain the role continuing education can play. In England, the National Health Service invested early in a "top–down" approach with centralized administration to equip the workforce to incorporate genomics through a range of education and training initiatives (Turnbull et al., 2018). Australian investment has been made in national research funding to provide evidence for the equitable, effective and sustainable integration of genomic medicine in healthcare through the Australian Genomics Health Alliance (Australian Genomics) (Long et al., 2019). Australian Genomics is a research partnership of clinicians, diagnostic geneticists and researchers from >80 organizations using a co-ordinated nationwide approach (Stark et al., 2019a). To inform the development and delivery of effective education and training in genomics across the broad health care system and adoption of genomics by numerous medical specialties the Australian Genomics Workforce & Education research program takes a whole-ofnation, "bottom up," research approach. A mixed-methods design for the research program is being undertaken to examine the perspectives of multiple stakeholder groups (Figure 1) (Creswell and Plano Clark, 2017; McClaren et al., 2020).

We report here the qualitative findings of this nation-wide study of medical specialists, the first to address diverse specialties and career stages. This study has informed the development of a nation-wide quantitative survey capturing representative data across medical specialties and healthcare settings. Specifically, the purpose of this study was to understand how medical specialists in Australia perceive the relevance of genomics to their practice as well as their views on continuing education in genomics to enhance clinician readiness. This manuscript presents findings related to medical specialists' needs for continuing education in genomic medicine.

#### MATERIALS AND METHODS

Key informant qualitative interviews were conducted and the semi-structured interview guide addressed: participant

<sup>1</sup> http://www.coaghealthcouncil.gov.au/Portals/0/Genomics%20Framework% 20WEB\_1.PDF

et al., 2018); <sup>g</sup> (Janinski et al., 2018); <sup>h</sup> (Nisselle et al., 2019a); <sup>i</sup> First meeting held August 2018, Sydney; <sup>j</sup> (Stark et al., 2019b). Participants groups represented in the studies within this program of research are: medical specialists (3, 9–16), genetic counselors (4, 11–15), clinical geneticists (2, 4, 11–15), bioinformaticians and medical scientists (11–15), genomic education providers (2, 12–15), general practitioners (5, 6, 12–15), patients or parents of patients (7, 15), system influencers and policy makers (8), oncologists (10), community practitioners (pharmacists, nutritionists, private practice genetic counselors (6).

characteristics; current role; experience with genomic medicine; participation in or attendance at education and training activities; and perceptions of future need for continuing education. This study had human research ethics approval (University of Melbourne, HREC: 1646785). As per the approved research protocol and in accordance with the National Statement on Ethical Conduct in Human Research (Section 2.2.5)<sup>2</sup> , interview participants gave verbal consent for interviews to be audio recorded, transcribed and for de-identified quotes to be used in publications or reports arising from the research. Purposive and snowball approaches were used for maximum variation sampling in order to gather data representative of various genomics experience levels and career stages (Patton, 2015). This included direct email invitation to individuals who have a medical degree and specialist training. The term "medical specialists" is used in this study to mean "doctors specialized in a field other than general/family practice or clinical/medical genetics" (Crellin et al., 2019, pg 1–2). We have separate studies (Figure 1) underway or completed with clinical (or medical) geneticists (Nisselle et al., 2019a) and general practitioners (GPs) as we anticipated that the needs of those who are specialized in genetics or those in primary care (i.e. GPs) may be quite distinct and therefore require separate consideration. In Australia, GPs have a different training pathway<sup>3</sup> to physicians (medical specialists) and their role is typically to refer patients with likely medical conditions to medical specialists or genetic specialists who will examine, investigate and deliver results to patients. Therefore, GPs may need broad knowledge about appropriately identifying and referring their patients who have further need of follow-up, whereas the role of medical specialists is to request diagnostic tests, interpret results, and deliver results to patients. Hence GPs were excluded from this set of interviews.

Interviews were conducted by telephone or face-to-face, and audio-recorded. Recordings were transcribed verbatim, checked for accuracy, and NVivo 123 used to manage qualitative analysis. Participants were stratified using content analysis to explore how their views might differ across genomics experience levels and career stages (Hsieh and Shannon, 2005; Patton, 2015). By

<sup>2</sup> https://www.nhmrc.gov.au/about-us/publications/national-statement-ethicalconduct-human-research-2007-updated-2018#toc:296, The National Statement on Ethical Conduct in Human Research (2007) (National Statement (2007) consists of a series of guidelines made in accordance with the National Health and Medical Research Council Act 1992. The National Statement is developed jointly by the National Health and Medical Research Council, the Australian Research Council and Universities Australia.

<sup>3</sup> NVivo. (2018). NVivo qualitative data analysis software. 12 ed (QSR International Pty Ltd).

inductively analyzing manifest content (self-reported current practice and genomics experience), participants were classified as belonging to one of three genomics experience levels (Table 1):


TABLE 1 | Participant characteristics.


a 20 medical specialties were approached with responses from 18: anesthesiology (n = 1), cardiology (n = 1), dermatology (n = 1), endocrinology (n = 4), fetal medicine (n = 2), general medicine (n = 1), hematology (n = 6), immunology (n = 17), infectious disease (n = 2), intensive care (n = 7), nephrology (n = 5), neurology (n = 5), neuropsychiatry (n = 4), obstetrics & gynaecology (n = 2), oncology (n = 6), general pediatrics (n = 4), pathology (n = 8) and rheumatology (n = 10). There were two further specialties approached but no response was received and therefore no interview could be completed: emergency medicine and ophthalmology.

b may also see adult patients.

c Some involvement = Listed on grants, referring patients into research studies, but not running studies themselves; Very involved = Leads research programs (gene discovery, testing patients), holds grants, doing PhD related to genomics.

e Data only collected for 63/86 participants due to difference in data collection tools used for immunologists and rheumatologists who were not asked about involvement in educating peers. <sup>4</sup>

• Experienced: current use of genomics in clinical practice and/or; active involvement in genomics research (molecular or clinical) and/or; participation in continuing education in genomics.

Participants were additionally categorized (Table 1) into their career stage according to the medical training pathways in Australia<sup>4</sup> :


A coding framework was developed based on the broad topics from the interview guide with further codes added in an inductive process. The analysis approach was iterative and involved reading and re-reading the transcripts using constant comparison to identify similarities and differences, and discussion between coders (BM, EC, MJ, LN) of emerging concepts (Vaismoradi et al., 2013; Patton, 2015). All the transcripts were coded once and the full codebook developed. All transcripts were then coded a second time using the codebook. Regular discussions between all four coders managed the development of codes, the emergent concepts and helped resolve conflicts among coders.

# RESULTS

From January 2017 and May 2018, 240 medical specialists were invited to participate in the study. Interviews were conducted with 86 medical specialists from 18 different specialties (Table 1). Interviews were held with all who responded and for whom an interview could be arranged, which allowed for an inclusive approach with broad representation of a variety of participants. Findings are shown below using representative quotes as exemplars and attributed to participants using study numbers and descriptors of their specialty. Some quotes have been truncated for readability without changing the meaning, indicated by "…".

All participants affirmed the need for continuing education in genomic medicine. Findings from participants related to continuing education for medical specialists are presented and summarized into the following themes: tailoring of education to the specialty and the individual; peer interactions contextualizes knowledge; experience will aid in developing confidence and skills. The concepts covered in the sections below are interlinked due to the nature of how the participants spoke about their interactions with genomic medicine and their needs for continuing education. While presented in separate sections, the illustrative quotes may convey more than one idea from more than one section. Figure 2 provides an overall conceptual representation of the emergent concepts: showing how a foundation of knowledge from formal sources is built on by interactions with peers to begin to contextualize knowledge. As

d Includes having any role in delivering peer education (not just genomic). Some=gives occasional talks to department, sought out by peers for information; Very=organizes and delivers education to peers, recognized as a genomic leader in their field. Of those who do educate their peers, only two have formal background/qualifications in education.

<sup>(</sup>https://ama.com.au/careers/becoming-a-doctor) Doctor Life Cycle.

medical specialists have opportunity to gain experience in genomic medicine, their confidence and skills grow. As well as the three themes, we present challenges to continuing education identified by the participants.

#### Continuing Education in Genomic Medicine Needs to be Tailored to the Specialty and the Individual

Medical specialists identified that one size does not fit all for approaches to continuing education in genomic medicine. Participants having experience with genomics, and therefore a greater level of confidence to practice genomic medicine, had different needs for continuing education compared with those who were less experienced.

"For me, (current education activities are) fairly good and adequate, I already come into it with a knowledge of genomics (through involvement in genomics research). I'm not sure for the general clinician whether there is enough opportunities … to upskill them." [MS23, senior, experienced, neurologist]

"Education needs … it'd be a few tiers of education, so a general education to the general health service providers, as well as a targeted, more in-depth education to those who (currently use genomics)." [MS53, senior, interested, nephrologist]

Genomics novices sought out basic information and updates, whereas others with more experience wanted greater detail.

"…if it goes into a lot of detail, I just start to get confused and tune out a little bit … I'm interested to have a bit of an idea about how it works." [MS54, early, novice, pediatrician]

"(What) I need is a refresher course with a very clinical tilt to it. I do not want to know the A T G C, but I want to know, when you say genome exome sequencing or whole genome sequencing, what do you do, what are the results you get, and how do you make those decisions after that, as in how do you report them." [MS57, mid, interested, intensivist]

Participants also discussed how career stage might influence how they would like to learn about genomic medicine.

"Registrars would probably be quite happy doing (webinars)…the older people get, the less inclined they probably are to engage in that way … there is a limit to

FIGURE 2 | A summary of the participant-described approaches to education and learning that can prepare a medical specialist to practice genomic medicine. Formal sources of education, such as structured programs, provide knowledge that is then contextualized through peer-to-peer interactions and opportunities for experiential learning; each of these can build upon each other although are not necessarily equal in quality and quantity. Defining preparedness is challenging and may vary for different types of specialists (Vassy et al., 2015); we use this term to encompass knowledge, attitude, skills and confidence (Crellin et al., 2019). <sup>a</sup> These activities are ones in which medical specialists would receive recognition from their relevant medical College, such as "points," for having completed the educational activity.

how much you can actually get from sitting there looking at the video."[MS21, senior, experienced, immunologist]

There were conflicting views regarding baseline understanding of genomics concepts: some thought recent medical graduates would have more knowledge, gained in their formal education than senior physicians, yet in contrast others described that current medical school and training curricula appeared to have limited genomics content.

"Someone who graduated from medicine in 2014 is going to have a very different baseline knowledge of genomics than someone like me who … finished the medical course in (196-)." [MS19, senior, novice, immunologist]

"I'm still surprised how little genetics the current trainees know … the actual training to be a physician or a sub-specialist, there still seems to be very little formal genetics training within that." [MS23, senior, experienced, neurologist]

We didn't do a lot (in medical training)…most people came (to medicine) from basic biomedical degrees…(In medical training) there'd be a mention of something in a lecture on pediatric cardiology about various different genetic conditions that you have for pediatric cardiac genetic mutations … We didn't, in medicine, go through the molecular basis of how that happens. [MS30, early, novice, general medicine physician]

A common theme from all participants was the importance for continuing education to be clinically relevant and tailored to the audience. Clinicians wanted the pitch and scope of information to be tailored to their specialty, and relevant to the patients they see now or anticipate seeing in the near future.

"The different specialties may be very different … if you're thinking about oncology genetics, the relevance for neurology would be very different." [MS27, senior, novice, neurologist]

"(A) seminar or some sort of update … put within a context that clinicians would recognise it as being directly relevant to what they do day-to-day, rather than relevant to them in 10 or 15 years' time." [MS32, mid, interested, rheumatologist]

To further identify continuing education needs, participants were also asked to suggest topics to be addressed for continuing education in genomic medicine to support their readiness to practice genomic medicine (Table 2). A spectrum of content was described, from basic, to practical, to technical and clinically applied/or advanced (e.g., the precise phenotypic information required to make decisions about gene lists or variant classification). Participants also identified other skills, such as communication and counseling, for example, helping families understand implications of genomic data storage and use, and interpretation of detected variants.

"(I'd like to learn about) the technology itself, the limitations, the patient selection, the counseling around the results and the meaning of the results and how you work through the variants that you're not sure of." [MS27, senior, novice, neurologist]

# Learning From Peers Contextualizes Knowledge in Genomic Medicine

Participants described how interactions with peers contextualizes formal learning in genomic medicine, which required participants to have peer networks they could draw on, physically, if the setting allowed it, or by phone.

"I can walk down the corridor and talk to (a clinical geneticist)." [MS04, senior, interested, endocrinologist]

"We're very spoiled here … pick up a phone or … get the geneticist to see them. It might be different for pediatricians out in the rest of the world." [MS54, early, novice, pediatrician]


TABLE 2 | Topics suggested by participants for continuing education to support their readiness to practice genomic medicine.

Some peers were identified as particularly useful: these were described by participants as "champions," who are medical specialists, usually within the same medical specialty, with a special interest in genomics and would readily share their genomics expertise with others.

"I think one of the hopes is that I will be a little bit of a link and help upskill…(using) my learning … spread a little bit of that (to my colleagues)." [MS47, early, experienced, neuropsychiatrist]

"Find a few people who are in the intensive care (ICU) scenario who are your champions, and have the ICU guys talk to the ICU guys … rather than have Genetics coming in, giving a talk and intensivists only understanding half of it." [MS63, senior, experienced, intensivist]

"Maybe keep it just in the hands of the few competent people in every specialty who can handle this and who can advise others about what the consequence of certain findings are." [MS52, senior, interested, nephrologist]

#### Opportunities to Gain Experience in Genomic Medicine Promotes Confidence and Skills

Few participants were aware of or had attended any formal continuing education courses or workshops in genomic medicine as shown in Figure 2. Participants described addressing these needs instead through experiential learning opportunities in the following ways: passive approaches such as attendance at conferences or seminars; active learning through research projects, seeing patients in clinical practice, or undertaking immersive cross-training; or multidisciplinary team (MDT) meetings where clinical cases are reviewed, which could be a combination of passive and/or active learning. Representative quotes of the ways in which participants identified learning through opportunities to gain experience are summarised in Table 3. Specifically, participants described how MDT meetings gave them the opportunity to learn passively by hearing cases of their peers, and also to be active contributors by nominating their own cases and taking part in discussions around gene list prioritization or variant classification.

"…multidisciplinary meeting with experts from different areas present in the room to assist in making management decisions about patients … As a learning exercise for clinicians it was incredibly valuable to be … benefiting from the expertise of scientists." [MS99, senior, interested, oncologist]

Opportunities to learn by gaining experience were variable for participants in this sample. For most participants, such experiential learning was possible due to genomic medicine increasingly becoming part of their clinical practice or likely to be in the near future. Participants recognised that some fields would have more opportunities for experiential learning compared with others because genomic medicine was more relevant and available.

"The microbiologists, the hematologists, the geneticists and the endocrinologists were all very early adopters of genomics because of the sort of conditions we see and the ease of sample collection. [MS03, senior, interested, endocrinologist]

Others described their limited experience with exome testing and that their current approach would be to refer to a geneticist and therefore they are not gaining experience themselves. Without these opportunities to learn and limited (to date) experience in delivering genomic medicine some participants felt they were less confident to practice.

"I have referred patients to geneticists with the specific question of 'is this patient suitable for exome sequencing?', but I haven't actually put in an order for it myself." [MS54, early, novice, pediatrician]

"I certainly wouldn't feel comfortable looking at reports myself, and relying on my own interpretation. I would think I'd need many more years of looking at that before I'd be comfortable." [MS11, early, interested, hematologist]

# Challenges to Learning Identified

Preferences for learning were asked of all participants and although formal continuing education activities such as workshops, short courses and online courses were raised, the following quote exemplifies the decisions participants made about the benefits and competing demands in attending education sessions.

"Can I physically attend this? Is it possible given my shift schedule, and then is this a skill I either want to get better at or I really need?" [MS30, early, novice, general medicine physician]

While learning through peer-to-peer interaction or experiential learning, were commonly-mentioned means of developing skills and confidence in genomic medicine, participants did not view these as "education" per se.

"It was just kind of ad hoc, learning as you go … I did spend some time in the molecular genetics lab … I did go to curation meetings … It worked for me, except it wasn't formal teaching where you actually get through the patients being presented, it was more, picking up and asking little questions here and there about very basic things. But it wasn't structured education or anything." [MS33, mid, interested, neurologist]

Participants described how experiential learning was also not equally available across different settings, for example less so in the private sector, or where genomic medicine was infrequently practiced. This was considered a barrier for some medical specialists to upskill in genomic medicine.

#### TABLE 3 | Participant descriptions of approaches to learning to support practice of genomic medicine.


a Multidisciplinary team.

> "It's hard in the private sector … in the public sector you have MDTs. We don't have much of that so I think that's where it's lacking."[MS25, mid, experienced, hematologist]

# DISCUSSION

This study provides new insights applicable to meeting the continuing education needs in genomic medicine of diverse medical specialists. The need for education in response to increasing availability of genomic testing in clinical settings has been previously demonstrated (Manolio et al., 2013; Bowdin et al., 2016; Burton et al., 2017; Paul et al., 2018). We extend findings from earlier studies of select medical specialists with this cohesive study exploring a large national sample with diverse specialties, career stages, public and private practice settings and (in)experience with genomics.

Our findings show that motivations to engage with continuing education about genomic medicine appear to be driven by a combination of: individual characteristics (interest in genomics, career stage, and medical specialty); perceptions of relevance to practice (current and future); and prior experience, such as that gained in research settings. We have shown that medical specialists contextualize their knowledge gained through formal education by engaging with their peers and seeking out opportunities for experiential learning. In fact, participants described how most genomics learning occurs outside of attendance at continuing education activities, which have been the previous focus of workforce development (Burton, 2011; Talwar et al., 2017).

#### Continuing Education Activities Should Include Opportunities for Experiential Learning

Experiential learning approaches are consistent with adult learning theory, which acknowledges the role of experience and relevance to work settings. Encountering clinical problems will be drivers for medical specialists to self-identify areas of education need and will motivate them to participate in activities to fulfil the gaps in competence or confidence (Grant, 2002; Metcalfe et al., 2008; Knowles et al., 2015). Opportunities for experiential learning should be provided alongside formal continuing medical education activities in genomics. Despite the calls for formal education programs for health professionals in genomics (Passamani, 2013; McGrath and Ghersi, 2016), the medical model of structured "bedside" teaching would also be an appropriate approach for integrating the skills to practice genomic medicine in real-life contexts (Peters and Ten Cate, 2014).

Learning, as described by participants in our study, may include a gradual building of experience, confidence and procedural skills that are specific to the way a specialist may practice genomic medicine. Learning in this context was described as most likely to come from their colleagues who were more experienced in genomics. Such people need to be fostered in their roles as "genomics champions" within their specialty to ensure they demonstrate appropriate competence and are given time and support to teach others. This collegial learning may be less accessible in more isolated sites, such as private practice and more geographically remote settings, so attempts to re-create these opportunities are needed, perhaps by teleconference or telemedicine. Future research is needed to assess the acceptability and feasibility of this approach. Telemedicine in oncology settings has been used effectively to convene virtual tumour boards and educate clinicians (Satcher et al., 2014).

#### The Complexity of Providing Continuing Education in Genomic Medicine: The Need for a Multi-Level Approach Across Broad Topics

A nuanced and comprehensive view of learning needs to be taken to ensure medical specialists are equipped to provide genomic medicine to their patients. Specific continuing education activities may provide one approach, but this study suggests that medical specialists will engage more with experiential learning. Such learning may be more likely to encourage medical specialists to adopt genomic medicine when: they feel confident in the clinical utility of genomic medicine; their clinical setting supports genomic testing; and they have developed networks and relationships with colleagues, including those seen to be "genomics champions" within their specialty. As highlighted in a review by Paul et al, evidence for the effectiveness and importance of educational activities is lacking, with current understanding from published studies suggesting many other important domains will contribute to the behaviour change required for the widespread adoption of genomic medicine (Paul et al., 2018).

Clearly, no one size or one time-point for education fits all; therefore, a multi-level approach will be needed to ensure lifelong learning is available to support the implementation of genomic medicine into healthcare. Our data suggests this might include efforts to ensure that foundational or threshold concepts of genomic medicine as well as practical skills (terminology, limitations, guidelines, required phenotypic information, and result generation) be included in continuing education (Table 2). As shown in Figure 2, knowledge of these topics can then be applied and contextualized over the professional life-course of the medical specialist. As the medical specialist encounters genomic medicine in their practice and has a developing sense of its relevance to their patients, they are likely to seek out continuing education to support their practice. Continuing education has the role, therefore, of providing practical examples of genomic medicine to enhance specialists' confidence and skills to practice.

Our findings also show content areas participants felt would be valuable to address in continuing education (Table 2). Regardless of the content topic in focus, relevance to clinical practice is essential for learning, therefore the specialist's clinical practice influences their perception of relevance of genomic medicine and motivation to undertake continuing education (Burke et al., 2006; Reed et al., 2016). A foundational, baseline understanding of genomic concepts allows a common language to be used and understood in communication, then practical training is needed to convert fundamental understanding into confident practice (Stanek et al., 2012). This common language and understanding would encourage good relationships between scientists and clinicians, which is essential for efficient clinical outcomes (Burton et al., 2017; Weipert et al., 2018).

If learning is occurring predominantly experientially rather than via structured activities, evaluation of teaching opportunities will be challenging. Australian Genomics has formed a working party to create a genomics education evaluation framework: international experts in genomics education, evaluation and implementation science met for a workshop in February 2018 to draft a program logic and evaluation framework, which is being refined and tested with member educational activities. A separate publication describes the framework and its development (Nisselle et al., 2019b).

#### The Importance of Needs Assessments in Developing Continuing Education Programs

To evaluate the extent of the use of formal education, we previously undertook a mapping exercise of continuing educational activities available to medical specialists in Australia for genomic medicine (McClaren et al., 2018). This mapping and interviews with providers of educational activities demonstrated that most people delivering such education are clinicians rather than educators (Janinski et al., 2018). They may, therefore not think of experiential learning as a strategy to include in the design of their educational activity. The recommendation in this paper to incorporate experiential learning into continuing education activities is aimed at clinicians (and educators) who are currently providing continuing education in genomic medicine for medical specialists. These findings can assist those who are charged with the continuing professional development of a single medical specialty or a hospital- or system-wide program to provide the most acceptable and feasible approaches for medical specialists to learn about genomics.

Our findings can also inform needs assessments ahead of producing continuing education programs in genomic medicine; such an approach has previously led to the development of successful and effective education programs (Gaff et al., 2007; Carroll et al., 2009; Houwink et al., 2011; Houwink et al., 2014; Reed et al., 2016). These qualitative findings have already been used to create a survey tool which can be used in international settings to measure physician preparedness for genomic medicine and their preferences for genomics continuing education (McClaren et al., 2020). When using a program logic model to develop education activities and initiatives, an important component of the planning phase is conducting a needs analysis (Nisselle et al., 2019b). The current study has served as a needs analysis to inform the development of educational activities locally after presentation of findings at a workshop in August 2018 (Figure 1, item 15. Establish genomic education network).

#### Limitations and Future Directions

Although all health systems have unique features, there is a commonality in the challenge of preparing health professionals for genomic medicine. A strength of this study, the broad sample interviewed, means that findings from our study may have wider relevance and inform local needs assessments. It is a limitation that, despite attempts, not every specialty of medicine is represented so there is further need to seek input from missing specialties. A qualitative approach provides a rich data set to inform future studies to assess the representativeness of our findings; this is underway, with an Australian survey of medical specialists (Figure 1) (McClaren et al., 2020). Further, these data are a point-in-time perspective of medical specialists suggesting the need for opportunities for experiential learning, within a largely pre-adoption of routine practice of genomic medicine. It is possible that as genomic medicine is more routinely practiced, the need for experiential learning may lessen.

# CONCLUSIONS

In summary, our data suggest that approaches to continuing education in genomic medicine should consider:

• Experiential, hands-on learning opportunities that are closely aligned to how genomic medicine will be delivered in practice


Ultimately, with the demand for genomic investigations increasing, medical specialists will need to, and are beginning to, practice genomic medicine. Our findings show that medical specialists expect to look to experts in their own medical specialty for specific and contextualized support to competently and confidently practice genomic medicine when appropriate to their clinical need. These findings have been used to create a survey tool which can be used to measure physician preparedness for genomic medicine and preferences for continuing education in a representative sample (McClaren et al., 2020). However, it is clear that continuing education in genomic medicine will need to be multifaceted to meet the diverse needs of medical specialists and should include opportunities for experiential learning.

#### DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

#### ETHICS STATEMENT

The studies involving human participants were reviewed and approved by University of Melbourne, Human Research Ethics

#### REFERENCES


Committee. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

#### AUTHOR CONTRIBUTIONS

BM, AN, SM, and CG conceived the idea for the manuscript, and BM and EC conducted the interviews. BM led the analysis which was also completed by EC, MJ, and LN. BM drafted the manuscript and all authors revised drafts, approved the final version, and agree to be accountable for all aspects of the work.

# FUNDING

This work was supported by the Victorian Government's Operational Infrastructure Support Program and a grant from the Australian National Health & Medical Research Council (GNT1113531); the contents are solely the responsibility of the individual authors and do not reflect the views of the NHMRC. EC was supported by the Helen R. Freeman Scholarship.

#### ACKNOWLEDGMENTS

The authors thank the medical specialists who participated in this study. The Australian Genomics Workforce & Education Working Group members not listed as authors are: Kate Dunlop, Dr Debra Graves, A/Prof Ainsley Newson and Bronwyn Terrill.


randomized controlled trial on oncogenetics. Eur. J. Hum. Genet. 22, 310–316. doi: 10.1038/ejhg.2013.163


based instructional model for community-based physicians. Genet. In Med. 18, 737–745. doi: 10.1038/gim.2015.144


Conflict of Interest: CG and SM are co-editors of the Research Topic where this manuscript is featured. All authors share their affiliations with CG and SM. CG and SM were not involved in the editorial or peer-review process; this was overseen by the third co-editor who is not affiliated with the authors' institutes.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2020 McClaren, Crellin, Janinski, Nisselle, Ng, Metcalfe and Gaff. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Development of an Evidence-Based, Theory-Informed National Survey of Physician Preparedness for Genomic Medicine and Preferences for Genomics Continuing Education

Belinda J. McClaren1,2,3\*, Emily A. King1,2, Erin Crellin1,2,3, Clara Gaff 1,2,3, Sylvia A. Metcalfe1,2,3 and Amy Nisselle1,2,3 on behalf of the Australian Genomics Workforce & Education Working Group

<sup>1</sup> Australian Genomics Health Alliance, Melbourne, VIC, Australia, <sup>2</sup> Genomics in Society, Murdoch Children's Research Institute, Melbourne, VIC, Australia, <sup>3</sup> Department of Pediatrics, The University of Melbourne, Melbourne, VIC, Australia

#### Edited by:

Go Yoshizawa, OsloMet—Oslo Metropolitan University, Norway

#### Reviewed by:

Holly Landrum Peay, RTI International, United States Katie Michelle Saulnier, McGill University, Canada

\*Correspondence:

Belinda J. McClaren belinda.mcclaren@mcri.edu.au

#### Specialty section:

This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Genetics

Received: 30 July 2019 Accepted: 17 January 2020 Published: 03 March 2020

#### Citation:

McClaren BJ, King EA, Crellin E, Gaff C, Metcalfe SA and Nisselle A (2020) Development of an Evidence-Based, Theory-Informed National Survey of Physician Preparedness for Genomic Medicine and Preferences for Genomics Continuing Education. Front. Genet. 11:59. doi: 10.3389/fgene.2020.00059 Despite some early implementation of genomic medicine globally, there is a lack of rigorous, large-scale assessments of medical specialists' current practice and continuing education needs. As a first step to addressing this gap, we describe the development of a robust, expert-reviewed, survey using a mixed-methods sequential study design. We conducted semi-structured qualitative interviews with 32 education providers and 86 nongenetic medical specialists about current genomic medicine practice and need for continuing education. Key concepts were identified and used as an initial framework for the survey. These were: personal characteristics (medical specialty, years of practice); current practice of genomics in clinical and research settings; perception of how proximal genomic medicine is to practice; perception of preparedness (competence and confidence); and, preferences for future roles and models of care in genomic medicine and for continuing education. Potential survey questions that related to at least one of these concepts were identified from the literature or were created if no suitable question existed. Using a modified, reactive Delphi approach, questions were reviewed by a panel of 22 experts. Experts were selected purposefully representing four areas of expertise: non-genetic medical specialties; clinical genetics; genetic/genomic education and evaluation; and implementation science. Three Delphi rounds assessed relevance, clarity and importance of each question. The questions were also mapped to the behaviour change wheel theoretical framework which encompasses capability, opportunity and motivation (COM-B). The survey (included as supplementary material) was then tested with a small group of non-genetic medical specialists and feedback was written or verbal in 'talk-aloud', cognitive interviews. The final survey was then piloted with a further 29 specialists. We describe the methodology to create a robust, data- and theory-informed survey. The final survey captures not only levels of experience, practice of genomics and preferences for education but also the challenges around engaging with education. Survey data will provide evidence for education providers to inform development of education which meets learner needs and contributes to a medical workforce that is literate in genomics and more confident to competently practice genomic medicine.

Keywords: survey development, genomic education, qualitative, Delphi, theory

# INTRODUCTION

Genomic medicine is increasingly present in clinical practice, requiring non-genetic medical specialists to 'develop and expand' expertise (Burton, 2011; Burton et al., 2017; Gaff et al., 2017). As the growing use of genomic investigations is rapidly exceeding the capacity of the clinical genetics workforce (Slade et al., 2016; Maiese et al., 2019), different approaches to the practice of genomic medicine will be needed. Consequently, it is likely non-genetic medical specialists will need to alter their current practices and behaviors to incorporate genomic medicine, with some taking on tasks previously in the remit of genetic health professionals (Bowdin et al., 2016; Ormond et al., 2019). This may include directly requesting tests for patients, and discussing results, rather than referring to a clinical genetic service.

Education has been suggested as an approach to address gaps in skills and confidence of non-genetic medical specialists to practice genomic medicine (Feero and Green, 2011; Paul et al., 2018; Crellin et al., 2019). To date, there has been no systematic approach to measure the educational needs of the medical workforce on a national scale in Australia, and to understand how these needs may differ across diverse disciplines. Therefore, there is little evidence available to inform the design and development of system-wide educational or training activities to support non-genetic medical specialists in acquiring the skills, confidence and competence they need to appropriately integrate genomic medicine into their clinical practice. The implementation of genomics in healthcare is being addressed at a national level in a number of countries (Stark et al., 2019a). For instance, in Australia the Federal Government has developed a National Genomics Health Policy Framework that identifies genomic literacy of health professionals as a national priority (Australian Government Department of Health, 2017). For countries with a publicly funded (socialized) health system, a health system-wide approach to understanding the practice and needs of diverse disciplines can enable government decisionmaking on how investment in education and training may best be deployed. Capturing details of current practice, perceptions of future practice and preferences for learning can also provide much needed evidence for education providers about the areas on which to focus their efforts and resources. For example, are there particular sub-specialties of medicine for whom the need and desire for educational activities in genomics is greatest? Are there other specialties in which genomic medicine seems far from relevant to clinical practice and therefore their engagement with educational activities is likely to be low? What might be the important, clinically-relevant topics to address in educational activities that medical specialists identify as being critical to their adoption of genomic investigations? Also of importance is understanding non-genetic medical specialists' preferences and expectations for their future practice of genomic medicine, as this will also provide insight into their needs for continuing education which can be specific to their clinical role.

Existing, published surveys address some of these research questions. Some focus on genetic concepts (e.g., taking family history) and tests (Jenkins et al., 2010; Calzone et al., 2012) and others are specific to local context (i.e. specialty/discipline or health service) (Bonter et al., 2011; Haga et al., 2012; Stanek et al., 2012; Marzuillo et al., 2013; Helman et al., 2016; Chow-White et al., 2017; Groisman et al., 2017; Johnson et al., 2017; McCauley et al., 2017). For example, Chow-White et al. (2017) surveyed oncologists' attitudes towards genomics and McCauley et al. (2017) focused only on physician training in genomics. These are not suitable without adaptation to be deployed across a diverse range of disciplines or services. There are no published surveys that cover the breadth of our research questions in the context of genomics.

We therefore aimed to develop an evidence-based survey that could be disseminated to a national sample of non-genetic medical specialists across diverse sub-specialties in Australia to ascertain their rationale for their practice of genomic medicine with a focus on their training needs. The purpose of this article is to describe in detail the methodology for developing this robust survey. Survey development was informed by literature (Chen and Kim, 2014; Gray et al., 2014; Chow-White et al., 2017; Carroll et al., 2019; Nisselle et al., 2019a; Stark et al., 2019b), theory and qualitative data, has had input from experts for content validity and was reviewed by non-genetic medical specialists representing target respondents for usability and functionality.

#### MATERIALS AND METHODS

#### Research Design

A mixed-methods exploratory sequential (survey development) design was used, involving an initial qualitative phase with key informant interviews (Creswell and Plano Clark, 2017). The qualitative findings then informed development of a contextspecific quantitative survey for dissemination nationally to nongenetic medical specialists in Australia. Data collection using the survey across Australia has been completed and will be reported in a separate publication. This study had human research ethics approval (University of Melbourne, HREC: 1646785). As per the approved research protocol, interview participants gave verbal consent for interviews to be audio recorded, transcribed and for de-identified quotes to be used in publications or reports arising from the research.

# Qualitative Phase: Key Informant Interviews

#### Sample

Two sample groups were approached for key informant interviews: those who provide continuing education in genomic medicine to medical specialists ('education providers') and medical specialists as the target learner group.

#### Education Providers

Individuals and organizations providing genomic education were identified through a desktop audit mapping relevant genomic educational activities or resources in Australia (McClaren et al., 2018). The desktop audit identified 59 distinct genomic educational interventions (37 substantive ongoing programs or resources; 20 postgraduate course or single subjects; two massive open online courses). Where contact information was available on a website or advertisement, convenors of each identified educational intervention were invited to participate in a semistructured interview. Those who responded were sent a plain language statement and consent form, and a phone or face-toface interview was scheduled at their convenience. These interviews with education providers collected information about the participant, including formal qualifications and relevant experience leading to their becoming the convenor of the particular intervention. As well as information about the educational intervention, providers were invited to comment on future education needs in genomic medicine, and to discuss potential barriers and facilitators to meeting these needs.

#### Non-Genetic Medical Specialists

Details of recruitment and data collection with these participants is described elsewhere (McClaren et al., in press). Briefly, a national sample of medical specialists across diverse disciplines was recruited for semi-structured interviews.

Exclusion criteria were:


The interview guide for medical specialists explored current practice of genomic medicine, and interviewees' preferences for the future of genomic medicine relevant practice. Potential barriers and enablers to the integration of genomic medicine into practice may be areas for future educational interventions to address.

#### Data Collection

Interviews were conducted by authors BM or EC and Dr. Zoe Prichard, by telephone or face-to-face, digitally audio-recorded and transcribed verbatim.

#### Data Analysis

A thematic approach was initially taken to analyse transcripts. Authors BM and EC read and re-read the transcripts to identify similarities and differences in the conversations with participants, through constant comparison. The interview guide topics formed the basis of a deductive coding framework that was refined through discussion of emerging concepts which is inductive coding between authors BM, EC, AN, CG, SM (Corbin and Strauss, 2008). NVivo 12 was used to organise the data and manage coding (NVivo, 2018). All transcripts were systematically coded according to the developed coding framework.

### Selection and Refining of Survey Questions

The selection of survey questions and the process of refining these for use in the final survey (including a Delphi review by experts) is shown in Figure 1. The literature were searched for existing surveys that assessed genetic or genomic practice and/or education and training needs of medical specialists; these included peer-reviewed publications, government reports, student theses and conference abstracts, encompassing both published and unpublished surveys (Chen and Kim, 2014; Gray et al., 2014; Chow-White et al., 2017; Nisselle et al., 2019a; Carroll et al., 2019; Stark et al., 2019b). Relevant survey questions were collated and evaluated against concepts identified in the qualitative phase. If there were no, or few, questions in a category, new questions were developed with expert input from the Australian Genomics Workforce & Education working group (Figure 1), to generate a question bank for expert input through a Delphi process. A total of 25 questions were included and/or adapted from prior surveys and three new questions developed for the final survey. The breakdown of these were: n = 15, (Nisselle et al., 2019a); n = 5, (Stark et al., 2019b); n = 3, (Chow-White et al., 2017); n = 2, (Carroll et al., 2019); n = 1, (Gray et al., 2014); n = 1, (Chen and Kim, 2014).

In a traditional Delphi process, experts first generate a list of items and refine over subsequent rounds for relevance and clarity (McKenna, 1994). Given that the initial question bank was informed by qualitative findings and existing surveys, a modified, reactive process was used (McKenna, 1994). The question bank was refined using three rounds to: assess each question for relevance and clarity; modify or develop new questions if required; apply a theoretical framework; and, reduce length (Goodman, 1987; Streiner et al., 2015). Experts

were selected purposefully representing four areas of expertise: non-genetic medical specialties; clinical genetics; genetic/ genomic education and evaluation; and implementation science. The experts were recruited through research and professional networks of the Australian Genomics Workforce & Education working group, plus snowball sampling to ensure national representation. Each round was open for comment for two weeks, with two weeks between rounds for data analysis. The process and data from the Delphi rounds were managed using an online REDCap database hosted at the Murdoch Children's Research Institute (Harris et al., 2009). The online data collection tool simplified the feedback process for experts because REDCap can be used on computers and portable devices at different times, with save and return functions. Using an online approach was also more efficient for the analysis of responses as data could be collated and exported from REDCap.

#### Round 1: Review Relevance and Clarity

Experts reviewed questions in the initial question bank for clarity and relevance to the survey objective, and suggested edits if necessary. Questions were included in subsequent rounds if there was 80% expert consensus to keep the question. To ensure transparency of disparate opinions between professions, data were stratified and prioritized by areas of respondent expertise and re-presented for Round 2 review by the entire Delphi expert group. For example, if a question assessed use of genomics in medical practice and expert consensus was not reached, the data of non-genetic medical specialists were given priority over data provided by other expert groups for that question.

#### Round 2: Reject or Ratify Changes

Experts were shown aggregate Round 1 feedback and reviewed the original and the amended versions of questions. For this round, experts were asked to rate their agreement with any proposed changes and their perception of question relevance and clarity for inclusion in the final survey (Figure 2). Questions were included in the final survey if there was 80% expert consensus to keep. Questions were excluded if there was a unanimous decision to exclude. All remaining questions progressed to the next round for review.

.

#### Mapping Survey Questions to a Behavior Change Theoretical Framework

A rigorously developed survey grounded in theory facilitates translation of the survey across a range of settings. To ensure a sound theoretical underpinning for the survey, remaining survey questions were then mapped to Michie's theoretical framework for behavior change, the COM-B model (Michie et al., 2011). This framework was chosen because it is likely that the data collected with the developed survey will inform educational interventions to target behavior change for the practice of genomic medicine by non-genetic medical specialists. The model proposes that behavior change is a result of interaction between three factors relating to an individual—capability, opportunity and motivation (Figure 3). These factors are then embedded in the Behaviour Change Wheel tool, providing the translational step to bridge findings from data collection into clinical care and therefore appropriate as a theoretical framework for the survey.

medicine (adapted from Michie et al., 2011).

For this study, we defined the target behavior as 'appropriate engagement with genomics', given that use of genomic medicine varies by medical specialty and health service delivery context. Survey questions were mapped were mapped according to the following definitions.

#### Capability

The knowledge and skills required to engage in genomic medicine, ranging from knowledge of basic genetics through to advanced genomics and clinical skills required to refer/order testing, etc. Example survey questions include self-reported genomic knowledge and confidence, ordering genomic tests, and current genomic continuing education.

#### Opportunity

Environmental factors that support or hinder genomic medicine practice and cannot be resolved with education or training, e.g., work environment where genomic testing is implemented, peer support, access to resources. Example survey questions include access to genetic services, funding and education activities.

#### Motivation

An individual's perception of the benefits and limitations of genomic testing and how genomic information can guide patient care. Example survey questions include perceptions of self and genomics, and activities that increase awareness, e.g., exposure through research.

#### Round 3: Reduce Survey Length

Questions were then grouped by the initial qualitative phase concepts and/or COM-B domains to identify any redundancies. Delphi experts then ranked questions by importance within these groups to shorten the survey; an example of this process is shown in Figure 4.

#### Piloting the Survey: Determining Face Validity and Functionality

First, face validity was confirmed. Members of the Delphi group nominated non-genetic medical specialists from their professional networks practicing in a range of settings, to

review the final survey. Participants provided written or verbal feedback at the end of the survey on any aspects they found difficult to answer and/or could be improved; verbal feedback was collected in 'talk-aloud' cognitive interviews (undertaken by EK) at a mutually convenient time (Czaja and Blair, 2005).

The final online survey in REDCap was then robustly tested for functionality. Non-genetic medical specialists who had been contacted to participate in the key informant interviews (qualitative phase) were re-contacted and invited to pilot the survey from 23 January to 15 February 2019. These specialists were asked to complete the survey in full to trial data capture systems in REDCap using a variety of devices and browsers; respondents could also provide optional feedback on their survey experience.

# Deploy the Survey: Data Collection

The online survey was open from February to September 2019 (manuscripts in preparation). A multipronged recruitment strategy aimed to reach as many medical specialists and trainees as possible across all career stages, Australian regions and specialties (excluding clinical genetics and general practice, as noted above). Additionally, this survey was not deployed to oncologists as the questions on genomic testing focus predominantly on germline testing; adapting the survey for oncology is the focus of further work. The survey was advertised via medical colleges, societies and hospital newsletters, member email distributions lists, internal communications and/or social media channels, and via Australian Genomics investigator networks and social media channels. All advertisements included a prompt to forward the survey link to relevant colleagues and respondents were also encouraged to share the survey among their Australian health professional networks.

# RESULTS

#### Qualitative Phase: Key Informant Interviews

Contact details of 39 education providers were obtained from the identified educational activities and 32 convenors responded and were interviewed (Janinski et al., 2018). Interviewee qualifications, which could be multiple, included nine clinical (genetic counseling, medical specialty, nursing or allied health), four pathology, and 24 doctorates (PhD) in science, social science or bioinformatics. Four interviewees held a tertiary qualification in education. The providers developed and/or delivered a wide range of educational interventions. These were (from most common to least common): continuing professional development (CPD) activities, formal education (e.g., university courses) or online courses/resources. Attendees or users of these interventions ranged from undergraduate students (e.g., medical, science, bioinformatics) to non-genetic health professionals, medical scientists and genetic specialists.

As described elsewhere, 240 medical specialists were contacted for interview and, of these, 86 were interviewed (McClaren et al., in press). The medical specialties included: anesthesiology, cardiology, dermatology, endocrinology, fetal medicine, general medicine, hematology, immunology, infectious disease, intensive care, nephrology, neurology, neuropsychiatry, obstetrics & gynaecology, oncology, general pediatrics, pathology, and rheumatology.

Interviewees were classified using their descriptions of current practice of genomic medicine and perceptions of their level of genomics experience, as: 'novice' (no (or rare) use of genomics in clinical practice and/or; no involvement in genomics research and/ or; ambivalence towards continuing education in genomic medicine), 'interested' (infrequent use of genomics in clinical practice and/or; some (or rare) involvement in genomics research and/or; interest in, but perhaps not attendance at, continuing education in genomics) or 'experienced' (current use of genomics in clinical practice and/or; active involvement in genomics research (molecular or clinical) and/or; participation in continuing education in genomics). These classifications, as well as the medical specialty, are shown as participant descriptors throughout to give context for illustrative quotes. The spread of self-reported genomic experience was: novice, n = 29 (34%); interested, n = 34 (39%); and experienced, n = 23, (27%).

Analysis of the transcriptsfrom these 118interviewswith education providers and medical specialists resulted in five emergent ('5P') concepts, which also formed the framework for survey development:

	- a. future roles and models of care in genomic medicine; and
	- b. continuing education in genomic medicine.

Interview quotes are used in the sections below to illustrate the concepts. Some quotes have been truncated for readability without changing the meaning, indicated by "…".

#### The '5Ps': Key Concepts Relevant to the Integration of Genomic Medicine Into Clinical Practice Personal Characteristics

Personal characteristics of the medical specialist participants influenced their description of their readiness for genomic medicine. These included: medical specialty, types of patients seen (adult/children; public/private settings) and years of clinical practice. Participants also described how teaching roles contributed to their understanding of genomics.

I used to teach undergraduate genetics for years … I would not, by any stretch of the imagination, attest to be an expert in these things, but I probably have a better background than most of my contemporary colleagues working here, just because of what I had done along the way. [MS36, mid-career, interested, nephrologist]

#### Current Practice of Genomics in Clinical and Research Settings

Clinical practice of genomic medicine ranged from limited to regular use amongst the medical specialist participants interviewed. Participants from specialties including cardiology, hematology, neurology, intensive care and oncology described how genomic medicine has high relevance to patient care.

My clinical practice is predominantly epilepsy and therefore, everything epilepsy has some genetic relationship, be it primarily genetic or the structuralvein abnormalities that also have genetic bases. So I guess a lot of my consultations do involve … some discussion at some point about the genetic contributions to the aetiology, be it complex genetics or single genes or somatic … it's a big part of my practice, not always possible to test for genes, but even just discussions with families around our understanding of the genetic contribution. [MS27, senior, novice, neurologist]

If a patient comes along, for example with melanoma, there's a handful of specific mutations that are known drivers of that disease. And we perform genomic studies to see whether those mutations are present. If they are present, then those mutations indicate specific therapies. If they're not present then we don't give the patient those therapies. [MS49, senior, interested, oncologist]

Other participants described their perception of a lesser relevance of genomic technology approaches to their care of patients in fields including immunology and nephrology.

We've been slow to move into this field in that, historically, a lot of the genetic disorders that we receive have come through to us from the pediatricians, often with a diagnosis, or there hasn't been (a need for) a genetic diagnosis, because (the patient) would either have a clinical diagnosis and the management would be just a pragmatic one of, trying to fix whatever was wrong or trying to manage the complications of their kidney impairment and therefore actually having the genetic diagnosis wasn't changing our practice. [MS36, mid-career, interested, nephrologist]

It was evident from the interviews that some participants had gained knowledge and skills about genomics through avenues other than their clinical role. In particular, participants gained experience through their laboratory or clinical research involvement.

I've been a (funding body details) researcher for the past 14 or 15 years. I have a background in genetic analysis … and we associate polymorphisms with risk of skin cancer, including melanoma. So I have a fairly good understanding of genetics and risk association, but not in a clinical setting as such. [MS22, senior, interested, dermatologist]

The most commonly described approach to current practice of genomic medicine was to refer patients to a specialist Genetics service which is consistent with their clinical guidelines to promote appropriate requests for genomic testing.

My two main areas of interest are gastrointestinal malignancies and breast cancer and certainly I'm sort of well aware of the guidelines for familial cancer screening and often refer a number of patients to the familial cancer centres. [MS35, early, novice, oncologist]

We have a very strong link with the Clinical Genetics unit….My colleagues and I have found it very helpful to refer first rather than to order the tests straight up. [MS64, mid-career, interested, clinical immunologist]

I look after a lot of children with genetic issues, or children with undiagnosed syndromes or medical conditions that are unexplained….I would, order the microarray and (single gene test for) fragile X. And then if I am concerned and haven't found results from there, from that then I send (refer) to Genetics. [MS54, early, novice, pediatrician]

Fewer participants described specialist-led clinics that had a particular emphasis on the inherited or genetic aspects of patient care.

In the clinic I have a dedicated interest in hereditary endocrine conditions so my clinic is skewed towards genetic conditions. [MS03, senior, interested, endocrinologist]

#### Perception of How Proximal Genomic Medicine is to Practice

Participants were asked to describe their perception of how near in the future genomic medicine was likely to be part of their clinical practice. For some medical specialists, genomic medicine was not something they anticipated in their practice for quite some time.

From a clinical day-to-day practice perspective, it doesn't really have much role at present. [MS24, senior, experienced, hematologist]

Within our clinical practice we would use genomics mainly in the context of endocrine tumours but there is no, currently, provision of testing for genetic mutations in endocrine tumours where we are in (name of state) ….We are just starting to use panel sequencing for bone fragility but this is despite the fact that we showed in our research that you could do it just as efficiently with whole exome sequencing, which costs a whole lot less. [MS05, senior, experienced, endocrinologist]

The varied proximity of genomic medicine to different specialties was echoed in the perspective of education providers.

In the renal space for example, genomics and genetic testing hasn't hit them in a big way, you know whereas a cardiologist is much more aware of genetic testing and the benefits and limitations and all of that in their field. [EDU010, convenor of ongoing program/resource]

There was, however, a sense that genomic medicine would become part of clinical practice, or was already being established.

It is going to pervade everything we do……particularly as it becomes more and more mainstream and more equipment becomes cheaper and cheaper it is going to be more diagnostic, so personalized medicine and diagnostics in hospitals. [EDU018, convenor of university course/subject]

In the last 2 or 3 years it's come up more……I think it's a field in its infancy, it's growing and it's going to find more applications. And when we know more about it, we're likely to use more, and I think it's certainly got a role and it's only going to expand. [MS60, mid-career, interested, intensivist]

Perception of Preparedness: (Competency and Confidence) Participants described a perception that medical specialists were un- or under-prepared for future practice of genomic medicine.

Many healthcare professionals not traditionally involved in genetic testing … their basic genetics 101…is not very strong, probably haven't used it for a very long time. The genetic potential that they learned 10–20 years ago was very much the classical style of genetics rather than what we know now from when a human genome project was finished … It's creating a lot of confusion … in practice as a healthcare professional. [EDU024, convenor of MOOC]

Medical specialists identified that developing confidence to practice would be important for future integration of genomic medicine into clinical care.

My confidence with the terms of the referral and feeling confident about what information I need to provide is much higher than my confidence in interpreting information … we absolutely rely on the expertise of the people writing the report in terms of variations of unknown significance … my confidence in terms of interpreting a VUS [variant of unknown significance] is very limited. [MS47, early, experienced, neuropsychiatrist]

#### Preferences: Future Roles and Models of Care in Genomic Medicine

Medical specialists had a preference that if they are to practice genomic medicine in the future, then there should be a multidisciplinary team in place for optimal patient care, in particular where the testing may have a predictive application.

Our genetic tests are ordered in conjunction with a multidisciplinary clinic that I run with my clinical genetic colleagues, and genetic counseling is conducted as part of that clinic. It is especially true for cancer syndromes. The Clinical Genetics department (here) has instituted what I think should be the gold standard process of gatekeeping where they will allow specialists from outside Clinical Genetics to order a genetics test on the proviso that adequate genetic counseling has been provided to the individual with the syndrome and that any positive test will then trigger Clinical Genetics review of predictive testing of family members … There is simply not enough space in Clinical Genetics to work but the clinical geneticists at our hospital are confident enough in the endocrinologists to be able to order a test for someone with a clinical syndrome, a phenotype where the risks of genetic testing are low because if you already have the phenotype you can't further damage the person by a molecular diagnosis. It is testing the asymptomatic individual where the risks have to be very carefully articulated. [MS03, senior, interested, endocrinologist]

They (Clinical Genetics department) certainly assist in making sure we order the right tests from the right lab … I think that's quite tricky. And they also have counselors … which means that I feel more confident that my patients getting the right information … And I certainly intend to keep using them because I think the patient gets better care. [MS64, mid-career, interested, clinical immunologist]

(We need to) encourage medical specialists to take this on and take it and work in partnership with each Clinical Genetic service … have some realization of the different types of tests and, "Gosh, I need to talk to someone about this'. Where a panel is appropriate here, and a single exome there, and a whole genome for that. [EDU019, senior, convenor of ongoing program/resource]

The emergence of genomics experts within specialties was proposed as a future model of practice in which a specialist gains specific expertise in genomics as relevant to their patients.

In my opinion the best model of care in terms of integrating genetics into clinical practice is to have specialist-led Genetics clinics where people…. Just like the specialists in cardiology who do angiograms and stick tubes in groins, I don't do that, that's not my specialist area. I'm a cardiologist but I don't do that. There should be a specialist for cardiology genetics, a specialist for neurology genetics, that sort of thing. That model of a specialist-led Genetics clinic is the best model because the phenotype is so important, you have to get the phenotype right before you can interpret any genetics information. [MS06, senior, experienced, cardiologist]

There just aren't enough geneticists or genetics counselors to deal with all the data that's going to be coming in in the next few years. I'm a strong believer that each discipline needs to understand the genetics of its disorders going forward. [MS23, senior, experienced, neurologist]

This was further emphasised by medical specialists wanting to manage genetic investigations for common or 'minor' conditions.

I think that I am quite capable of speaking to people about testing their family without involving genetic counselors. I actually don't need to have them involved for those minor genetic disorders. So it really depends on, I think, the clinical significance of the genetic disorder. [MS24, senior, experienced, hematologist]

#### Preferences: For Continuing Education in Genomic Medicine

The most valued approach described by educators and medical specialists was for learning through continuing education when there was the opportunity to gain 'hands-on experience'.

I think that you really need hands-on experience, you have to have a mixture of didactic lectures, case examples, and hands-on experience, people rotating through workshops … You don't have to curate, but getting in there, and doing a couple helps you understand the process, helps you understand the complete process … If people understand the process, then I think they get much more out of the MDT (multidisciplinary team) meetings. [EDU007, convenor of ongoing program/resource]

Speaking as a clinician it would be important to me that it [education] had a practical focus … It could still be lecture-based or small group-based … But you know, clinically, practical-focused. [MS43, senior, interested, nephrologist]

There was also a strong preference from participants that continuing education is delivered in a clinically relevant way, although they recognized that this was challenging as different medical specialties, and individual specialists, would have different perceptions of what is clinically relevant.

I talk to a lot of people in my role and this goes all the way from genetic counselors to clinical geneticists, medical specialists … the information they require I find really differs depending on what field they're in … the different fields and different specialists are at very different stages and requirements. [EDU010, convenor of ongoing program/resource]

Everyone would love to have time to get educated but the reality is attendance to that sort of activity often comes second, particularly when you've got busy clinics and patients coming through. But if you have a patient who is really challenging you, and you have the opportunity to improve the management of that patient if you go along to this tumor board (meeting), then you all of a sudden have another reason why you should attend when the forum is integrated with basically patient care. [MS49, senior, interested, oncologist]

As long as it's clinical, you know, we all get basic genetics at university but it's sort of how it's applicable to clinical practice that matters most for clinicians. [MS51, mid-career, interested, neuropsychiatrist]

#### Overlap and Intersection of the 5P Concepts

The 5P concepts that formed the framework for the survey development can be considered separately as shown in the section above, but they do intersect and overlap. For example, as shown in Box 1, a medical specialist perceives genomic medicine to be very proximal to their practice because they currently include genomic investigations in their usual care. In doing so, they have experience in requesting genomic investigations, receiving reports and interpreting results for

BOX 1 | An example of overlapping 5P concepts, illustrated with quotes from an experienced, mid-career clinical immunologist who sees adult patients [MS65].

Genomics proximal to their practice (special interest in primary immunodeficiency). Proximity motivated them to upskill (to become prepared)

One of the main areas, I think, with primary immunodeficiencies is clearly the genomics and that side of the field. So I started to get interested in it from there.

Immunologists are quite diverse so there are some people who don't do a lot of immunodeficiency or autoinflammatory and deal mostly with allergies, and that's their interests. But certainly there's a lot of interest around where I am.

Having experience with genomic testing in their practice has contributed to perceived preparedness (competence and confidence)

I probably do have a reasonable understanding of the technology, and as I said, I do have some exposure to the technology through my other work (in pathology)...The technology itself is something that takes a bit to get your head around. And I obviously see the type of immunodeficiency patients, I've got some clinical involvement, so I think I am managing to keep up with it (genomics).

Preference for future model of care is influenced by their practice; experience has suggested a multidisciplinary model of care works best and they want this to continue because it provides opportunity to learn from peers (preference for education).

It's a complicated thing immunodeficiency. You need someone with expertise in that as well as someone with genetics expertise...I think there'll be more collaboration (going forward).

It's always helpful to have collaboration with the, sort of genetic scientists, clinical geneticists and the involvement of the genetic counselors in the process. All of those things definitely help (to navigate genomics), it's kind of a hard to do as a single practitioner.

their patients. This experience may contribute to their perception of being prepared. Medical specialists in these contexts may have different preferences for continuing education than other specialists who do not currently request genomic investigations and/or do not anticipate doing so in the near future (genomic medicine is distal to practice).

#### Quantitative Phase: Survey Development The Delphi Review

Twenty-six experts were contacted to participate in the Delphi review of the initial survey question bank. Of those invited, 22 agreed to participate (Figure 1). The final Delphi expert group comprised six medical specialists, nine genetics specialists, six genomic educators, and one implementation scientist from across Australia. Of the 22 experts recruited, 17 completed all three rounds of the modified Delphi process. The numbers of questions at each round are shown in Table 1. See Table 2 for an example of a question Delphi feedback and modifications throughout the rounds.

#### Round 1: Review Relevance and Clarity

All experts completed Round 1, which included 45 questions for review. The experts reached agreement (consensus) on relevance and clarity for five questions, which were retained to be included in the final survey. For three questions where consensus was less than 80% and qualitative feedback unanimously excluded the questions, these were removed. For the remaining 37 where there was less than 80% agreement but qualitative feedback was varied, written feedback from prioritized perspectives was used to amend questions (see Table 2 for an example). Questions that addressed similar concepts were combined and the Delphi experts suggested four new questions, which brought the total number of questions requiring further review to 27.

#### Round 2: Reject or Ratify Changes

Eighteen experts completed Round 2. Of the 27 original questions presented for review, 25 were included, two were excluded. The three extra questions suggested from Round 1 were also reviewed and agreement reached to include these. In total, after Round 2, there were 33 questions remaining (five questions had already reached consensus in Round 1).

#### Map Questions to the COM-B Model

The results of mapping of questions to the COM-B theoretical framework (Figure 3), are shown in Table 3.

#### Round 3: Reduce Survey Length

In Round 3, the survey was reviewed for overall length to be mindful of the time it would take for respondents to complete. All demographic items and three questions assessing involvement in genomic research, awareness of clinical guidelines and confidence in genomic knowledge, were deemed essential for inclusion by the Australian Genomics Workforce & Education working group and so were not reviewed by the Delphi group for potential exclusion from the final survey. The remaining questions were organised into groups based on the 5P concepts and/or aspects of the COM-B domains (Table 3).

Seventeen Delphi experts ranked the questions within each subset by preference of inclusion in the final survey. Where consensus was reached, the questions considered most important were included in the survey (Table 1). Where there was a lack of consensus, the Australian Genomics Workforce & Education working group reviewed rankings and feedback to decide which questions to include in the final survey.

# Piloting the Survey

#### Face Validity

To obtain feedback from non-genetic medical specialists (the target population for the final survey), the Delphi group nominated colleagues who were then invited to provide insights on face validity. Five participated in the talk-aloud 'cognitive' interviews and one completed written feedback (Czaja and Blair, 2005). These medical specialists were from three Australian states and five specialties. Feedback suggested alterations to question response options (e.g., lists of specialties) and gathering more in-depth information about contact with genetic services and level of engagement with education and training. Questions to address the last two suggestions were sourced from the GEC-KO Family Medicine Genetics Survey (Carroll et al., 2019).

Changes were also made to the survey at this stage to ensure data quality, e.g., adding a question to exclude respondents who did not practice clinically.

#### Functionality

The survey was sent via email to 240 addresses of those invited to initial key informant interviews, with 29 surveys completed online. Of these, 13 individuals provided additional detailed feedback on their use of the survey (ten written, three verbal). Feedback related to survey functionality in REDCap and clarifying ambiguity of questions or instructions. Table 4 provides illustrations of feedback and subsequent amendments during functionality


Questions are mapped to more than one concept or domain; C, capability; O, opportunity; M, motivation; B, behavior (Michie et al., 2011).

#### TABLE 2 | An example of question evolution using a modified Delphi process and mapping questions to the COM-B framework.

#### Round 1 (relevance and clarity)

Original question

	-
	-
	-
	-




#### Round 2 (ratify or reject changes)


(Continued)

#### TABLE 2 | Continued


C, capability; O, opportunity; M, motivation; B, behavior (Michie et al., 2011).

testing. For example, during an interview, a medical specialist commented that they did not know what a 'rollover definition' was or how to use it despite this being explained in the introduction to the survey. These rollover definitions were crucial for appropriate and consistent interpretation of terminology in questions. To ensure definitions were read by all participants, the definitions were therefore also added underneath each question. Other minor changes were made to survey questions to improve participant understanding of questions before finalizing the survey for deployment.

The final survey is included as 'Supplementary Data Sheet 1 final survey' and, in sum, consisted of 28 questions, noting the source of questions or topic items from existing surveys (Table 1 and Table 3).

#### DISCUSSION

This mixed-methods study describes the development of a survey designed to measure a wide range of non-genetic medical specialists' current practice of genomic medicine and their preferences for future practice and continuing education in genomic medicine. This instrument was used to survey a national sample of non-genetic medical specialists practicing in Australia (manuscript in preparation).

A strength of this survey is that it has an embedded theoretical framework and is informed by qualitative data collection. Using the concepts that emerged from the qualitative data as a framework for the survey has ensured that identification, selection and development of survey questions covers the breadth of topics related to current practice and needs for continuing education in genomic medicine. The qualitative analysis demonstrated the way in which these concepts can be considered individually but also importantly that there is overlap; sections of interview transcripts could be coded at more than one overarching concept (Box 1).

This overlap is also evident in the final survey questions (Table 3), which means that the patterns seen in the qualitative work, and how participants discuss issues of continuing education and future practice, have been maintained in the development of survey questions. The final survey is a flexible tool that can assess individual or multiple concepts simultaneously. The survey is therefore useful for a range of research questions. For example, using practice questions (single question or suite of questions) if data are required to demonstrate the current non-genetic medical workforces' use TABLE 3 | Final survey questions mapped (shown with an X) to concepts from the initial qualitative findings and the domains of capability, opportunity, motivation and behaviour of the behaviour change wheel theoretical framework (Michie et al., 2011).


Existing surveys were reviewed and relevant questions were selected (and modified in the Delphi rounds) for inclusion in the developed survey. Original sources were: <sup>a</sup> (Nisselle et al., 2019a); <sup>b</sup> Crafted by Australian Genomics Program 4 Working Group; <sup>c</sup> GECKO survey (Carroll et al., 2019); <sup>d</sup> (Stark et al., 2019b); <sup>e</sup> (Chow-White et al., 2017); <sup>f</sup> (Gray et al., 2014); <sup>g</sup> (Chen and Kim, 2014); C, capability; O, opportunity; M, motivation; B, behaviour (Michie et al., 2011).

TABLE 4 | Examples of pilot survey feedback and amendments on ecological validity and functionality.


of genomic investigations; or only using the questions mapped to preparedness for non-genetic medical specialists in a particular setting.

Basing the survey development in an emergent qualitative framework and a theoretical framework means that the survey can inform the selection of, or identify the need for development of, educational interventions to support non-genetics medical specialists as they develop competence to practice genomic medicine. Data from this survey can determine if and how educational interventions need to be tailored to the needs of individual sub-specialties and even individuals within those groups, based on clinical need. The data collected using this survey will provide much needed detail for education providers about which specialties are likely to engage with and participate in education interventions; this will enable resourcing to be focused on creating specific elements. Resultant interventions should consider evaluating their learning objectives against core competencies such as those identified by National Coalition for Health Professional Education in Genetics<sup>1</sup> (NCHPEG) which set out three domains from which a clinician can assess their practice and need for further education and training. However continuing education is not the only answer; a suite of interventions will be required for the effective integration of genomics into clinical practice (McClaren et al., in press; Paul et al., 2018; Crellin et al., 2019). This survey can contribute to identifying other key factors for which interventions may be targeted.

The modified, reactive Delphi process used in developing the survey allowed input from a geographically disparate, heterogeneous sample of experts. Individual feedback was collected in a structured manner using an online platform. Importantly, using a Delphi approach provided the opportunity for evaluation of group views during Round 2 to take the input beyond the individual and make use of the collective expertise. Further, in Round 3, questions were ranked to inform decision-making about the inclusion or exclusion of questions for the purpose of evaluating the length, and therefore the time required by potential respondents to complete the final survey.

The importance of including a pilot phase in survey development was highlighted in our study; ensuring functionality with future users is critical and assumptions must be tested, such as presuming users would understand how to access 'rollover definitions' in the online survey platform. A possible limitation of the functionality testing approach we used is that we had a response rate of 12% in this final stage of survey development. This level of response is not uncommon in surveys with health professionals; Selkirk et al. (2013) report a similar response rate (13%) for email invitations of physicians to complete a survey about preparedness for genomics. Of the 29 users who tested functionality in our pilot, only 13 provided additional feedback. Ideally, testing functionality of a survey would be with larger number of the target population.

Comparing the qualitative data collected from education providers and non-genetics medical specialists proved challenging, as the data had different emphases: the education providers had few comments on current and future practice of genomic medicine, while the medical specialists had generally not participated in continuing education for genomic medicine so their preferences reflected hypothetical views rather than what has worked well for them in learning about the application of genomic technologies in their practice. We therefore prioritized the perspectives of different expert groups during the Delphi process for particular questions. This assumption was decided on as a way to resolve disparity in views about the survey questions but may have biased the results of the Delphi process. The Delphi experts were all very engaged with genomics, even across their perspective groups, and therefore may not represent fully the perspectives of the target group of all medical specialists.

Use of iterative review and applying theory in survey design has been previously described. Jenkins et al. (2010) used rounds of iterative review to develop a national survey of US physicians in genomics, based in Rodgers' Diffusion of Innovation theory. This theory was chosen by the authors because it is a useful framework to predict adoption of genomics and to guide the selection of genomic education interventions to support clinical practice. By contrast we selected the COM-B model to design a

<sup>1</sup> https://www.jax.org/education-and-learning/clinical-and-continuing-education/ ccep-non-cancer-resources/core-competencies-for-health-care-professionals

survey that would measure, at the level of the individual, concepts that influence their behavior in appropriate engagement with genomics in clinical practice. A rigorously developed survey grounded in theory facilitates translation of the survey across a range of settings, which can be used to draw comparisons across these settings. For example, Jenkins et al. (2010) then adapted their survey for nurses using a modified version of the methodology (Calzone et al., 2012). We are similarly adapting our survey for oncology and international settings using qualitative interviews with key informants to review the current survey questions and assess each for relevance and suitability, such as nuances of germline versus somatic testing and local health service contexts. The strength of our survey development process based in qualitative and theoretical frameworks means that changes to specific wording of questions can be made according to the setting in which the survey will be used but the questions can still be classified using the framework concepts, making comparisons between settings possible. Future users of the survey may review items for relevance to research questions and local contexts or needs, then amend or add items.

As has been previously described, education is not the only answer for the changes to behavior needed for non-genetic medical specialists to competently and confidently practice genomic medicine. Educational interventions, however, will be and should be used as part of such strategies (Nisselle et al., 2019b). For education to be part of any effective strategy, interventions need to be evidence-based, with focus and content informed by understanding of the needs of the target audience. These needs, as shown by our qualitative data, are related to the characteristics of the specialist, their current practice of genomic medicine, their perception of how proximal genomics is to their practice, how prepared they feel they are to practice and their preferences for future clinical practice and future continuing education. We have created a robust, data- and theory-informed survey which captures not only levels of experience, practice of genomics and preferences for education but also the challenges around engaging with education. Survey data will provide evidence for education providers to inform their interventions so that effective education can be available to contribute to establishing a medical workforce that is literate in genomics and more confident to competently practice genomic medicine.

# DATA AVAILABILITY STATEMENT

The datasets generated for this study are available on request to the corresponding author.

# REFERENCES


## ETHICS STATEMENT

The studies involving human participants were reviewed and approved by University of Melbourne, HREC: 1646785. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

# AUTHOR CONTRIBUTIONS

BM, AN, SM and CG conceived the idea and design for the study, and BM and EC conducted the interviews and analysed the data. EK and AN led the survey development and the Delphi review. SM and CG, and the Working Group, provided advice throughout the study, and assisted with recruitment of experts for the Delphi group. BM drafted the manuscript and all authors revised drafts, approved the final version and agree to be accountable for all aspects of the work.

# FUNDING

This work was supported by the Victorian Government's Operational Infrastructure Support Program and a grant from the Australian National Health & Medical Research Council (GNT1113531); the contents are solely the responsibility of the individual authors and do not reflect the views of the NHMRC. EC was supported by the Helen R. Freeman Scholarship.

# ACKNOWLEDGMENTS

The authors acknowledge the following contributors: Dr. Zoë Prichard who conducted the interviews with education providers; the Australian Genomics Workforce & Education Working Group members not listed as authors (Kate Dunlop, Dr. Debra Graves, A/Prof. Ainsley Newson and Bronwyn Terrill); the Delphi experts; and students and interns who co-coded interview transcripts using the 5Ps coding framework (Monika Janinski, Cecilia Pynaker, Jacqui Hunter, Jaitika Duggal).

# SUPPLEMENTARY MATERIAL

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fgene.2020. 00059/full#supplementary-material

SUPPLEMENTARY DATA SHEET 1 | Australian Genomics 5Ps survey.docx.

oncology, cardiology and family medicine. BMJ Open 1, e000110.10.1136/ bmjopen–2011-000110. doi: 10.1136/bmjopen-2011-000110


Conflict of Interest: CG and SM are co-editors of the Research Topic where this manuscript is featured. All authors share their affiliations with CG and SM. CG and SM were not involved in the editorial or peer-review process; this was overseen by the third co-editor who is not affiliated with the authors' institutes.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2020 McClaren, King, Crellin, Gaff, Metcalfe and Nisselle. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

digital media

of impactful research

article's readership