Examining a Decade of Research in Online Teacher Professional Development

This literature review documents research in the area of online teacher professional development (oTPD) and seeks to inform developers and facilitators on the complex and unique empirical indicators that are important in designing, developing, implementing, evaluating, and researching oTPD. The 73 studies analyzed in this review suggest that the research in oTPD is progressing toward more rigorous empirical methods and theoretically grounded design, implementation, and evaluation. Research in oTPD is moving forward in more sophisticated ways and adding to our understanding of high-quality practices that engage teachers in meaningful teacher professional learning in online contexts.


INTRODUCTION
Online teacher professional development (oTPD) has experienced such explosive growth in the last decade that the production of teacher professional development (TPD) courses and programs has outpaced the research. The COVID-19 pandemic has hastened the construction of, participation in, and need for online courses and programs . The primary subject of this new pandemic crop of oTPDs aims to teach teachers how to teach online . Surely oTPD then ought to model the best online teaching design features and practices. What is needed is a clearer identification of the most suitable oTPD design features and practices that support teacher change and student learning (Dede, 2006). This literature review pursues such clarity by culling and analyzing pre-pandemic research focused on identifying strong design features and practices to support teacher change and student learning. This review has the potential to inform developers, researchers, and facilitators about the complex and unique empirical indicators that are important in designing, developing, implementing, evaluating, and researching oTPD. The following research question guided this review: What themes from select pre-pandemic studies of oTPD emerge and how do they inform the design and practices of future oTPD?
We first describe the pre-COVID-19 pandemic rationale for oTPD and report on multiple calls for a research agenda to guide research on oTPD, articulate potential research agendas, and identify gaps in the research. In the methods section we describe the methodology and selection criteria used in designing and conducting this review. In the findings section, we first concisely summarize the select characteristics of the research on oTPD reviewed in this article. Next, we examine the common practices and recommendations that emerged through a thematic synthesis of the research. In our discussion, we conclude by identifying gaps in the status of research on oTPD and discussing implications for future research, providing findings on overarching categories, potential guidelines, and the progress of research on oTPD.

Pre-COVID-19 Pandemic Rationale for oTPD
Over the past decade, the number of oTPD programs has continued to rapidly grow because the perceived benefits of improved access, flexibility, networking, and lower costs of offering teacher professional development (TPD) online have been accepted as accurate and well-entrenched. Additionally, it is believed that the instructional effectiveness of the modalities is comparable (Dede et al., 2009;Brooks and Gibson, 2012;Fishman et al., 2013;Albers et al., 2015;Stevens et al., 2016;Darling-Hammond et al., 2017). Providers of oTPD believe they can offer benefits of access and delivery, reaching more people at lower costs (Albers et al., 2015;Darling-Hammond et al., 2017).
Schools, districts, and nations often adopt oTPD as an attractive solution because they want to draw upon resources not available locally or even nationally, and they see it as an effective way to reach teachers in rural locations (Brooks and Gibson, 2012;Stevens et al., 2016). Those generating oTPD believe it has the benefit of providing increased opportunities for broad collaboration and communication across distances and see it as a way to offer just-in-time inservice support while accommodating teachers' full schedules. Furthermore, online formats offer additional possibilities for individualized mentoring and support within online communities of practice, and the asynchronous, text-based, and dialogic character of online instruction has the potential for engaging teachers in rich and ongoing reflections on their practice (Dede et al., 2009;Borko et al., 2010).
Apart from these often-cited general advantages, states and districts frequently turn to oTPD to address particular pedagogical, content, or standards concerns on a broad scale (Cavanaugh and Dawson, 2010;Dash et al., 2012). Teachers in more specialized areas such as Special Education, Gifted and Talented Education, or English as a Second Language Education also see oTPD as a cost-effective solution for informing teachers about advances in content, pedagogy, and practices specific to these areas (Little and Housand, 2011;Chu, 2013). The increase in online platform availability and diversity is also seen by some as crucial for preparing not only 21st century students but building 21st century skills in teachers (Saavedra and Opfer, 2012).
The COVID-19 pandemic has radically shifted the conversation around the rationale for oTPD. As has been described above, the pre-pandemic rationale for oTPD focused on identifying and demonstrating various logistical benefits of and instructional effectiveness of oTPD. The necessity for social distancing and other public health preventative measures makes the turn to oTPD a matter of pure logistical necessity to promote public health. Discussions of oTPD's benefits and effectiveness certainly must still be carefully considered (thus the need for this literature review), but the COVID-19 pandemic has added an entire new level of urgency to the construction of, participation in, and need for online course and program options in education.

Debates on TPD Research's Similarity to oTPD
Research has not kept pace with the creation and delivery of oTPD programs and courses. There is a lack of systematic research, clear guidelines, and general consensus regarding optimal characteristics of oTPD and practices that effectively support teacher change and student learning. Additionally, high demand, access and delivery requirements, and higher upfront development costs may drive many oTPD designs and their specifications in lieu of theoretically or empirically guided practice (Vrasidas and Zembylas, 2004). In this absence, general principles and guidelines relevant for in-person TPD are frequently assumed as applicable to online courses and programs without deeply considering whether specific researchbased practices actually transfer well to online environments.
The assumption of similarity between in-person TPD and oTPD is debated in the literature. Some key authors suggest that the gaps in TPD research are similar to the gaps in oTPD and can be addressed through research regardless of the TPD's structure or context, suggesting that research in TPD has the potential to contribute to our knowledge concerning oTPD and vice-versa (Desimone, 2009;Borko et al., 2010). On the other hand, while recognizing that the two approaches to TPD have many common themes, Dede et al. (2009) argue that the implementation of each approach, including effectiveness of specific strategies, differ significantly based on modality, substantiating a specialized research agenda for oTPD.
Some research suggests that concepts and ideas related to TPD influencing teacher change as well as core features of high-quality TPD may be transferable to online and technologymediated settings . However, since implementation of specific TPD features and concrete practices in online settings may substantially vary from in-person models, those findings and related implications may be much more modality-and contextually dependent (Opfer and Pedder, 2011;Moon et al., 2014).
The general principles guiding a research agenda in TPD, such as phases of research (Borko et al., 2010) and recommended types of studies and methodologies (Dede et al., 2009;Borko et al., 2010;Opfer and Pedder, 2011) may transfer to oTPD settings. Conversely, when it comes to specific methods and tools utilized in studying oTPD, these practices may be modality-specific. The acknowledged, though debatable, relevance of general TPD principles and guidelines to oTPD leads to calls in the literature for more rigorous research, particularly highlighting research unique to oTPD where applicable.

Multiple Calls for High Quality Research in TPD and oTPD
Beginning in 2009, key authors in the field of TPD, including oTPD, called for researchers to step up their rigor, produce more scholarly investigations of oTPD based in theory, and use highquality research methods and methodologies, including more empirically valid methods of studying TPD (Dede et al., 2009;Desimone, 2009;Borko et al., 2010). Furthermore, research on TPD both in in-person and online environments should take into account issues of TPD design as well as enactment and implementation in order to understand whether certain program designs work and also provide evidence of why they may be effective (Borko, 2004;Dede et al., 2009;Borko et al., 2010). Dede et al. (2009) called for a focused oTPD research agenda, highlighting key oTPD areas in need of research based on a previous review (Dede, 2006). Both Dede et al. (2009) andBorko et al. (2010) reported that TPD/oTPD studies often relied on less than rigorous methods. They called for empirical research that would carefully study better design elements of oTPD, ...[which] focuses on types of knowledge that are missing or lacking in the current literature for improving oTPD and on assessment of oTPDs' strengths and limitations. In particular, [they] highlight two areas that should serve as the nucleus of future research ventures: (a) research questions that address understudied areas and (b) design and methodological strategies for studying these questions (Dede et al., 2009, p. 9).
Thus, they recommended that researchers identify and seek to study areas that are ignored or that receive little attention. However, a comprehensive understanding of what has been studied in terms of oTPD would be needed to carry out their recommendations.
In the past, oTPD research has also struggled with issues of causal claims, lack of replication across contexts, and non-linearity of teacher learning and teacher change processes. More advanced methodologies that recognize the complexity of the phenomenon of oTPD, taking into account the systems, situatedness, and contextualized nature of schools and teacher change are also needed. Dede et al. (2009) recommended generative, problem-solving, iterative methodological approaches such as design-based research (DBR) capable of producing fine-grained theoretical understandings.
They asserted that such research should seek empirical evidence of best practices and explore what kind of measures most effectively provide evidence of the results of oTPD. They called for research designed to examine the teacher-student interaction in these settings and the kinds of tools needed. Borko (2004) and Borko et al. (2010) do not dismiss positivist methods that have been recognized as rigorous but rather allow these methods to be used more appropriately, such as when the research questions call for qualitative methodologies, quantitative methodologies, confirmation of causal relationships, etc. Additionally, many have cautioned about oversimplifying TPD and emphasized the importance of considering teacherswho they are, where they are in their learning, and what they need to move forward in their learning and growth. It is important to recognize that oTPD features may "collectively work together in different ways under different circumstances in different contexts" (Opfer and Pedder, 2011, p. 386).
These methodological suggestions have further implications for data collection and measurement. As TPD programs are enacted, they produce data that can be easily collected and that offer an opportunity to explore the nuances of teacher learning (Opfer and Pedder, 2011). Such data can focus on the structures of the TPD, including the role of learning analytics, in-depth text analysis, etc. Furthermore, online approaches enable the use of new technologies, which bring additional options for data collection and analyses that may have not been previously possible. Many of these methods, such as social network analysis, make use of digitized data, recognize the complexities of phenomena, and take context into account. Above all, it is important to carefully consider assumptions and implications associated with types of methodologies and methods used and utilize methodologies currently available that would recognize the complexity and contextual nature of human behavior, and specifically teaching and learning within a classroom.
Research recommendations calling for stronger research in oTPD have not consistently recognized the ways in which the data produced through engagement in online environments may allow researchers to develop a deeper understanding of the process of teacher learning or teacher development. The major critique of extant oTPD research is that there is a limited amount and what is available does not focus on the processes of teacher learning, the optimal conditions for oTPD, and whether what teachers learn impacts student achievement. Therefore, it is important that research in oTPD seek empirical evidence of best practices and explore what kind of measures most effectively provide evidence of the results of oTPD (Dede et al., 2009). Further, it is important to validate self-reported data on instructional practice against direct observation or some other independent measure of practice (Penuel et al., 2007).
Researchers also call for more rigorous research on TPD practices and learning, although with different areas of focus. Desimone (2009) highlighted the importance of understanding structures that lead to optimal learning within a TPD and Opfer and Pedder (2011) emphasize the importance of studying how teacher learning unfolds within TPD. This is a critical distinction, one which prompts us to be more systematic in understanding the whole process as we focus on smaller parts and individual elements. Studies need to consider issues of design and development, such as optimal features of oTPD and design guiding principles. Implementation studies would involve issues of enactment, context, facilitator support, the unfolding of learning within oTPD, and teacher change. Studies of scalability would examine issues related to pilots, implementation in different contexts and for different populations, and scaling up of courses and programs. Borko (2004) and Borko et al. (2010) challenge researchers to conduct research on both online and in-person TPD that moves beyond proof of concept at a single site to research that provides a more comprehensive view of quality TPD and promotes quality teacher learning across multiple sites. They suggest a three-phase framework for TPD/oTPD research that mirrors design-based research phases proposed by McKenney and Reeves (2012). Studies in Phase 1 provide single-site proofs of concept and focus on soundness and feasibility. Studies in Phase 2 research well-specified TPD programs across more than one site, determining local viability, institutionalization, and scalability and may extend to examining student achievement. Studies in Phase 3 involve a comparison of multiple, well-defined programs at multiple sites and typically study effectiveness and impact on teacher and student learning.

Need for Current Study
It has been about a decade since the calls were issued for a robust research agenda. During that decade, there has been continued exponential growth in the creation and provision of oTPD across the U.S. and the world. Based on the calls in the literature, highquality research into oTPD must attend to factors such as (1) in what contexts oTPD is usually conducted; (2) which teachers are most likely to participate in and complete oTPD courses; (3) which online tools are most helpful and supportive of delivery and teacher learning; and (4) what are the possible impacts of oTPD on student achievement through teacher change. The rapid pivot to online teaching and learning caused by the COVID-19 pandemic has increased the urgency to better understand how the existing categories and foci of research in oTPD map on to the current landscape of oTPD in light of these conceptual frameworks described a decade ago.

METHOD
This review began by looking for studies published between January 2009 and March 2019, the beginning year marking the voiced concern of several researchers in the field for more empirical research methods in oTPD (Dede et al., 2009;Borko et al., 2010). Next, a study screening was conducted, and criteria for selection were determined and are described in more detail below. Articles were filtered to glean information generated from individual studies, and relevant studies were then summarized. In total, 73 article summaries formed the basis of analysis for this review. The procedures for selection criteria and search strategies are explained next, followed by description of the data analysis.

Parameters for the Review
In order to examine this research, we searched in the following four education databases: ERIC, Academic Search Ultimate, PsycInfo, Education Full Text. Searches were limited to peerreviewed articles in English but also included studies conducted in any international K-12 setting. Using titles and abstracts, we looked for particular terms. To target research pertaining to teachers, we looked at "online professional development" or "oTPD" in the titles and abstracts. To find research on oTPD for practicing teachers and to avoid research on instructors in higher education, we used terms such as "K-12, " "in-service, " "elementary, " or "secondary." To further focus on professional learning for practicing teachers, we excluded "pre-service" in order to remove studies focused on pre-service teacher research.
Reviewing the abstracts of the 158 articles identified using the search parameters described above established a pattern of general topics and conversations pertinent in oTPD. However, not all authors referred to their program as oTPD. Therefore, we took additional steps to ensure a thorough identification of articles within our search parameters. We widened the search by applying wider descriptors and breaking apart the phrase "online professional development for teachers" into discrete terms. This strategy, though taken to ensure thoroughness of our search, produced tens of thousands of results which clouded rather than clarified the purpose of this literature review. Therefore, we used our original list of search terms described above to After removing duplicate records, we screened the articles for eligibility. In this stage, we removed articles that were not published in peer-reviewed journals (e.g., dissertations), did not occur in a K-12 setting (e.g., home-based preschools or higher education), or were not empirical (e.g., reports). We scanned titles and abstracts to ensure the population, research methods, journal, and topics met these criteria. After screening, 85 more studies were excluded and 73 articles were determined to meet the criteria of this review.

Analysis
At this point, each one of the 73 selected articles was analyzed in detail. The first author created a spreadsheet in Excel to summarize the following information from each article: Research Questions, Context/Location, Content/Purpose, Participants, Operationalization of the oTPD, Data Collection, Data Analysis, Findings, Conclusions, and Notes. In this way we could identify and record highlights and trends in program design, scale, location, implementation, etc. This spreadsheet was also used to record all results of our various analytic steps under the Notes column.
The first author conceptualized the methodology and, in consultation with the fourth author, developed the design for collecting and reviewing the literature. The first and fourth authors continued to consult during construction of the literature database and during critical moments in the decision-making process. The first and second author independently examined the findings and conclusions of each article to address our research questions regarding themes across recent research on oTPD, characteristics of successful oTPD, and gaps in the research. The first and second author first compared the foci independently identified across the 73 articles. They then worked collaboratively to combine foci identified into salient themes across the 73 articles. Consensus was reached regarding three themes. The three themes were examined independently by both authors to determine key lessons about oTPD. After these initial analytic steps, the third and fourth authors joined the first and second authors in reviewing the three identified themes across the 73 research articles. All authors participated in final revisions. This review of the three themes consisted of comparing and contrasting them to the current discussions on guidelines for research of oTPD from Dede et al. (2009), Desimone (2009), Borko et al. (2010, and Opfer and Pedder (2011). Analysis notes from all authors were recorded on the spreadsheet. Then, the notes were analyzed by all authors to further clarify the three themes in the 73 articles. We synthesized the results recorded on the spreadsheet into a summary form which appears in Supplementary Appendix B.

FINDINGS
Analysis of the 73 articles yielded insight into how oTPD research has been conceived, operationalized, and implemented in the past decade (Supplementary Appendices A,B). (Please see the two appendices for a more complete description of the methodological approaches, context of studies, operationalization of the oTPD studies over the last decade). Here, we first present three general categories of research into oTPD identified in this literature review: (1) Teachers and Students and oTPD; (2) Teachers and oTPD; and (3) Incidental Teacher Learning During oTPD. Second, we identify three themes that emerged across the three categories.

Categories of Research on oTPD
Analysis of the 73 studies on oTPD included in this literature review indicated that there are three general categories of studies: (1) Teachers and Students and oTPD; (2) Teachers and oTPD; and (3) Incidental Teacher Learning During oTPD. (Please see Table 2 for the distribution of articles falling in each category). The studies in this review sometimes included several research strands, qualifying them to more than one theme. In these circumstances, we placed the study under the category based on its most prominent focus.

Teachers and Students and oTPD
The studies identified by this category investigated whether oTPD brought about changes in teacher knowledge and performance that then resulted in improved student outcomes. These studies typically employed advanced statistical methods to address nested data that would point to changes in student knowledge and practices. These studies were typically large, grant-funded, quasiexperimental designs. Findings commonly suggested that a single course can influence teacher knowledge but is unlikely to have a strong impact on student outcomes (e.g., Goldenberg et al., 2014). Some studies did report evidence of improved student outcomes, as measured by a test score, and some did not.

Teachers and oTPD
The majority of studies in this category were focused on understanding the influence of the oTPD on a population of teachers, specifically using teachers' orientations and perceptions of the oTPD as a measure of its efficacy (e.g., Renninger et al., 2011;Frazier and Boehm, 2012). By exploring teachers' responses to the characteristics and practices of the online program, researchers could potentially improve efficacy of the oTPD in meeting the outcomes set for the program. This category included a wide variety of methodological approaches and investigated a wide variety of questions related to design and implementation (e.g., Masters et al., 2010;Fiel et al., 2018). Often, survey questions asked teachers for feedback on whether best practices identified for in-person professional development were also best practices for online settings (e.g., Chen et al., 2009;Holmes et al., 2011;Collins and Liang, 2015). Since teacher engagement is often linked to oTPD dropout rates and learning outcomes, some of these studies operated on the assumption that the more time teachers spent with the program and the more they liked it, the more they would learn.

Incidental Learning of Teachers During oTPD
A persistent concern in teacher education research is understanding changes in teacher thinking. The studies in this literature review that exemplify this third category used data that was collected during oTPD to develop more fine-grained understanding of the incidental teacher learning and knowledge growth that emerged during the oTPD. The context of these studies was oTPD; however, the focus of these studies was on the use of the data gathered during oTPD as a site for developing deeper knowledge about how teachers learn and how teacher thinking shifts, as well as to answer other incidental questions afforded by the data.

Themes of Research on oTPD
Three themes emerged across articles reviewed for this literature review. These themes illustrate the progress of oTPD over the last decade. First, we noted advances in research design, data collection, and evaluation in oTPD. Second, we noted attention paid to the physical layer and the pedagogical layer of online teacher learning environments. Third, we recognized a strong trend emphasizing the importance of and innovation in teacher collaboration and communities of practice in online settings. We

Advances in oTPD Research
This review documented advances in the kinds of data gathered and questions posed in oTPD settings. While there were fewer studies attempting to document the effects of oTPD from teacher to student outcomes, these studies from the first category of Students and Teachers and oTPD tended to be in advanced stages of evaluation. Researchers identified valuable data other than student test scores as potential markers of success, such as teacher retention, student discipline data, and rates for dropout and college-bound seniors in high schools (Shaha and Ellsworth, 2013). Research in the second category of Teachers and oTPD also showed advances in data collected and questions posed. Previous research relied heavily on teachers' reported satisfaction with oTPD as participants, sometimes in relationship to those who were engaged in in-person versions of the same TPD programs. Not one study in this review asked this question, indicating that the research field has moved beyond this outdated basic comparison of in-person with online to more nuanced questions to capture teachers' experience with oTPD. For example, Masters et al. (2010) addressed the question of whether a learning community model of oTPD had demonstrable effects on teachers' content knowledge and instructional practices in the context of fourth grade English Language Arts. The study was a random controlled design involving participants from 8 states. In another example, Seraphin et al. (2013) studied the effects of an oTPD implemented state-wide, which was designed to support teaching energy science using inquiry pedagogy. Findings indicated that the oTPD encouraged participation by focusing on enhanced online opportunities and practice with the teaching science through inquiry (TSI) process. The results also revealed areas to be improved, such as how to engage teachers of all levels of experience with TSI in more in-depth use of the phases of the TSI learning cycle. In these two studies and others like them, the researchers asserted that effective oTPD can improve teachers' understanding and implementation of content knowledge and pedagogical knowledge.
Such innovations in oTPD research design were also evident in the third category of Incidental Learning of Teachers during oTPD, describing studies that were designed to explore learning within oTPD environments. In one study, Chapman et al. (2010) sought the answer to the question "Does the digital divide still exist?"-tapping the opportunity of using data obtained from a large-scale oTPD to understand changes in teacher thinking on a much larger scale. Teachers were evaluated about their perceived digital learning abilities while participating in an oTPD that involved thousands of participants across 8 states. Surveys from the 10,831 participants in e-Learning for Educators oTPD were gathered, and participants were asked to assess their technology and technical skills. Alongside other questions, the study explored several demographic factors, confirming that the context of the school where teachers practiced-high need versus non-high need-was strongly correlated with teachers' self-reported skill with digital literacy.
In contrast to such a large-scale question, some studies such as Cady and Rearden (2009) (n = 8), provide a fine-grained analysis of such responses. Other studies, like Chapman et al. (2010) and Gu et al. (2015), explore a particular aspect or question but in the context of a large-scale project. Although very different in focus and design, these types of studies both large and small provided beneficial findings about teacher thinking and teacher change, perhaps as a result of participation in oTPD.
The research designs in the third category of Incidental Learning of Teachers during oTP tend to be proof-of-concept and design-soundness studies based on deeper understanding of the process of teacher change. They provide valuable insights into teacher thinking and teacher change in online contexts. Exploring these questions in the oTPD context leads to deeper understanding of policy issues such as the continued existence of a digital divide (Chapman et al., 2010) or the enduring effects of an oTPD when participants link up in communities of practice (Cady and Rearden, 2009). Indeed, evidence from all three categories indicates that researchers have developed more sophisticated survey, evaluation, and research approaches to look at issues such as teachers' perceptions of their learning and engagement with the format, content, and tools used in the oTPD.

Attention to the Physical and Pedagogical Layer of oTPD
The study of design processes provides insight into what makes oTPD distinct from TPD. Using a simplified theory of instructional design layers, Graham et al. (2014) propose two critical layers that need to be considered and aligned when designing technology-mediated instruction: the physical layer and the pedagogical layer. The physical layer represents the technology, allowing delivery of the instruction. This layer is often the priority when developing online and blended instruction, being driven by the cost, design specifications, and accessibility requirements. On the other hand, the pedagogical layer is represented by the underlying strategies and pedagogy that lead to achievement of learning outcomes and is often overlooked and underrepresented. In this review, we noticed a focus in the research that attended to both the physical and pedagogical layers in oTPD in design and evaluation. Many studies intentionally addressed how their oTPD gave attention not just to the physical layer but especially the pedagogical layer of underlying strategies (Graham et al., 2014).
An analysis of the research from the first category of Students and Teachers and oTPD, studying the impact of oTPD from development to evaluation of student outcomes, revealed various aspects of high quality oTPD design. In one such example, Frumin et al. (2018) studied the impact of teacher engagement in an online forum for teachers of Advanced Placement (AP) science on student outcomes. Teacher use of the online forum was measured through easily obtained frequency and duration data and analysis of lurking and posting behavior. This study provided a unique context to examine teacher learning and student outcomes since the AP exams present a common measure taught in thousands of US secondary schools nationwide. AP scores tend to matter to students, and AP-trained teachers are offered a broad range of TPD. Many aspects of this study were therefore generalizable to similar situations in countries where teachers prepare pre-college students with knowledge to bridge to higher education (Frumin et al., 2018).
In the second category of Teachers and oTPD, researchers were interested in which features of high-quality oTPD were noted and valued by the participants. The oTPD had specifically attended to physical aspects of design, including improvements in the technology such as navigability, availability of technical support, and other helps (Pape et al., 2015;Prestridge and Tondeur, 2015). However, much of the research in this category also targeted pedagogical aspects of design, as revealed in their surveys (Helenius et al., 2017;Sato and Haegele, 2017). For example, Collins and Liang (2015) delivered a program designed to reach a high number of teachers and to encompass specific valued components of high quality oTPD, including content relevancy, online features and delivery quality, online participation and duration, transformational learning for instructional practices, and adult learning theory. Even when responses from participants indicated that they did not experience these aspects as planned, it was significant that the oTPD was developed with this pedagogical layer, intending to engage teachers using these best pedagogical practices for adult learners in online settings.
In the third category of Incidental Learning of Teachers in oTPD, researchers also gathered evidence of participants' engagement with pedagogical tools, practices, and knowledge within a variety of content areas. For example, Albers et al. (2015) were interested in critical literacy being enacted in an online space. Others explored meaning-making activities such as teacher discourses, communities of practice, and a sense of belonging (Erixon, 2016). Others explored the use of discussion board prompts and facilitator interactions (Jarosewich et al., 2010;Park et al., 2013). Kibler and Roman (2013) were interested in teachers' thinking about native language use in the classroom. Researchers were also interested in how both pedagogical tools (e.g., test structure, Gu et al., 2015) and pedagogical practices (e.g., adopting problem-based learning, An, 2013) could be successfully managed and implemented in online formats. Repeated findings in these studies include the tensions experienced by teachers shifting from in-person to online formats, the role of facilitators, and questions about the complexity of tasks required online, teacher connection, and reflection Dana et al., 2017).
Indeed, it was evident in all three categories that the field of oTPD has taken what is known from the research into best practices in TPD and oTPD and intentionally addressed improvements in both the physical and pedagogical layers. In order to take full advantage of technology while designing technology-mediated instruction, these two layers must be aligned. The studies in this review indicate a clear trend in oTPD design toward both improved technology delivery and improved pedagogical practices in digital spaces. This multilayer view of technology-mediated instruction postulates further complexity for oTPD research.

Innovation in Communities of Practice and Collaboration
A recurring theme in all three categories of oTPD research was the importance of collaboration and development of communities of practice leading to improved learning and participation by teachers in oTPD. In the first category of Students and Teachers and oTPD, Frumin et al. (2018) studied the participation of AP science teachers in an online forum designed and provided by the College Board. The forum offered many ways to access information but also encouraged collaboration by designing digital spaces that made it possible for teachers to mentor others, share materials, or participate in discussions related to content or pedagogy. While the online forum was not the only form of TPD provided for AP science teachers, findings concluded that for teachers who spent time in the online forum, even as passive participants, their students subsequently performed higher on the AP exams. This finding points to the effectiveness of such identified high-quality practices of TPD as having rich and varied resources available, peers and facilitators available for support, and supports the use of co-constructed oTPD that is more peer-driven than top-down models.
In the second category of Teachers and oTPD, researchers showed a strong interest in collaboration and building communities of learning for teachers (e.g., So et al., 2009;Masters et al., 2010;Rodesiler, 2017). For example, a study based on teachers' perceptions of their learning studied implementation of an existing wide-reaching oTPD in Australia where the participating teachers were enrolled in the oTPD as part of their leadership accreditation process (Fasso, 2010). The oTPD sought to implement communities of practice, an identified component of high-quality TPD, as part of the program. In response to previous program feedback, the researchers examined whether linking participants as partners early on would lead to an enhanced perception of 'belonging, ' perhaps encouraging teachers to have greater interaction, to develop networks, and to build a stronger online community of practice.
One assumption of the researchers was that when participants had dispositions toward creating and participating in collaboration and community, communities of practice were more likely to develop. In other words, they posited that the development of networking relationships could be predispositional. Because they felt there was an important professional and learning benefit when such networks developed, the designers/researchers of the oTPD engaged participants in cycles of networking and community activity in order to support the development of collaborative relationships, communities of practice, and strong networks. Finally, because of their assumptions about predispositions for engagement being vital for completion of the program, they also sought to identify the characteristics of individuals most at risk of early cancelation of enrollment due to early isolation. The researchers employed a widely used temperament questionnaire first to determine participants' dispositions toward collaboration and learning and then to pair them as learning buddies within the oTPD. They then observed what impact intentionally pairing learners had on participants' completion of the course and their perception of their experiences within it. Teachers reported positively that they liked being "matched" to someone like them to collaborate with during the oTPD and also liked participating in the oTPD. An increased number of participants also completed the course. This study demonstrated how problem-solving about responses to an oTPD can lead designers and researchers to adjust a program and then study the influence of making such adjustments on measurable completion rates as well as teacher perceptions of the program.
In the third category of Incidental Learning of Teachers during oTPD researchers also showed a strong interest in studying and developing online communities of teacher engagement (Fasso, 2010;Francis and Jacobsen, 2013;Dana et al., 2017). A study by Cady and Rearden (2009), conducted in a rural district, explored the impact of an oTPD focused on changing teachers' content and pedagogical knowledge of mathematics teaching. This mixed methods study proposed to evaluate teacher learning of just 8 participants but the focus of the study actually explored in deep detail teacher discourse and the formation of communities of practice. They found that teachers used more careful and concise language in discussing mathematics and mathematics instruction as the oTPD progressed. Teachers also used communities of practice to clarify concepts and deepen their understanding. Analysis of their talk revealed that they shifted in the way they talked about teaching mathematics. Since the oTPD was made of clusters of participants where two or three teachers came from the schools located close to each other, at the end of the study, one group convinced their principal to fund a science and math lab, and they independently continued to develop their STEM knowledge beyond the oTPD. While similar to another study based on developing communities of practice (Fasso, 2010), Cady and Rearden (2009) went beyond program effectiveness to deeply inquire what happens when people are linked as collaborators in learning content.
Across all three categories, researchers have built evidence that high-quality practices in TPD, especially by intentionally designing space for collaborative discourse, formation of communities of learning and practice, or meaning-making activities participated in jointly are relevant in oTPD and need further exploration in online contexts (Dede et al., 2009;Desimone, 2009;Borko et al., 2010;Opfer and Pedder, 2011;Darling-Hammond et al., 2017). Indeed, there is evidence of a shift in the research from a focus on simple teacher satisfaction with oTPD to more complex explorations of how teachers engaged in collaboration and online communities of practice in oTPD may provide designers with new understandings about meeting the needs of adult learners enrolled in these programs.

CONCLUSION
In conducting this review of pre-pandemic studies over the past decade, we examined both quantitative and qualitative studies in order to uncover what research revealed about quality oTPD and what questions remained unanswered. This review has uncovered evidence of features and practices that are important in designing, developing, implementing, evaluating, and researching teacher learning in online settings. The three themes that emerged in our analysis of the reviewed articles illustrate the progress in oTPD. We noted that research in oTPD has made advances in research design, data collection, and evaluation; attention to pedagogy of online teacher learning environments; and a trend toward innovation in teacher collaboration and communities of practice in online settings.
Despite such progress in the field, some concerns remain. The popularity and explosive growth of oTPD still outpaces rigorous empirical research. The current delivery demands have only further stretched thin resources to facilitate the cycle of implementation followed by systematic research. Moving forward, employing methodologies and methods that acknowledge the complexity and situatedness of oTPD will be particularly crucial as conditions of crisis, and even trauma, characterize the design and implementation of oTPD in the circumstances of the COVID-19 pandemic. By recognizing the changing needs of schools, designers and researchers of oTPD can better choose what skills, applications, or teacher engagement strategies to most effectively influence desired teacher changes and support teachers through these changes.

AUTHOR CONTRIBUTIONS
CL set up the search criteria, screening studies and designing the methodology, with the help of RK. CL and BA independently examined the 73 articles, abstracts, findings, and conclusions to address research questions. All the authors compared the emerging themes to reach consensus. CL and BA analyzed the themes, wrote the results, and summarized the Supplementary Appendices. RC and RK participated in the final revisions.