Training in eight low-and middle-income countries: lessons learned from a pilot study using the WHO-TDR dissemination and implementation massive open online course

Introduction

Non-communicable diseases (NCDs) are the leading cause of mortality worldwide that disproportionately impact low and middle-income countries (LMICs) (1). With 80% of deaths from NCDs occurring in LMICs, the role of local research capacity and relevant research informing policy and practice is crucial (2). Despite this, there has been a particular lack of funding, training and mentorship for NCD investigators in LMICs (3).

The Special Programme for Research and Training in Tropical Diseases (TDR) at the World Health Organization (WHO) developed the Massive Open Online Course (MOOC), which aims to disseminate implementation research concepts (4). The primary goal of the course is to strengthen capacity building and improve training opportunities, targeting local public health researchers, practitioners and policy-makers (4). The course delivers dissemination and implementation (D&I) research education in LMICs where access to formal learning pathways, such as university courses in implementation research, may be limited (5). Investing in research capacity and training in LMICs reduces disease burden by building local research capacity and ensuring that those who are being trained are best equipped to address the needs of their communities (6–9).

MOOCs have steadily gained popularity given the accessibility, affordability, and effectiveness of the courses (4, 10, 11). The TDR MOOC on Implementation Research (IR) has shown to improve participant knowledge and understanding of implementation research and increased participants' ability to apply the course concepts to professional practice (12). While this MOOC was developed with a focus on infectious diseases of poverty, the course concepts can be applied to strengthening implementation research capacity for NCDs and other disciplines (12, 13).

The goals of this paper are to describe the pilot evaluation outcomes of one of the 2020 MOOC-D&I trainings conducted by the Global Research on Implementation and Translation Science (GRIT) consortium as part of the GRIT's ongoing mentorship and capacity building programs, and share barriers, facilitators, and recommendations to enhance future D&I training opportunities in LMICs.

Methods

TDR MOOC on IR

The TDR MOOC on IR is a step-by-step online training for public health researchers and decision-makers that focuses on design and implementation of research projects (12). Core concepts of implementation research are addressed in five modules including: (1) identifying the challenges of various health settings; (2) assessing the appropriateness of existing disease control strategies; (3) developing new interventions and strategies by working with communities and stakeholders; (4) specifying implementation research questions; and (5) designing rigorous research projects, including how to identify implementation outcomes, evaluating effectiveness, and making plans to scale-up implementation in real life settings (12). The course includes homework assignments, the requirement of completing and passing at least four quizzes and a final assignment with a peer-review component. The five modules were open to participants from May 11, 2020, to Sept. 25, 2020. The final exam was available to take after the five modules were completed until October 23, 2020, and participants were required to complete the peer-review assessment by November 6, 2020. Participants completed an electronic survey at the beginning and conclusion of the MOOC to evaluate the change in knowledge and self-assessed competencies. Subsequently, participants were asked to share barriers and facilitators to completing the MOOC. Participants who completed all course components received a MOOC certificate of completion.

Participants

There were two sets of participants in this study. The first were participants from the Global Research on Implementation and Translation Science (GRIT) Consortium funded by the National Heart Lung and Blood Institute. The consortium consists of research teams from eight countries, five of which (Guatemala, Ghana, Kenya, India, and Vietnam) test implementation strategies to deliver evidence-based interventions within these countries for the prevention, treatment, and control of hypertension and three of which (Malawi, Nepal, and Rwanda) provide capacity building in NCD and D&I research needed to close the gap between research and practice (3, 6). Specifically, all countries have partnership between D&I mentors and hypertension physicians in the U.S. and in country. Members from all countries were invited to GRIT workshops about implementation science and hypertension care, and all countries have developed formal and informal infrastructures of mentoring in D&I and research in general (14). The MOOC was an added structure in which consortium members decided to engage to support enhancing D&I knowledge for GRIT members.

The invitation to participate in the MOOC was open to all consortium members. Additionally, GRIT participants were encouraged to share the announcement with their respective networks. The second set of participants were not part of the GRIT Consortium and were recruited through snowball sampling through the GRIT network. We did not have inclusion or exclusion criteria. Our recruitment email invited anyone interested in the MOOC with a brief description of the course, timeline and expectations. Enrollment was open from April 6 to May 5, 2020 and the course ran from May 11, 2020 to November 6, 2020.

Measures

This was a pilot study aimed to see if training D&I via an online platform was feasible across eight LMICs. The primary outcome of this study was competency in D&I research. Surveys were distributed via Qualtrics (15). We also examined four secondary outcomes including: (1) D&I mentor access and quality; (2) general research mentor access and quality; (3) general research skill competencies, in manuscript writing, proposal writing, making scholarly presentations, and grant writing; and (4) a qualitative assessment of barriers and facilitators to completing the MOOC. While the TDR MOOC does not have a formal mentorship as part of the course, GRIT members are connected formally or informally with their D&I members in either delivering interventions or enhancing capacity building in D&I and hypertension care. Additionally, unique to this training was the expectation that results from the MOOC training could be used as potential future research ideas as part of GRIT capacity building efforts.

The current study is a single-group, pre-post study design to assess changes in D&I research competencies, measure mentor access and quality, and describe general research skills among participants in the TDR MOOC. Additionally, barriers and facilitators to completing the course were examined. Researchers originally developed the competency measure for D&I trainings in the United States (5, 8, 16). Others have subsequently used this measure to assess D&I competencies for the WHO MOOC internationally (10, 17). The 43 item self-report measure is organized into four subscales: (1) definitions, background, and rationale, (2) theory and approaches, (3) design and analysis, and (4) practice-based considerations (5, 8) using a 5-point Likert scale (i.e., Not at all to Extremely).

A secondary outcome of this study was D&I mentor access and quality, measured through three questions added to the original survey. The first question asked trainees whether they had access to a D&I mentor (answer options: yes, no, not sure). If the trainee had a D&I mentor, they were asked two follow-up questions. The first follow-up question assessed the quality of the mentoring (“how would you rate the overall quality of the mentoring you received from your D&I mentor?”). Trainees answered using a 7-point Likert scale with anchoring verbiage at 3 points (1- very low, 7-very high). The second follow-up question assessed the degree to which the D&I mentoring met their expectations: “to what extent do you feel your D&I mentor is meeting your expectations?” Trainees answered using a 7-point Likert scale with anchoring verbiage at 3 points (1-Not at all, 7-Completely).

A third outcome examined the general research mentor access and quality, with three additional survey questions. The first research mentor question asked trainees whether they had access to a research mentor (answer options: yes, no, not sure). Trainees with a research mentor were asked a follow-up question about mentoring quality and the degree to which the mentoring met their expectations. The same questions, with answer options, that were asked to assess the quality of the mentorship and met expectations for their general research mentor were asked for those with a D&I specific mentor.

A fourth outcome measured general research skill competencies, in manuscript writing, proposal writing, making scholarly presentations, and grant writing. Trainees rated their level of competency for each of these items using a 5-point Likert scale (1 not at all to 5 extremely).

The final outcome examined was barriers and facilitators to completing the MOOC. Trainees who completed the MOOC were asked: “what enabled you to complete the MOOC?” Trainees who completed some but not all of the MOOC were asked: “what enabled you to complete some of but not all the MOOC components?” Trainees who did not complete all of the MOOC were asked: “what prevented you from being able to complete the MOOC?” All trainees were asked: “What changes/support would help future participants complete the MOOC?” The questions pertaining to barriers and facilitators were open-ended and included in the post-assessment survey.

MOOC participant demographic information was also collected. Specifically, participants provided their gender, age, education, country, work position, work location, and GRIT participation.

Analysis

Quantitative analyses were conducted in Stata 16.1. A paired t-test was used to determine if trainees' D&I competencies and general research competencies changed from pre- to post-test. The trainees' average total D&I competency score and their average scores for each sub-scale were calculated (8, 10). The analytic sample for our primary outcome only included trainees with complete pre- and post- D&I competency measures; those with missing data were excluded. Chi-square tests, Fisher exact tests, and independent two sample t-tests using demographic variables were obtained to determine if the trainees without complete D&I competency measures differed from those with complete pre- and post-measures. Some respondents had missing demographic variables and could not be included in the comparison assessment. Demographic variable tests were run separately; the lowest number of missing variables was 5 and the highest was 9.

The results from that analysis suggest that there were no meaningful differences between those with complete D&I competency scores and those without. As such, only the results for trainees with complete pre- and post-test D&I competency measures are reported. Similarly, the analytic sample for the fourth outcome measuring general research competency only included respondents with complete pre- and post-data for that measure. There were more respondents with complete pre- and post-data for general research competency compared to the D&I competency completers. The same diagnostic tests were run on the general research competency completers and non-completers. No meaningful difference was found between the two groups.

A univariate statistic was used to analyze the data for the second and third outcomes. Data for those outcomes came from the pre-test survey data. Observations with missing data were removed. Finally, the qualitative data for the final outcome, barriers and facilitators, were analyzed using thematic analysis to identify themes, patterns and areas of overlap in participant's open ended survey responses (18).

Results

MOOC participation

247 individuals from the GRIT Consortium and ancillary networks enrolled in the MOOC; 116 (47%) completed the pre-assessment survey, 101 (41%) attempted any quizzes, 59 (24%) completed all quizzes, 35 (14%) completed the final exam, and 32 participants (13%) completed all course requirements, but only 21 (8%) completed both pre and post-surveys and therefore these are the ones included in the analysis.

Table 1 outlines the demographic characteristics of the trainees who completed both the pre- and post-competency measures (n = 21) and the demographics for all participants that initially enrolled in the MOOC (n = 116). Most of the trainees with complete pre-and post-D&I competency measures were female (57%); had a master's degrees (76%); were from Rwanda (33%) and Malawi (24%); were not GRIT Consortium members (62%); did not have previous D&I training experience (67%); and the average participant age was 35 ± 4.5.

Table 1

Table 1. Demographic characteristics of trainees.

D&I competencies

The self-reported D&I competency scores improved for those with completed pre- and post-competency measures (see Table 2). Trainees' average score improved on the full scale by 1.45 points., the, The definitions, background, and rationale subscale improved by 1.36 points. The theory and approaches subscale improved by 1.63 points. The design and analysis subscale improved by 1.45 points. practice-based considerations subscale improved by 1.37 points.

Table 2

Table 2. D&I competencies pre- to post test change.

Participants reported that access to D&I mentoring was low (not reported in Table). Only 12% (n = 14) of the MOOC trainees who completed the pre-survey (n = 116) indicated that they had a D&I mentor. Of those that had a D&I mentor, 63% reported the quality of mentoring was above average, with 21% rating the quality as “very high.” 71% reported a 4 or 5 out of 7 regarding that their mentor met their expectations for mentorship, where 4 reflected moderately meeting expectations.

Access to general research mentoring was reported by the participants as being higher compared to D&I mentoring. Forty percent (n = 46) of the MOOC trainees that completed the pre-survey reported having a research mentor. Around 52% of those with a research mentor rated the mentoring quality as above average. Similar to the D&I mentoring, 46% reported a 4 or 5 out of 7 reflecting that the mentor met expectations for mentorship, where 4 represented moderately meeting expectations.

Research competency scores among those with complete pre- and post-survey responses (n = 33) varied. While the scores generally improved from pre- to post-survey, the differences were not statistically meaningful. Participant manuscript writing scores remained constant between pre- and post-test [3.48 ± 0.97 vs. 3.48 ± 1.03, t(32) = 0.00, p > 0.05]. Proposal writing improved slightly from [3.42 ± 0.90 vs. 3.45 ± 1.12, t(32) = 0.19, p > .05] Scholarly presentation scores improved 0.18 points (3.58 ± 1.17 vs. 3.76 ± 1.12, t(32) = 1.00, p > 0.05. Grant writing scores improved also increased 0.18 points [2.24 ± 1.03 vs. 2.42 ± 1.12, t(32) = 1.18, p > 0.05].

Barriers and facilitators

Participants reported major barriers preventing them from completing the course including lack of time, other work commitments or additional responsibilities placed on them due to the COVID-19 pandemic, and lack of stable and consistent internet connection. Participants identified time management skills, an interest in the topics addressed by the course, and recognizing the opportunity to learn as driving factors in completing the course. Additional facilitators included collaborating with other participants, supervisors and colleagues; the course flexibility (i.e., pre-recorded sessions to adapt to participant's schedule as opposed to live sessions); and increased time to work on the course due to personal or professional changes during the COVID-19 pandemic.

Course recommendations

Participant recommendations for future MOOC sessions included: (1) greater mentorship from the GRIT stakeholders throughout the course; (2) greater collaboration among participants across LMICs; (3) having the ability to retake modules or quizzes for greater understanding of a specific topic; (4) incorporating an NCD module or more NCD related examples; (5) minimizing website navigation challenges; (6) facilitating access to a reliable internet connection; and (7) more course flexibility. To enhance flexibility, participants suggested having a flexible deadline for the peer assessment, having all modules accessible at the beginning of the course with a final deadline, and having extra time for assignments.

Discussion

This study examined the experience of participants from eight LMICs in one of the 2020 TDR MOOCs on IR. Using the data from the pre-and post-assessment surveys, the self-reported D&I competencies were analyzed as well as barriers and facilitators to completing the course, which provide recommendations and implications for future MOOCs. Although there was a low retention rate in the MOOC, participants completing the post survey showed improvements in their D&I competencies. Participants reported low access to D&I mentors, limited access to general research mentors, and low self-reported competency for manuscript and scientific writing.

Existing literature on previous MOOCs have generally shown lower completion rates (10, 19–23), including a systematic review reporting the majority of MOOCs in the study reported completion rates of less than 10% (23). The course completion rate in this MOOC (13%) was likely impacted by a couple of variables. First, internet access was a major barrier for retention in this study, which has been shown in similar studies (10). MOOCs, by design, enroll large groups of students, including both active and passive participants. Reconceptualizing retention to only include participants who substantively engaged with the course might provide a more accurate picture of program metrics (19). Third, the timeline in which we started the MOOC was challenging. Enrollment took place in April 2020 with a course start date in May 2020, right before several of the participating countries started the lock down to prevent further spread of the SARS-CoV-2 virus. Fourth, as soon as the cohort started the training, the MOOC website moved to be hosted by another company and the transition posed some issues with access to the videos. With the larger movement towards online courses and trainings, future guardrails to develop and maintain websites for online learning will be important (20–22).

Due to the global uncertainty of the COVID-19 pandemic, coupled with TDR platform issues, the first module was extended three months until the end of July 2020. The remaining modules adhered to the original timelines, with a spacing of two weeks between each module. During this period, 53% of enrolled participants no longer engaged in course activities. When asked about the barriers for participation in the post-assessment survey, participants shared that the lack of stable internet, other work commitments and responsibilities, and needing more time to complete the course were key barriers that affected their participation in this course. These barriers have been reported by participants from previous MOOCs (10, 13, 17), but they were likely intensified by the COVID-19 pandemic and associated lockdowns.

This was a pilot study aiming to see if we could provide capacity building in D&I using a MOOC platform in LMICs. The results indicate strong evidence of improvement with self-reported D&I competencies similar to previous courses (10). The subscale that had the largest change was theory and approaches and the subscale with the least change was definitions, background and rationale and practice-based considerations. These results differ from other D&I trainings where a majority of participants reported the largest change in the definitions, background, and rationale subscale (8, 10). The difference in results may be related to the composition of participants, where 33% of participants in this study had previous D&I training before the course. The general research competency scores in manuscript writing, proposal writing, and giving scholarly presentations did not change in a meaningful way from pre-to-post test. These findings suggest that general research capacities, not specific to D&I, should be targeted by future capacity-building activities, particularly grant writing. Accordingly, the TDR WHO has developed a flexible and interactive D&I toolkit to support capacity building and proposal writing (24).

The need for increased mentoring and guidance was a prominent theme in the recommendations submitted by the participants, as only 14% reported having a mentor in D&I research. Even though every country has a D&I consultant, the limited access to D&I mentors may be a reflection of very few researchers being trained in the emerging field of implementation science in LMICs. Evaluation of other D&I trainings in high resource settings have shown the importance of networking and mentoring, as well as time, for the development of academic outcomes (8, 25), and previous MOOCs with added support beyond mentorship (i.e., meetings for participants to discuss modules, Q&A sessions, discussion forums) demonstrated an increase in participant engagement (10, 26). Future D&I capacity trainings in LMICs should include greater mentorship and support throughout the course as it could contribute to higher course completion and improved overall D&I competency reporting (7, 27). It is also worth noting that many participants were not a part of the GRIT Consortium, but rather recruited and enrolled through collaboration with GRIT Consortium members, demonstrating that MOOCs are an effective tool to extend trainings beyond existing consortiums and partnerships.

Limitations

The major limitation of this paper is the small number of participants that completed the course. Additionally, evaluation data was comprised of only self-report data and, therefore subjected to bias and social desirability. Lastly, we did not follow up with participants to ask whether they were able to apply what they learned. The lack of opportunity to practice what they learned has been a challenge described by participants in previous MOOCs (25, 28).

Future directions and implications

Despite the challenges and limitations, partnering with the Special Programme for Research and Training in TDR MOOC is a feasible and scalable strategy to increase D&I training in LMICs. The use of D&I competency metrics allows for further evaluation on how to design training in D&I. In the future, research partners may add specific modules, such as hypertension care and D&I grant writing. Some of this is already being done as part of capacity building initiatives (17, 29). In moving forward, setting up and strengthening a collaborative practice whereby mentoring and peer collaboration across countries could be beneficial to all in enhancing the capacity for D&I research training (29).

Conclusions

This was a pilot study, and as such, the main hypothesis was to see if we could foster D&I capacity building in LMICs using the TDR MOOC platform. Using pre-post surveys, augmented by the analysis of the open questions from the trainees, this study follows similar designs of other capacity building efforts and adds to the literature in capacity building in D&I in LMIC, showing that self-report D&I competencies improved after the training.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent from the participants was not required to participate in this study in accordance with the national legislation and the institutional requirements.

Author contributions

AB and PL conceived the study and oversaw implementation of the course. AR helped coordinate the course and drafted the manuscript. AR drafted the paper. CH analyzed all data, and helped to draft and edit the manuscript. MPF helped draft and edit the manuscript. AB, MPF, MW, LP, HN, MR, AM, KA, MH, VD-R, PL and JG all provided feedback on manuscript sections. All authors contributed to the article and approved the submitted version.

Funding

This study was supported by the National Heart, Lung, and Blood Institute (NHLBI) under award numbers HL136789, HL136790, HL136791, HL138631, HL138635, HL138636, HL138638, HL138647. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NHLBI, the National Institute of Allergy and Infectious Diseases, the National Institutes of Health, or the U.S. Department of Health and Human Services.

Acknowledgments

We would like to thank all MOOC participants for taking part in the course. We also like to thank the TDR Special Programme for Research and Training in Tropical Diseases co-sponsored by UNICEF, UNDP, the World Bank, and WHO for their financial support for the MOOC program. Finally, TDR also received support from national governments, international institutions, as well as designated funding for specific projects within current TDR priorities. A full list of TDR donors can be found at: https://www.who.int/tdr/about/funding/en/. Lastly, the authors would like to acknowledge the tremendous and impactful contributions that Dr. Jacob Plange-Rhule made to research and capacity building efforts within the GRIT Consortium.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Islam SM, Purnat TD, Phuong NT, Mwingira U, Schacht K, Fröschl G. Non-communicable diseases (NCDs) in developing countries: a symposium report. Global Health. (2014) 10:81. doi: 10.1186/s12992-014-0081-9

PubMed Abstract | Crossref Full Text | Google Scholar

2. Haregu TN, Byrnes A, Singh K, Sathish T, Pasricha N, Wickramasinghe K, et al. A scoping review of non-communicable disease research capacity strengthening initiatives in low and middle-income countries. Glob Health Res Policy. (2019) 4:31. doi: 10.1186/s41256-019-0123-1

PubMed Abstract | Crossref Full Text | Google Scholar

3. Peprah E, Iwelunmor J, Price L. Assessing stakeholder engagement for translation research and implementation science in low- and middle-income countries: lessons from Ghana, Guatemala, India, Kenya, Malawi, Nepal, Rwanda, and Vietnam. Glob Heart. (2019) 14(2):99–101. doi: 10.1016/j.gheart.2019.05.009

PubMed Abstract | Crossref Full Text | Google Scholar

4. Launois P, Maher D, Certain E, Ross B, Penkunas MJ. Implementation research training for learners in low- and middle-income countries: evaluating behaviour change after participating in a massive open online course. Health Res Policy Syst. (2021) 19(1):59. doi: 10.1186/s12961-021-00703-3

PubMed Abstract | Crossref Full Text | Google Scholar

5. Padek M, Colditz G, Dobbins M, Koscielniak N, Proctor EK, Sales AE, et al. Developing educational competencies for dissemination and implementation research training programs: an exploratory analysis using card sorts. Implement Sci. (2015) 10:114. doi: 10.1186/s13012-015-0304-3

PubMed Abstract | Crossref Full Text | Google Scholar

6. Aifah A, Iwelunmor J, Akwanalo C, Allison J, Amberbir A, Asante KP, et al. The kathmandu declaration on global CVD/hypertension research and implementation science: a framework to advance implementation research for cardiovascular and other noncommunicable diseases in low- and middle-income countries. Glob Heart. (2019) 14(2):103–7. doi: 10.1016/j.gheart.2019.05.006

PubMed Abstract | Crossref Full Text | Google Scholar

7. Airhihenbuwa CO, Ogedegbe G, Iwelunmor J, Jean-Louis G, Williams N, Zizi F, et al. Claim your space: leadership development as a research capacity building goal in global health. Health Educ Behav. (2016) 43(1 Suppl):17s–24s. doi: 10.1177/1090198116633455

PubMed Abstract | Crossref Full Text | Google Scholar

8. Padek M, Mir N, Jacob RR, Chambers DA, Dobbins M, Emmons KM, et al. Training scholars in dissemination and implementation research for cancer prevention and control: a mentored approach. Implement Sci. (2018) 13(1):18. doi: 10.1186/s13012-018-0711-3

PubMed Abstract | Crossref Full Text | Google Scholar

9. Tabak RG, Padek MM, Kerner JF, Stange KC, Proctor EK, Dobbins MJ, et al. Dissemination and implementation science training needs: insights from practitioners and researchers. Am J Prev Med. (2017) 52(3 Suppl 3):S322–9. doi: 10.1016/j.amepre.2016.10.005

PubMed Abstract | Crossref Full Text | Google Scholar

10. Hooley C, Baumann AA, Mutabazi V, Brown A, Reeds D, Cade WT, et al. The TDR MOOC training in implementation research: evaluation of feasibility and lessons learned in Rwanda. Pilot Feasibility Stud. (2020) 6:66. doi: 10.1186/s40814-020-00607-z

PubMed Abstract | Crossref Full Text | Google Scholar

11. Launois P, Allotey P, Reidpath D, Maher D, Certain E, Ross B. Lessons learnt from a professional development MOOC: engaging culturally and linguistically diverse learners from low- and middle-income countries. Eur J Open Distan E-Learn. (2019) 13:127–37. doi: 10.5944/openpraxis.13.1.1172

Crossref Full Text | Google Scholar

12. TDR. Massive open online course (MOOC) on implementation research: Infectious diseases of poverty. (2021). Available Online at: https://www.who.int/tdr/capacity/strengthening/mooc/en/ (Accessed October 12, 2021).

13. Barteit S, Sié A, Yé M, Depoux A, Louis VR, Sauerborn R. Lessons learned on teaching a global audience with massive open online courses (MOOCs) on health impacts of climate change: a commentary. Global Health. (2019) 15(1):52. doi: 10.1186/s12992-019-0494-6

PubMed Abstract | Crossref Full Text | Google Scholar

14. Baumann AAF, Fort A, Weber M, Akwanalo MB, Nguyen C, Ha D, et al. Global implementation research capacity building to address cardiovascular disease: an assessment of efforts in eight countries. Science of dissemination and implementation in health; Arlington, VA (2019).

15. Qualtrics. Experience Design + Experience Improvement. Provo: Qualtrics. (2021). Available at: https://www.qualtrics.com (Cited June 28, 2021).

16. Padek M, Brownson R, Proctor E, Colditz G, Kreuter M, Dobbins M, et al. Developing dissemination and implementation competencies for training programs. Implement Sci. (2015) 10:A39. doi: 10.1186/1748-5908-10-S1-A39

Crossref Full Text | Google Scholar

17. Baumann AA, Mutabazi V, Brown AL, Hooley C, Reeds D, Ingabire C, et al. Dissemination and implementation program in hypertension in Rwanda: report on initial training and evaluation. Glob Heart. (2019) 14(2):135–41. doi: 10.1016/j.gheart.2019.06.001

PubMed Abstract | Crossref Full Text | Google Scholar

18. Clarke V, Braun V. Thematic analysis. In: Teo T, editors. Encyclopedia of Critical Psychology. New York, NY: Springer New York (2014). p. 1947–52.

19. Greene JA, Oswald CA, Pomerantz J. Predictors of retention and achievement in a massive open online course. Am Educ Res J. (2015) 52(5):925–55. doi: 10.3102/0002831215584621

Crossref Full Text | Google Scholar

20. Santos J. E-service quality: a model of virtual service quality dimensions. Manag Serv Qual. (2003) 13:233–46. doi: 10.1108/09604520310476490

Crossref Full Text | Google Scholar

21. Rowley J. An analysis of the e-service literature: towards a research agenda. Internet Res. (2006) 16(3):339–59. doi: 10.1108/10662240610673736

Crossref Full Text | Google Scholar

22. Demir A, Maroof L, Sabbah Khan NU, Ali BJ. The role of E-service quality in shaping online meeting platforms: a case study from higher education sector. J Appl Res High Educ. (2020) 13:1436–63. doi: 10.1108/JARHE-08-2020-0253

Crossref Full Text | Google Scholar

23. Liyanagunawardena TR, Adams AA, Williams SA. MOOCs: a systematic study of the published literature 2008-2012. Int Rev Res Open Dis Learn. (2013) 14(3):202–27. doi: 10.19173/irrodl.v14i3.1455

Crossref Full Text | Google Scholar

24. TDR. TDR Implementation Research Toolkit. Available Online at: http://adphealth.org/irtoolkit/ (Accessed October 12, 2021).

25. Luke DA, Baumann AA, Carothers BJ, Landsverk J, Proctor EK. Forging a link between mentoring and collaboration: a new training model for implementation science. Implement Sci. (2016) 11(1):137. doi: 10.1186/s13012-016-0499-y

PubMed Abstract | Crossref Full Text | Google Scholar

26. Jung Y, Lee J. Learning engagement and persistence in massive open online courses (MOOCS). Comput Educ. (2018) 122:9–22. doi: 10.1016/j.compedu.2018.02.013

Crossref Full Text | Google Scholar

27. Airhihenbuwa CO, Shisana O, Zungu N, BeLue R, Makofani DM, Shefer T, et al. Research capacity building: a US-South African partnership. Glob Health Promot. (2011) 18(2):27–35. doi: 10.1177/1757975911404745

PubMed Abstract | Crossref Full Text | Google Scholar

28. Hales S, Lesher-Trevino A, Ford N, Maher D, Ramsay A, Tran N. Reporting guidelines for implementation and operational research. Bull World Health Organ. (2016) 94(1):58–64. doi: 10.2471/BLT.15.167585

PubMed Abstract | Crossref Full Text | Google Scholar

29. UNU-IIGH. Implementation Research Training Framework 2.0: Making the whole greater than the parts. Kuala Lumpur (2008) 2021-06-08.