Training in eight low-and middle-income countries: lessons learned from a pilot study using the WHO-TDR dissemination and implementation massive open online course

Introduction Non-communicable diseases (NCDs) are a leading cause of morbidity and mortality in low-and middle- income countries (LMICs). Despite this, a lack of funding, training and mentorship for NCD investigators in LMICs exists. In an effort to gain knowledge and skills to address these gaps, participants from the Global Research on Implementation and Translation Science (GRIT), a consortium of studies in eight LMICs and their networks, attended the dissemination and implementation (D&I) massive open online course (MOOC) developed by the Special Programme for Research and Training in Tropical Diseases at the World Health Organization to strengthen D&I capacity building. Here, we report on the pilot of this MOOC, which was implemented during the SARS COVID-19 pandemic from April- November 2020. Methods Participants completed pre-and post-training questionnaires to assess self-reported D&I competencies, general research skills, and research mentor access and quality. D&I competencies were measured by use of a scale developed for a US-based training program, with change in competency scores assessed by paired t test. We used univariate statistics to analyze the data for all other outcomes. Results Of the 247 participants enrolled, 32 (13%) completed all course requirements, 21 (9%) completed the pre-and post-surveys and are included in the analysis. D&I competency scores suggest improvement for those who had complete pre- and post-assessments. Trainee's average score on the full competency scale improved 1.45 points (0–5 scale) from pre- to post-test; all four subscales also showed evidence of improvements. There were small but not significant increases in competencies for grant writing, proposal/ manuscript writing and presentations from pre- to post-test assessment. 40% of trainees reported access to a research mentor and 12% reported access to a D&I specific mentor. Participants reported barriers (e.g., unstable internet access and challenges due to COVID-19) and facilitators (e.g., topical interests, collaboration with colleagues) to completing the MOOC. Conclusions Although COVID-19 affected program usage and completion, the MOOC was feasible. We also had signals of effectiveness, meaning among LMIC participants completing the course, there was improvement in self-report D&I competency scores. Recommendations for future D&I trainings in LMICs include (1) adding more topic specific modules (i.e., NCD research, general research skills) for scalability; (2) fostering more collaboration with participants across LMICs; and (3) establishing partnerships with D&I mentors for course participants.

Introduction: Non-communicable diseases (NCDs) are a leading cause of morbidity and mortality in low-and middle-income countries (LMICs).Despite this, a lack of funding, training and mentorship for NCD investigators in LMICs exists.In an effort to gain knowledge and skills to address these gaps, participants from the Global Research on Implementation and Translation Science (GRIT), a consortium of studies in eight LMICs and their networks,

Introduction
Non-communicable diseases (NCDs) are the leading cause of mortality worldwide that disproportionately impact low and middle-income countries (LMICs) (1).With 80% of deaths from NCDs occurring in LMICs, the role of local research capacity and relevant research informing policy and practice is crucial (2).Despite this, there has been a particular lack of funding, training and mentorship for NCD investigators in LMICs (3).
The Special Programme for Research and Training in Tropical Diseases (TDR) at the World Health Organization (WHO) developed the Massive Open Online Course (MOOC), which aims to disseminate implementation research concepts (4).The primary goal of the course is to strengthen capacity building and improve training opportunities, targeting local public health researchers, practitioners and policy-makers (4).The course delivers dissemination and implementation (D&I) research education in LMICs where access to formal learning pathways, such as university courses in implementation research, may be limited (5).Investing in research capacity and training in LMICs reduces disease burden by building local research capacity and ensuring that those who are being trained are best equipped to address the needs of their communities (6)(7)(8)(9).
MOOCs have steadily gained popularity given the accessibility, affordability, and effectiveness of the courses (4,10,11).The TDR MOOC on Implementation Research (IR) has shown to improve participant knowledge and understanding of implementation research and increased participants' ability to apply the course concepts to professional practice (12).While this MOOC was developed with a focus on infectious diseases of poverty, the course concepts can be applied to strengthening implementation research capacity for NCDs and other disciplines (12,13).
The goals of this paper are to describe the pilot evaluation outcomes of one of the 2020 MOOC-D&I trainings conducted by the Global Research on Implementation and Translation Science (GRIT) consortium as part of the GRIT's ongoing mentorship and capacity building programs, and share barriers, facilitators, and recommendations to enhance future D&I training opportunities in LMICs.

TDR MOOC on IR
The TDR MOOC on IR is a step-by-step online training for public health researchers and decision-makers that focuses on design and implementation of research projects (12).Core concepts of implementation research are addressed in five modules including: (1) identifying the challenges of various health settings; (2) assessing the appropriateness of existing disease control strategies; (3) developing new interventions and strategies by working with communities and stakeholders; (4) specifying implementation research questions; and (5) designing rigorous research projects, including how to identify implementation outcomes, evaluating effectiveness, and making plans to scale-up implementation in real life settings (12).The course includes homework assignments, the requirement of completing and passing at least four quizzes and a final assignment with a peerreview component.The five modules were open to participants from May 11, 2020, to Sept. 25, 2020.The final exam was available to take after the five modules were completed until October 23, 2020, and participants were required to complete the peer-review assessment by November 6, 2020.Participants completed an electronic survey at the beginning and conclusion of the MOOC to evaluate the change in knowledge and self-assessed competencies.Subsequently, participants were asked to share barriers and facilitators to completing the MOOC.Participants who completed all course components received a MOOC certificate of completion.

Participants
There were two sets of participants in this study.The first were participants from the Global Research on Implementation and Translation Science (GRIT) Consortium funded by the National Heart Lung and Blood Institute.The consortium consists of research teams from eight countries, five of which (Guatemala, Ghana, Kenya, India, and Vietnam) test implementation strategies to deliver evidence-based interventions within these countries for the prevention, treatment, and control of hypertension and three of which (Malawi, Nepal, and Rwanda) provide capacity building in NCD and D&I research needed to close the gap between research and practice (3,6).Specifically, all countries have partnership between D&I mentors and hypertension physicians in the U.S. and in country.Members from all countries were invited to GRIT workshops about implementation science and hypertension care, and all countries have developed formal and informal infrastructures of mentoring in D&I and research in general (14).The MOOC was an added structure in which consortium members decided to engage to support enhancing D&I knowledge for GRIT members.
The invitation to participate in the MOOC was open to all consortium members.Additionally, GRIT participants were encouraged to share the announcement with their respective networks.The second set of participants were not part of the GRIT Consortium and were recruited through snowball sampling through the GRIT network.We did not have inclusion or exclusion criteria.Our recruitment email invited anyone interested in the MOOC with a brief description of the course, timeline and expectations.Enrollment was open from April 6 to May 5, 2020 and the course ran from May 11, 2020 to November 6, 2020.

Measures
This was a pilot study aimed to see if training D&I via an online platform was feasible across eight LMICs.The primary outcome of this study was competency in D&I research.Surveys were distributed via Qualtrics (15).We also examined four secondary outcomes including: (1) D&I mentor access and quality; (2) general research mentor access and quality; (3) general research skill competencies, in manuscript writing, proposal writing, making scholarly presentations, and grant writing; and (4) a qualitative assessment of barriers and facilitators to completing the MOOC.While the TDR MOOC does not have a formal mentorship as part of the course, GRIT members are connected formally or informally with their D&I members in either delivering interventions or enhancing capacity building in D&I and hypertension care.Additionally, unique to this training was the expectation that results from the MOOC training could be used as potential future research ideas as part of GRIT capacity building efforts.
The current study is a single-group, pre-post study design to assess changes in D&I research competencies, measure mentor access and quality, and describe general research skills among participants in the TDR MOOC.Additionally, barriers and facilitators to completing the course were examined.Researchers originally developed the competency measure for D&I trainings in the United States (5,8,16).Others have subsequently used this measure to assess D&I competencies for the WHO MOOC internationally (10,17).The 43 item self-report measure is organized into four subscales: (1) definitions, background, and rationale, (2) theory and approaches, (3) design and analysis, and (4) practice-based considerations (5, 8) using a 5-point Likert scale (i.e.,Not at all to Extremely).
A secondary outcome of this study was D&I mentor access and quality, measured through three questions added to the original survey.The first question asked trainees whether they had access to a D&I mentor (answer options: yes, no, not sure).If the trainee had a D&I mentor, they were asked two follow-up questions.The first follow-up question assessed the quality of the mentoring ("how would you rate the overall quality of the mentoring you received from your D&I mentor?").Trainees answered using a 7point Likert scale with anchoring verbiage at 3 points (1-very low, 7-very high).The second follow-up question assessed the degree to which the D&I mentoring met their expectations: "to what extent do you feel your D&I mentor is meeting your expectations?"Trainees answered using a 7-point Likert scale with anchoring verbiage at 3 points (1-Not at all, 7-Completely).
A third outcome examined the general research mentor access and quality, with three additional survey questions.The first research mentor question asked trainees whether they had access to a research mentor (answer options: yes, no, not sure).Trainees with a research mentor were asked a follow-up question about mentoring quality and the degree to which the mentoring met their expectations.The same questions, with answer options, that were asked to assess the quality of the mentorship and met expectations for their general research mentor were asked for those with a D&I specific mentor.
A fourth outcome measured general research skill competencies, in manuscript writing, proposal writing, making scholarly presentations, and grant writing.Trainees rated their level of competency for each of these items using a 5-point Likert scale (1 not at all to 5 extremely).
The final outcome examined was barriers and facilitators to completing the MOOC.Trainees who completed the MOOC were asked: "what enabled you to complete the MOOC?" Trainees who completed some but not all of the MOOC were asked: "what enabled you to complete some of but not all the MOOC components?"Trainees who did not complete all of the MOOC were asked: "what prevented you from being able to complete the MOOC?"All trainees were asked: "What changes/ support would help future participants complete the MOOC?"The questions pertaining to barriers and facilitators were openended and included in the post-assessment survey.
MOOC participant demographic information was also collected.Specifically, participants provided their gender, age, education, country, work position, work location, and GRIT participation.

Analysis
Quantitative analyses were conducted in Stata 16.1.A paired ttest was used to determine if trainees' D&I competencies and general research competencies changed from pre-to post-test.The trainees' average total D&I competency score and their average scores for each sub-scale were calculated (8,10).The analytic sample for our primary outcome only included trainees with complete pre-and post-D&I competency measures; those with missing data were excluded.Chi-square tests, Fisher exact tests, and independent two sample t-tests using demographic variables were obtained to determine if the trainees without complete D&I competency measures differed from those with complete pre-and post-measures.Some respondents had missing demographic variables and could not be included in the comparison assessment.Demographic variable tests were run separately; the lowest number of missing variables was 5 and the highest was 9.
The results from that analysis suggest that there were no meaningful differences between those with complete D&I competency scores and those without.As such, only the results for trainees with complete pre-and post-test D&I competency measures are reported.Similarly, the analytic sample for the fourth outcome measuring general research competency only included respondents with complete pre-and post-data for that measure.There were more respondents with complete pre-and post-data for general research competency compared to the D&I competency completers.The same diagnostic tests were run on the general research competency completers and non-completers.No meaningful difference was found between the two groups.
A univariate statistic was used to analyze the data for the second and third outcomes.Data for those outcomes came from the pre-test survey data.Observations with missing data were removed.Finally, the qualitative data for the final outcome, barriers and facilitators, were analyzed using thematic analysis to identify themes, patterns and areas of overlap in participant's open ended survey responses (18).

MOOC participation
247 individuals from the GRIT Consortium and ancillary networks enrolled in the MOOC; 116 (47%) completed the preassessment survey, 101 (41%) attempted any quizzes, 59 (24%) completed all quizzes, 35 (14%) completed the final exam, and 32 participants (13%) completed all course requirements, but only 21 (8%) completed both pre and post-surveys and therefore these are the ones included in the analysis.
Table 1 outlines the demographic characteristics of the trainees who completed both the pre-and post-competency measures (n = 21) and the demographics for all participants that initially enrolled in the MOOC (n = 116).Most of the trainees with complete pre-and post-D&I competency measures were female (57%); had a master's degrees (76%); were from Rwanda (33%) and Malawi (24%); were not GRIT Consortium members (62%); did not have previous D&I training experience (67%); and the average participant age was 35 ± 4.5.

D&I competencies
The self-reported D&I competency scores improved for those with completed pre-and post-competency measures (see Table 2).Trainees' average score improved on the full scale by 1.45 points., the, The definitions, background, and rationale subscale improved by 1.36 points.The theory and approaches subscale improved by 1.63 points.The design and analysis subscale improved by 1.45 points.practice-based considerations subscale improved by 1.37 points.
Participants reported that access to D&I mentoring was low (not reported in Table ).Only 12% (n = 14) of the MOOC trainees who completed the pre-survey (n = 116) indicated that they had a D&I mentor.Of those that had a D&I mentor, 63% reported the quality of mentoring was above average, with 21% rating the quality as "very high."71% reported a 4 or 5 out of 7 regarding that their mentor met their expectations for mentorship, where 4 reflected moderately meeting expectations.
Access to general research mentoring was reported by the participants as being higher compared to D&I mentoring.Forty percent (n = 46) of the MOOC trainees that completed the presurvey reported having a research mentor.Around 52% of those with a research mentor rated the mentoring quality as above average.Similar to the D&I mentoring, 46% reported a 4 or 5 out of 7 reflecting that the mentor met expectations for mentorship, where 4 represented moderately meeting expectations.

Barriers and facilitators
Participants reported major barriers preventing them from completing the course including lack of time, other work commitments or additional responsibilities placed on them due to the COVID-19 pandemic, and lack of stable and consistent internet connection.Participants identified time management skills, an interest in the topics addressed by the course, and recognizing the opportunity to learn as driving factors in completing the course.Additional facilitators included collaborating with other participants, supervisors and colleagues; the course flexibility (i.e., pre-recorded sessions to adapt to participant's schedule as opposed to live sessions); and increased time to work on the course due to personal or professional changes during the COVID-19 pandemic.

Course recommendations
Participant recommendations for future MOOC sessions included: (1) greater mentorship from the GRIT stakeholders throughout the course; (2) greater collaboration among participants across LMICs; (3) having the ability to retake modules or quizzes for greater understanding of a specific topic; (4) incorporating an NCD module or more NCD related examples; (5) minimizing website navigation challenges; (6) facilitating access to a reliable internet connection; and (7) more course flexibility.To enhance flexibility, participants suggested having a flexible deadline for the peer assessment, having all modules accessible at the beginning of the course with a final deadline, and having extra time for assignments.

Discussion
This study examined the experience of participants from eight LMICs in one of the 2020 TDR MOOCs on IR.Using the data from the pre-and post-assessment surveys, the self-reported D&I competencies were analyzed as well as barriers and facilitators to completing the course, which provide recommendations and implications for future MOOCs.Although there was a low retention rate in the MOOC, participants completing the post survey showed improvements in their D&I competencies.
Participants reported low access to D&I mentors, limited access to general research mentors, and low self-reported competency for manuscript and scientific writing.Existing literature on previous MOOCs have generally shown lower completion rates (10,(19)(20)(21)(22)(23), including a systematic review reporting the majority of MOOCs in the study reported completion rates of less than 10% (23).The course completion rate in this MOOC (13%) was likely impacted by a couple of variables.First, internet access was a major barrier for retention in this study, which has been shown in similar studies (10).MOOCs, by design, enroll large groups of students, including both active and passive participants.Reconceptualizing retention to only include participants who substantively engaged with the course might provide a more accurate picture of program metrics (19).Third, the timeline in which we started the MOOC was challenging.Enrollment took place in April 2020 with a course start date in May 2020, right before several of the participating countries started the lock down to prevent further spread of the SARS-CoV-2 virus.Fourth, as soon as the cohort started the training, the MOOC website moved to be hosted by another company and the transition posed some issues with access to the videos.With the larger movement towards online courses and trainings, future guardrails to develop and maintain websites for online learning will be important (20)(21)(22).
Due to the global uncertainty of the COVID-19 pandemic, coupled with TDR platform issues, the first module was extended three months until the end of July 2020.The remaining modules adhered to the original timelines, with a spacing of two weeks between each module.During this period, 53% of enrolled participants no longer engaged in course activities.When asked about the barriers for participation in the post-assessment survey, participants shared that the lack of stable internet, other work commitments and responsibilities, and needing more time to complete the course were key barriers that affected their participation in this course.These barriers have been reported by participants from previous MOOCs (10,13,17), but they were likely intensified by the COVID-19 pandemic and associated lockdowns.
This was a pilot study aiming to see if we could provide capacity building in D&I using a MOOC platform in LMICs.The results indicate strong evidence of improvement with selfreported D&I competencies similar to previous courses (10).The subscale that had the largest change was theory and approaches and the subscale with the least change was definitions, background and rationale and practice-based considerations.These results differ from other D&I trainings where a majority of participants reported the largest change in the definitions, background, and rationale subscale (8,10).The difference in results may be related to the composition of participants, where 33% of participants in this study had previous D&I training before the course.The general research competency scores in manuscript writing, proposal writing, and giving scholarly presentations did not change in a meaningful way from pre-topost test.These findings suggest that general research capacities, not specific to D&I, should be targeted by future capacitybuilding activities, particularly grant writing.Accordingly, the TDR WHO has developed a flexible and interactive D&I toolkit to support capacity building and proposal writing (24).
The need for increased mentoring and guidance was a prominent theme in the recommendations submitted by the participants, as only 14% reported having a mentor in D&I research.Even though every country has a D&I consultant, the limited access to D&I mentors may be a reflection of very few researchers being trained in the emerging field of implementation science in LMICs.Evaluation of other D&I trainings in high resource settings have shown the importance of networking and mentoring, as well as time, for the development of academic outcomes (8,25), and previous MOOCs with added support beyond mentorship (i.e., meetings for participants to discuss modules, Q&A sessions, discussion forums) demonstrated an increase in participant engagement (10,26).Future D&I capacity trainings in LMICs should include greater mentorship and support throughout the course as it could contribute to higher course completion and improved overall D&I competency reporting (7,27).It is also worth noting that many participants were not a part of the GRIT Consortium, but rather recruited and enrolled through collaboration with GRIT Consortium members, demonstrating that MOOCs are an effective tool to extend trainings beyond existing consortiums and partnerships.

Limitations
The major limitation of this paper is the small number of participants that completed the course.Additionally, evaluation data was comprised of only self-report data and, therefore subjected to bias and social desirability.Lastly, we did not follow up with participants to ask whether they were able to apply what they learned.The lack of opportunity to practice what they learned has been a challenge described by participants in previous MOOCs (25,28).

Future directions and implications
Despite the challenges and limitations, partnering with the Special Programme for Research and Training in TDR MOOC is a feasible and scalable strategy to increase D&I training in LMICs.The use of D&I competency metrics allows for further evaluation on how to design training in D&I.In the future, research partners may add specific modules, such as hypertension care and D&I grant writing.Some of this is already being done as part of capacity building initiatives (17,29).In moving forward, setting up and strengthening a collaborative practice whereby mentoring and peer collaboration across countries could be beneficial to all in enhancing the capacity for D&I research training (29).
Abbreviations D&I, Dissemination and Implementation; GRIT, Global Research on Implementation and Translation Science; LMICs, Low and middle-income countries; MOOC, Massive open online course; NCDs, Non communicable diseases; TDR, Training in Tropical Diseases; WHO, World Health Organization.

TABLE 1
Demographic characteristics of trainees.
a Values sum to more than 100% because respondents could select multiple work locations.b Sample size for the age was (n = 111).

TABLE 2
D&I competencies pre-to post test change.