Evaluation of Three Blended Learning Courses to Strengthen Health Professionals' Capacity in Primary Health Care, Management of Sexual and Reproductive Health Services and Research Methods in Guinea

Background Three blended courses on Primary Health Care (eSSP), Management of Sexual and Reproductive Health Services (eSSR), and Research Methods (eMR) were developed and implemented between 2017 and 2021 by the Maferinyah National Training and Research Center in Rural Health, a training and research institution of the Ministry of Health in Guinea. The study objectives were to evaluate the reasons for dropout and abstention, the learners' work behavior following the training, and the impact of the behavior change on the achievements of learners' organizations or services. Methods We evaluated the three implemented courses in 2021, focusing on levels 3 and 4 of the Kirkpatrick training evaluation model. Quantitative and qualitative data were collected through an open learning platform (Moodle), via an electronic questionnaire, during the face-to-face component of the courses (workshops), and at learners' workplaces. Descriptive statistics and thematic analysis were performed for quantitative and qualitative data, respectively. Results Out of 1,016 applicants, 543 including 137 (25%) women were enrolled in the three courses. Over the three courses, the completion rates were similar (67–69%) along with 20–29% dropout rates. Successful completion rates were 72% for eSSP, 83% for eMR and 85% for eSSR. Overall success rate (among all enrollees) ranged from 50% (eSSP) to 58% (eSSR). The majority (87%) of the learners reported applying the knowledge and skills they acquired during the courses through activities such as supervision (22%), service delivery (20%), and training workshops (14%). A positive impact of the training on utilization/coverage of services and increased revenues for their health facilities were also reported by some trainees. Conclusion These findings showed fair success rates and a positive impact of the training on learners' work behavior and the achievements of their organizations.

INTRODUCTION on hold several face-to-face training and led to develop a novel blended learning approach. Three blended learning courses were developed and implemented in Guinea. Evaluations of the first two courses (eSSP and eSSR) have already been reported (16,17). However, these reports primarily focused on completion and success rates, factors associated with success, and learners' perceptions (reactions) of the training (levels 1 and 2 of the Kirkpatrick model) (16,17). Moreover, reasons for dropout and abstention among training participants, the change in their work behavior (level 3), and the impact of this behavior change on the results of their organizations (level 4) were not assessed. Thus, the main focus of this study was to evaluate (1) reasons for dropout and abstention, (2) change(s) in work behavior reported by learners following the training, and (3) the impact of the work behavior change on the achievements of the organizations or services where the learners work.

Course Development and Implementation
The three courses were developed in French and implemented between 2017 and 2021 by the Maferinyah National Training and Research Center in Rural Health or Maferinyah Center, a training and research institution of the Ministry of Health in Guinea located in Forecariah district (50 km from Conakry). The development and implementation of the courses were funded by the Belgian Development Agency (Enabel), and the Directorate-General for Development Cooperation and Humanitarian Aid (DGD). More details on the Guinean context and the actual development of the courses have been provided earlier (16,17). The first two courses on Primary Health Care (eSSP) and Management of Sexual and Reproductive Health Services (eSSR) addressed the local health system context. They targeted health professionals working in health facilities and institutions, as well as medical students at the end of their medical school studies, before starting to work in the field ( Table 1). The third course on Research Methods (eMR) targeted medical students before completing their thesis research component at the end of the medical school and health professionals already or intending to be involved in public health and research ( Table 1). The "Analyze, Design, Develop, Implement and Evaluate (ADDIE)" instructional design model was used to develop the training modules (18). The capacity to develop and implement the course was built within the Maferinyah Center team through specific courses on e-learning and continuous coaching. The three courses were delivered through an online open Learning Management System (LMS), the Moodle platform. The eSSP, eSSR, and eMR respectively counted seven, nine and six twoweek online modules ( Table 1). The three courses consisted of a major online component or asynchronous learning with modules that could be downloaded in "Html" format. Synchronous learning included a continuous discussion forum, a Zoom call scheduled for each module, and was initiated during the COVID-19 epidemic (with increased use for the last training cohorts), and a five-day face-to-face capacity building workshop at the end of the training for a sample of trainees ( Table 1).

Study Design and Period
The three implemented blended courses (eSSP, eSSR and eMR) were evaluated from June to August 2021. A cross-sectional study using a mixed-methods approach was used. We collected quantitative and qualitative data in four stages: (i) through the learning platform (Moodle course statistics); (ii) via an electronic questionnaire, (iii) during the learners' capacity building workshops (opinion on short-term effect), and (iv) in the field to interview participants at their workplaces.

Theoretical Framework for Training Evaluation
We used Kirkpatrick's four-level model (Reaction, Learning, Behavior, and Results), which is one of the widely known evaluation models adapted to education (14, 15) (Supplementary Figure 1).

Operational Definitions
Completion rate was the number of learners who completed the course by performing all learning activities over the total number of enrollees. Dropout rate was the number of learners who dropped out from the course after completing some activities over the total number of enrollees. Abstention rate was the number of enrollees who ultimately did not log into the online learning platform although they had received all necessary information to access it, over the total number of enrollees. Successful completion rate or success rate was the number of learners who passed the course (with an overall mean of marks greater than or equal to five out of ten), over the number of learners who completed the course. Overall success rate was the number of learners who passed the course, over the total number of enrollees.

Quantitative Data Collection
We collected selected sociodemographic characteristics and results of courses (participation, completion, and success rates) for all participants in the three courses through the Moodle open platform statistics. This information aimed to provide some basic information about the course before addressing level 2 of the Kirkpatrick model. An additional electronic questionnaire (KoboToolbox) was sent to all course participants by email to better understand the behavioral changes at work following the training experience (level 3 of the Kirkpatrick training evaluation model).

Qualitative Data Collection During the Capacity Building Workshops
During three workshops, which took place between June and July 2021 for 5 days per workshop, time was set aside to collect data from learners who attended the different workshops. In-depth individual interviews (IDIs) and focus group discussions (FGDs) were carried out by the primary author (TMM) and JMK using an interview checklist. As participants were accommodated at the Maferinyah Center, the IDIs and FGDs were performed during the evenings and lasted around 50 and 70 min, respectively. This qualitative data collection during the workshops aimed at achieving objectives 3 and 4 of the evaluation. Five learners per cohort were purposively selected for the workshops according to their performance levels -the first two learners on the list, one from the middle, and the last two learners (with a low score and/or not having completed the course). The choice of interviewees was purposive and respected the principle of maximum variation and the data saturation model (19).

Additional Qualitative Field Data Collection
A team of three people collected additional qualitative data in the field for seven days to complement the already available data, more specifically to reach levels 3 and 4 of the evaluation model. We collected data from (1) learners who were not selected for the consolidation workshops (having validated or not the courses); (2) learners in their professional context and situation. We purposively selected interviewees along the Enabel intervention axis (regions of Conakry, Kindia, and Mamou), considering the principle of maximum variation and the data saturation model (19).

Data Analysis
For the quantitative analysis, the data was extracted from Moodle platform and Kobo Toolbox through an Excel spreadsheet and imported into SPSS version 21 for analysis. Descriptive statistics were performed as proportions for categorical variables and median with standard deviation for continuous variables.
Regarding qualitative data, the interviews recorded in French were transcribed in full. We applied thematic analysis (20,21) to analyze the responses to open-text questions. Each open-text response was coded in NVivo (TD). Inductive coding was used with codes discussed with the research team, grouped into themes, and based on the inductive thematic saturation model (19). Qualitative data were further analyzed to identify and document connections between the themes. Quotes from the opentext responses were used to illustrate findings from themes and quantitative results.

Ethical Considerations
The research protocol was approved by the National Ethics Committee for Health Research in Guinea (No: 022/CNERS/2020) and the ITM Institutional Review Board in Belgium (Reference Code: 1363/20). Regarding the qualitative component of the study, free, informed, and oral consent was obtained from each selected participant before carrying out the interviews. Both quantitative and qualitative data were only accessible to the research team. The database is stored on a computer protected by a password at the Maferinyah Center.

Characteristics of Learners of the Three Blended Courses
Overall, 1016 health professionals, including 235 (23.1%) women, applied for the courses (eSSP, eSSR and eMR) through calls for applications launched online between 2017 and 2020 using professional [District.Team Guinea (22)] and social (Facebook) networks.
A total of 543 learners, including 137 (25.2%) women, were selected out of the applicants ( Table 2). Overall, the majority of participants were between 30 and 40 years old (n = 379, 70%), male (n = 406, 75%), medical doctors (70%), Guinean nationals, or based in Guinea (87 and 88%, respectively). Women were more represented in the eSSR course (32%) than in the other two courses. Only 23 nurses were enrolled in the three courses, and 14 midwives participated exclusively in the eSSR course. Although the majority of participants resided in Guinea (93%), some learners resided outside Guinea (n = 71) ( Table 2).

Main Results for the Three Blended Courses
Results of the three courses presented in Table 3 show that a similar proportion of participants completed all course modules (ranging from 67 to 69%) with dropout rates varying from 20% (eSSP) to 29% (eMR). Success rate (among those who completed the courses) ranged from 72% (eSSP) to 83% (eMR) and 85% (eSSR). Overall success rate (among all enrollees) ranged from 50% (eSSP) to 58% (eSSR). More detailed results on factors associated with successful completion rates have been described for eSSP and eSSR courses and published elsewhere ( Table 3) (17).

Characteristics of Participants in the Evaluation of the Courses
A total of 233/543 (43%) participants completed the online questionnaire, and 53 participated in individual in-depth interviews (IDIs) and five focus group discussions (FGDs). Sociodemographic characteristics of the participants in the evaluation or respondents are in line with those of course participants (Supplementary Table 1  Reasons for Dropout During the Course or Not Attending After Enrolment (Abstention) The main reasons reported for dropping out during the course (after starting at least one module) were interference with other courses or work overload (62.5%), lack of technological skills (10.4%), travel or displacement (8.3%), computer breakdown or loss (6.3%), internet connection problem (4.2%), and illness (4.2%) ( Table 4). On the other hand, participants reported that the main reasons for not attending the courses after being enrolled (without even starting one module) were moving to or changing city where the internet network is not stable (31.8%), work overload (18.2%), interference with another ongoing course (18.2%), difficulty in accessing the learning platform (13.6%) and late receipt of information (13.6%) ( Table 4).

Application of Acquired (New) Knowledge and Skills
The majority (86.5%) of the participants reported that they put into practice the knowledge they had acquired during the course Frontiers in Digital Health | www.frontiersin.org  Table 5). An eSSR learner stated that: "During supervisions, especially integrated supervisions, I am often given the reproductive health part, as they know that I took the eSSR course. So, I calculate and interpret indicators and develop improvement plans for bottlenecks" (EIA, eSSR, female). An eMR learner asserted: "The studies we carry out in the cardiology department, of course, require the drafting of a research protocol first of all. So, I put into practice the knowledge I acquired in the eMR course by writing research protocols and correcting students' theses, but also by writing scientific articles for publication. These activities really allow me to apply what I learned in the eMR course" (EIA, eMR, male).
Some participants (13.5%) did not apply what they had learned because the course was not applicable to their daily work or current job.
"I have not yet applied the knowledge and skills I have acquired, but I would like to do so in the future because currently my job is not related to public health; I am in research" (EIA, eSSP, male).
Although they were not yet applying the acquired knowledge, other participants stated that they would continue to read the courses to keep up to date, hoping to practice what they had learned in the future.

Change in Behavior at Work or in Professional Practice
Participants who applied the knowledge and skills they acquired in the course reported that they had changed their work behavior, which is directly related to the course. The main findings are presented in Box 1. Many respondents reported feeling more confident and more comfortable at work following the training. The training also increased the visibility of some participants in their workplaces and was a source of inspiration for new activities in their work. For some respondents, the training allowed them to improve their professional practice and that of their colleagues in the same department. The training also enabled some participants to change their behavior in the workplace and provide support to other health services when called upon.

Impact of the Training on the Achievements of Learners' Organizations or Services
A positive impact of the training on utilization/ coverage of services and increased revenues for the health facility was also reported by trainees (Box 1). An impact of the training on public health outcomes of the organizations or services where the participants' work was reported by many participants, for instance, through improved supervision (Box 1).

DISCUSSION
This study documented a blended learning experience consisting of a major online component recently implemented in Guinea. Results showed that fair success rates could be achieved despite challenges learners face in a low-resource setting such as Guinea. Applied knowledge and an impact on behavior and performance in trainees' workplaces were also reported by participants following the training.
E-learning and blended learning are not yet commonly used in Africa. A recent systematic review of global health capacity building and evaluations in low-and middle-income countries (2009-2019) showed that despite a sharp recent increase in elearning or blended courses during the last 4 years, most studies conducted in Africa documented face-to-face teaching modalities (75%) (23). The same review reported that evaluations, in general lacked standardization, especially regarding the tools, and only face-to-face initiatives were evaluated in the long term beyond the individual level (23). Many evaluations are restricted to knowledge assessment through pre-and post-tests, which are indicative but limited (24, 25).
As a result, the findings of this study will further enrich the existing literature on e-learning and blended learning, shedding light on the design, implementation, and evaluation of e-learning and blended learning programs.
We found a similar course completion rate (70%) to that of a blended learning course for building workforce capacity for effective use of health information systems in Namibia and Tanzania (73%) (26), but less than a tuberculosis course completion rate (90%) in Ethiopia (27). Nonetheless, our findings align with those of studies that highlighted avenues to enhance the completion rate on online courses, though many other studies reported lower completion and success results (28)(29)(30)(31). The success rates among enrollees reported in our study in the three courses varying from 50 to 58% are greater than those reported by several universities specialized in e-learning in Thailand, India, the United Kingdom, and France (17-48%). As Karsenti et al. (32) reported, our results demonstrated that in the African setting where internet coverage and quality and ownership of using new technologies in education remain a challenge, fair to high success rates can be achieved in e-learning or blended learning. The requirement for obtaining good results is the deployment of elearning efficiency conditions such as a simple, attractive, and BOX 1 | Reported behavioral outcomes and impact on work organizations/services.

Theme Quotes
Behavioral outcomes More confidence and comfort in their work "Since I am in the research field, whenever there are issues to discuss, my interventions are not like before because now I reason with confidence and am at ease" (EIA, eMR, male). Increased visibility at work "Yes, of course, because before, few people entrusted me with their work (protocols and theses), but when I finished this training, we produced a protocol,l and since they saw it, many lecturers in the department entrust me with their theses to transform them into articles because they see that I am doing an excellent job, and all this thanks to this training that I followed" (EIA, eMR, male) Training as a source of inspiration at work Well, my usual professional practice is general medicine; consulting patients is everything; maybe what I can say is that for some time, I have been much more interested in reviewing the files of the consulted patients to make articles. It came to me through this course. It's something I didn't realize at first, but I'm talking to my boss about doing articles based on the patients we consult" (EIA, eMR, female). Training helped participants to improve not only their professional practice, but also that of their colleagues in the same department "Yes, there has been a net improvement not only in my professional practice but also in the professional practice of my health workers because today, although our activities are done routinely, there is a certain organization that is done at the level of the service regarding data management and even professional behavior. Many things have changed" (EIA, eSSR, male). "I admit that the course has influenced my way of doing things; firstly, I am the medical coordinator of an epidemic treatment center, so this course has enabled me to put forward arguments during the daily staff meetings by explaining to my medical staff the value of patient-centered care, and it is really from the module on quality of care that I got these tools; Secondly, in parallel with the coordination of the epidemic treatment center, the strategy office of the Ministry of Health often calls me for supervision missions (…) during which I analyzed the services at the deconcentrated level by collecting data and calculating indicators. Then, I identified the weaknesses that I analyzed and coached these health structures to develop improvement plans" (FGD, eSSP, participant N • 7). "So, as I said earlier, I took advantage of the eMR course to strengthen my ability to accompany students. Since I took the eMR course, I have become much more effective in accompanying students who are developing their thesis in medicine. Also, I have started to be a research assistant for a PhD student in epidemiology and statistics, and I feel comfortable working with her because I have been really well equipped" (EIA, eMR, female). Impact on institutions Impact of the training on utilization/ coverage of services, health outcomes "We have a health center here where I often went to help the new head of the center who came because he had difficulties with the family planning service concerning the calculation of indicators, which enabled him to present a good FP coverage. Also, during the monitoring that we are doing today, we made mistakes. It was at the prefectural technical health committee (CTPS) that these mistakes were corrected. However, the course we followed has now enabled us to go and support the other health centers, to help them with the monitoring so that when the CTPS is presented, we don't find enough mistakes" (EIA, eSSP, male). "Yes, that's what I was telling you; this training has enabled us to really improve the quality of the data; in any case, the monitoring indicators have improved. When we came here to the CTPS, people said, "Wait a minute, what did you do? Can you explain a bit? So I would tell them about the course I had taken" (EIA, eSSP, male). "Of course, in the health centers I was supervising, there were fewer deaths because the system of timely referral was well established and respected through my supervision" (EIA, eSSP, female). Impact on increased revenues for the health facility was also reported by trainees.
"The revenue generated in the six months following my training has increased compared to the revenue of the previous six months, which was 26 million Guinean francs" (EIA, eSSR, male). "Yes, of course, of course! It has helped to improve everything that is an indicator. For example, family planning coverage has improved because we have changed the strategies with users" (EIA, eSSP, male). easily accessible delivery method with user-friendly navigation; but also a variety of communication tools for synchronous and asynchronous interaction between instructors and learners, and between learners themselves. Other prerequisites or enablers are quality content, a motivating pedagogical approach with clear objectives, varied learning resources, a continuous technical and pedagogical support to learners and instructors, ethical aspects, and continuous evaluation system incorporated (32).
Completing (and dropping out from) online courses remains a significant concern (31). After starting at least one module, reasons for dropping out from e-learning or blended learning were mostly related to workload, other work issues, or lack of technical skills. Very few respondents who dropped out mentioned internet connection as a reason. However, among the respondents who were enrolled but did not attend (even one-course module), internet connection or technical issues represented a major problem, and this corroborates the studies reporting that technological limitations can act as a barrier to e-learning within a faculty and geographical context (33)(34)(35). A recent Cochrane review identified factors that impact e-learning, such as interaction and collaboration between learners and facilitators, considering learners' motivation and expectations, utilizing user-friendly technology, and putting learners at the center of pedagogy. The same review called for a better understanding of the issues related to enablers and barriers associated with e-learning, and a broader framework for making e-learning effective (36).
Capacity-building programs aim to provide the human resources of organizations with the proper knowledge that will enable them to perform their tasks and improve their skills and capabilities to innovate in their activities. Bouzguenda argues that applying acquired knowledge and skills deals with action and the validity of learning (37). About nine out of ten respondents apply the knowledge gained during the training, reflecting the relevance of course content to local capacity-building needs or, in short, the validity of the training. However, some learners (one over ten) suffer from the lack of applying the gained knowledge due to the lack of link between what they had learned and their current job or tasks; while expressing the willingness to apply the acquired knowledge in the future. This is in line with one theory of factors determining learning transfer, which advocates for a favorable work environment for employees to innovate (37).
Several systematic reviews have assessed e-learning training outcomes among health professionals compared to traditional learning and have shown similar results. Two reviews relating to nursing education reported high level of satisfaction among nurses or students (38,39). A recent Cochrane review addressing the effectiveness of e-learning among health professionals concluded that e-learning could make little, or no difference in patient outcomes or health professionals' behaviors, skills or knowledge (40). The impact on public health results such as health service coverage and health outcomes in settings such as Guinea has been less documented. Nevertheless, elearning offers an alternative method of education. In the Guinean context, it had the added value of training health professionals in their working sites throughout the country, avoiding useless travel costs and leaving health facilities without key human resources.
We used the Kirkpatrick model as a framework to help us with a more standardized evaluation. The widely used Kirkpatrick model represents a simple tool and language in dealing with the different outcomes of training and how information about these outcomes can be obtained as well as a practical approach for the typically complex evaluation process. Similarly, it leads to some limitations -it fails to grasp complexity of the context and related factors -it shows a certain rigidity -and the hierarchical order and importance of the different levels can be questioned (41,42). According to Bove and Little, "many training evaluation methods, including Kirkpatrick, are insufficient because they were not created considering the types of data now available. Further, many misconceptions about the relationship between training and workplace performance (e.g., the commonly held myth that completing a task in learning automatically translates to successful performance on the job) have led to poor evaluation practices" (43). Therefore, the same authors recommend the use of evaluation and data collection methods that are appropriate for the given training context (43).
Among study strengths, we can point out that it assessed higher levels of the Kirkpatrick evaluation model that are not often addressed in training evaluation. Reasons behind abstention and dropout were explored, and the qualitative component included participants who both completed, succeeded, or did not complete the courses. As limitations, first, the relationship between outcomes and sociodemographic characteristics included in an earlier paper (17) was out of the scope of this mixed-methods paper. Second, although this study did assess the impact of the training on learners' daily professional activities, qualitative interviews were conducted among a limited number of learners, and it relied on self-report that is subject to bias (socially acceptable response); we could not interview the learners' hierarchy or colleagues to crosscheck the provided information.
CONCLUSION This evaluation showed fair success rates and a positive impact of the training on learners' work behavior and the achievements of their organizations.

DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

ETHICS STATEMENT
The studies involving human participants were reviewed and the research protocol was approved by the National Ethics Committee for Health Research in Guinea (No: 022/CNERS/2020) and the ITM Institutional Review Board in Belgium (IRB Reference Code: 1363/20). The patients/participants provided their written informed consent to participate in this study.

AUTHOR CONTRIBUTIONS
TM, KK, TD, and AD conceived and designed the research protocol. TM, JK, TD, and AD elaborated the evaluation tools. TM and JK collected the data (online survey, interviews and focus group discussions) and performed transcriptions. TM and TD analyzed and interpreted data and drafted the manuscript. All authors critically revised the manuscript and approved the final version before submission, and they are accountable for all aspects of the work.

FUNDING
This evaluation was funded by the Belgian Development Agency (Enabel).