How Management and Leadership Training Can Impact a Health System: Evaluation Findings From a Public Health Management Training Program in Cambodia

In 2017, the National Institute of Public Health in Cambodia collaborated with the U.S. Centers for Disease Control and Prevention to provide management and leadership training for 20 managers and senior staff from 10 health centers. We conducted a mixed methods evaluation of the program's outcomes and impact on the graduates and health centers. From June 2018 (baseline) to January 2019 (endpoint), we collected data from a competency assessment, observational visits, and interviews. From baseline to endpoint, all 20 participants reported increased competence in seven management areas. Comparing baseline and endpoint observational visits, we found improvements in leadership and governance, health workforce, water, sanitation, and hygiene, and health centers' use of medical products and technologies. When evaluating the improvements made by participants against the World Health Organization's key components of a well-functioning health system, the program positively contributed toward building four of the six components—leadership and governance, health information systems, human resources for health, and service delivery. While these findings are specific to the context of Cambodian health centers, we hope this evaluation adds to the growing body of research around the impact of skilled public health management on health systems.

Introduction

As a complex adaptive system, health care and health systems demand competent managers and transformational leaders throughout its organizations to optimize health outcomes. Frontline health managers are often overlooked in health systems strengthening research and practice, although they directly influence the health care workers and the operational levels of the health care system responsible for delivering products and services to the population (1).

In an effort to improve coverage and the quality of health care services, the Government of Cambodia introduced health care reform in 1995, classifying the public health system into national and provincial levels. In each province, operational districts were formed based on the size of the population. Referral hospitals provide level one to level three specialist services depending upon the area. Health centers play an integral role in the Cambodian health system, providing frontline health services to the public; each health center provides basic primary care services to populations of 8,000 to 10,000 persons. Health centers receive feedback and overall guidance from the community through a health center management committee (HCMC) and are under the management and supervision of the operational district (2). A health center chief leads the daily activities of the health center, generally supervising a deputy chief and at least seven additional staff members. The health center chief also operationalizes health policy from national and sub-national levels to support health center staff and the communities they serve.

Despite their integral role, health center chiefs do not receive management and leadership training prior to assuming their positions. Thus, in 2017 the National Institute of Public Health (NIPH) in Cambodia collaborated with the U.S. Centers for Disease Control and Prevention (CDC) to address this need and provide management and leadership training for its health center managers and senior staff. With the approval of the Cambodia Ministry of Health, NIPH formed a technical working group to discuss the necessary knowledge and skills a health center manager needs. The technical working group included representatives from the national and sub-national offices of the Cambodia Ministry of Health, NIPH, CDC, and the Korea International Cooperation Agency (KOICA). After a series of workshops, the technical working group developed a 6-month management and leadership capacity building program, utilizing technical support from CDC and financial support from KOICA. The program was adapted from the Improving Public Health Management for Action (IMPACT) program, originally launched by the CDC in 2016 (3).

Previous thought scholarship in global health workforce development has acknowledged that better management is a necessary component to improving global health outcomes. Bradley et al. argued that a three-part strategy is needed to build the field of management as a key pillar of global health: training and education, practice, and research (4). Additionally, Yeager and Bertrand note that little evidence exists to inform public health management needs and best practices in global health settings (5–7). Thus, this paper will assess evaluation findings from the IMPACT Cambodia program and evaluate these findings against the World Health Organization's (WHO) key components of a well-functioning health system (Table 1). While these findings are specific to the context of Cambodian health centers, we hope this evaluation adds to the growing body of research around the impact of skilled public health management on health systems. In sharing the findings of our evaluation, we aim to contribute to the discussion around the importance of public health management in health systems strengthening.

TABLE 1

Table 1. IMPACT Cambodia classroom modules and the related World Health Organization's key components of a well-functioning health system (8).

Materials and Methods

The NIPH Technical Working Group selected Battambang and Preah Vihear provinces in northern Cambodia to pilot the IMPACT Cambodia program. Participants were nominated by their Provincial Health Directors and selected by NIPH. IMPACT Cambodia aimed to build capacity through interactive classroom training and regular mentorship at health center sites. Classwork was comprised of 6, 1-week modules covering topics listed in Table 1. The last module of the program was a symposium, providing participants with the opportunity to share how they have implemented lessons learned. After the first five training modules, a team of mentors visited each health center to provide technical guidance on fellows' activities, answer lingering questions from the modules, and provide direct coaching to the fellows. NIPH recruited seven individuals from its staff to serve on the mentorship team; mentors were not involved in the technical working group or other aspects of program implementation. However, mentors were provided with mentorship training by CDC prior to program initiation in August 2018.

The evaluation of the IMPACT Cambodia program utilized a mixed methods approach, employing a baseline and endpoint competency assessment, observational visits, and interviews. The technical working group provided direction and technical support for program implementation and evaluation design. The evaluation's goal was to determine the extent to which the program reached its intended outcome (participants' increased management competence) and what impact the program had on graduates and health centers.

The evaluation received a non-research determination by the CDC Center for Global Health's Institutional Review Board (Tracking number: 2018-169) and approved by the Cambodia National Ethics Committee for Health Research (#134NECHR; 21 June 2018). The key ethical principles of voluntary and informed participation, confidentiality and safety of participants was used in evaluator and participant interactions. Participants were provided with written information about the evaluation, informed that their participation was voluntary, and that they may withdraw from participation at any time.

From June 2018 to January 2019, data were collected from the 20 participants (nine health center chiefs and 11 health center staff members) from 10 health centers in Battambang and Preah Vihear provinces in Cambodia. Quantitative methods were used to measure the change in participant competence and at the health centers; qualitative methods were deployed to provide context to those changes. Analytic approaches varied by data source and type. Data sources and respective analytic approaches included the following:

Competency Assessment

To measure the competency change for participants, we utilized baseline (August 2018) and endpoint (January 2019) competency assessments. We asked participants to rank their knowledge and skills in performing selected tasks in seven management areas: analytics and assessment; communications; program planning; community engagement; financial planning and management; leadership and systems thinking; and emergency preparedness, planning and response. Participants ranked their knowledge and skills in each area on a Likert scale of one to four as described in Table 2. The assessment was created by the Council of Linkages (9) which was validated by Edgar et al. in their study of the construct validity and reliability of the Council of Linkages' core competencies for public health professionals (10). The assessment and was translated into Khmer, Cambodia's official language. To ensure accurate translation, we used human translation with an additional review by our research assistant, a native Khmer speaker. Using R statistical software, we calculated the mean and standard deviation for each management competency at the baseline and endpoint (11). We conducted paired t-tests to compare baseline and endpoint for each management competency and set statistical significance at p < 0.01.

TABLE 2

Table 2. Likert scale for baseline and endpoint competency assessment, created by the Council of Linkages (9).

Observational Visit Quantitative Checklist

At baseline (June/July 2018) and endpoint (January 2019), we visited all 10 health centers to conduct observational visits. To reduce observer bias, one to three members of the evaluation team visited each health center with the aim of yielding more comprehensive, accurate observational data (12). In health centers where more than one observer was present, this also helped to provide inter-observer reliability in completion of the checklist.

We completed a quantitative checklist of indicators related to the management of each health center. The checklist used closed-ended questions and unambiguous response categories to reduce observer bias. The checklist was unique to the context of health centers in Cambodia; the evaluation team and key technical working group members developed the checklist based on the curriculum, and roles and responsibilities of health center chiefs and staff. It covered seven categories (service delivery; health information systems; leadership and governance; health workforce; water, sanitation, and hygiene [WASH]; health financing; and medical products and technology), with a total of 41 indicators (Table 3). The evaluation team scored indicators through direct observation or document review. Observations were recorded promptly to reduce bias. Checklist results were compiled in Microsoft Excel. Nominal values were averaged, and frequencies calculated, where necessary.

TABLE 3

Table 3. Change in indicators, from baseline to endpoint, of IMPACT Cambodia participants' health centers, by health system category.

Interviews

In addition to the checklist, we conducted semi-structured interviews with all 20 participants from each health center at baseline and 19 participants at endpoint. We were unable to interview one participant at endpoint due to illness. Participants were asked about management processes and challenges at baseline and endpoint, as well as if and how they are applying skills learned from the training (Table 4). All interviews were conducted in Khmer by a trained evaluator. Interview data were translated by a native Khmer speaker to English for analysis.

TABLE 4

Table 4. Questions asked during semi-structured interviews with IMPACT Cambodia participants, at baseline and endpoint.

We conducted an analytic induction of the qualitative data using Microsoft Excel and ATLAS.ti. We started the analysis with deductive themes based on the evaluation goals and continued to look for undiscovered patterns and emergent themes throughout the analysis (13). We used this qualitative analysis method to ensure that we were meeting the evaluation goals, while also allowing for unexpected themes to emerge.

Findings

Baseline and Endpoint Competency Assessment

Participants reported increased competence in all management areas (Table 5). The response rates for both baseline and endpoint competency assessments were 100%. All t-values were negative and statistically significant (p < 0.01).

TABLE 5

Table 5. Change in mean competence, from baseline to endpoint, of IMPACT Cambodia participants, by management area.

Observational Visit Quantitative Checklist

Comparing baseline and endpoint observational visits, we found improvements in four of the seven categories: leadership and governance, health workforce, WASH, and medical products and technology (Table 3). For leadership and governance, we found increases in the use of monthly and quarterly plans; the use of data when developing the monthly and quarterly plans; the development of outbreak/disaster preparedness plans; and the creation of outbreak/disaster plans.

Findings indicated that the average number of days since the last health center committee (HCMC) meeting decreased from 96 to 67 days. However, the average number of days since the last all-staff meeting increased from 32 to 35 days.

For the health workforce category, we found increases in evidence of minimum required education for all staff roles; use of formal performance reviews; and use of staff duty roster. For the WASH category, we found increases in three out of the seven indicators—existence of a toilet to meet the needs of people with disabilities; separate toilets for patients and staff; and separate toilets for men and women. We found a decrease in the availability of hand hygiene materials both at toilets and points of care.

We found no changes in the indicators of service delivery, health information systems, and health financing.

Interviews

In conducting an analytic induction of the semi-structured interviews, we found seven recurring themes: the importance of mentorship; leadership and governance; WASH; service delivery; community engagement; health equity; and self-efficacy.

The Importance of Mentorship

In interviews, participants reported that the mentorship component played a key role in their success. According to participants, mentors reinforced what they learned in class, explained concepts they were still confused about, and reminded them to complete their homework.

Leadership and Governance

In the area of leadership and governance, all health center chiefs reported using a more collaborative leadership approach after participation in the training. They now encouraged staff to share ideas about decisions that affected the whole health center. Health center chiefs reported involving staff members in tasks like technical assistance, budget planning, and annual planning. Health center chiefs also reported learning the importance of recognizing good performance and encouraging and motivating staff. Further, participants reported taking the initiative to solve issues at the health centers, rather than waiting for direction from the Operational District, Provincial Health District, or NIPH. For example, to address ambulance shortages in their rural district, the Koh Ke Health Center Chief and a staff member set up four donation boxes in the community, with proceeds paying for transport to the referral hospital, which is one and a half hours away. In addition, participants highlighted learning the importance of developing a better understanding of community needs, experiences, and access. For example, six health centers used lessons on mapping to create community maps allowing them to visualize the households in their catchment area; health center staff planned to use these maps to reach out to villagers that did not seek service at the health center.

Participants also reported learning the value of planning, particularly as it relates to implementing activities. Additionally, each health center involved in the training created four to six new standard operating procedures (SOPs), including clinical management guidance documents, with the exact number depending upon the needs of the health center. SOP topics included management of finances, outbreaks of food-borne illnesses, medical and non-medical waste, first aid for emergencies, first aid for newborns, road traffic-related accidents, and complications after childbirth. SOPs were a new concept for the health centers but were reportedly well-received and operationalized. At the time of our endpoint visit (January 2019), subordinate health center staff who did not participate in the IMPACT program were already trained or in the midst of receiving training on the SOPs. Staff at some health centers were already reporting success in their use. One example was at the Kamrieng Health Center, where staff used the new SOP for complications after childbirth. Following the SOP, they efficiently referred a woman to the hospital who had significant blood loss after giving birth.

Water, Sanitation, and Hygiene

In the area of WASH, we found that participants improved waste management at their health centers, increasing the number of waste bins from three to four or five, and ensuring separate bins for different waste types (e.g., non-medical and medical including pathological, toxic, infectious, and needles). Some health centers also created structures to store waste in the rainy season and burn it in the dry season. During our observational visits, we found a 10% decrease in the availability of hand hygiene materials and toilets at points of care (Table 5), however, in our interviews, staff at four health centers reported an increase in staff usage of hand hygiene materials at points of care and toilets.

Service Delivery

Although the service delivery indicators in the quantitative checklist did not change from baseline to endpoint observational visits, participants used knowledge from the course to improve the patient experience at their health center. Nine out of 10 health centers posted a board outside the health center directing patients where to go based on their needs.

Some health centers reported that the greatest change due to the training was instituting a patient triage system. For example, the Chrey Health Center now gives patients color coded squares upon arrival to define the severity of their condition and ensure those that need care urgently are prioritized.

Community Engagement

Participants reported learning new ways to communicate with the community and how to use the community as a resource in their work at the health centers. As described above, participants either planned to visit the community and reach out to community leaders or had already done so by the time of our visits. For example, at the Boribo Health Center, the chief and staff member realized during the training that they can begin mobilizing needed resources for the health center from the community rather than waiting to receive them through provincial health department channels. The health center is now being more resourceful by gathering wood from villagers and local non-governmental organizations to build a pre-delivery room for pregnant women.

Health Equity

Participants also reported health equity was a new concept for them and that it had an impact on the way they viewed their work. This new perspective led to changes in patient flow and setting up triage systems. At one health center, a chief reported that she's delegating tasks and responsibilities to her staff more equitably without regard to factors such as age or socioeconomic status.

Self-Efficacy

In the area of individual self-efficacy, participants reported feeling more confident to help the chief, share their ideas at staff meetings, and provide feedback to other staff on issues like tardiness. Participants also reported sharing the knowledge they learned at the training with other staff. This knowledge-sharing, in turn, encouraged other staff to share their knowledge with the rest of the team.

Discussion

From these findings, we determined that the program did achieve its intended outcome—participants reported an increase in competence over seven key management areas. Through observational visits and interviews, we found that participants applied what they learned from the program. This finding is consistent with findings from the Philippines Field Management Training Program, a management capacity building program for local health workers similar to IMPACT. The Philippines Field Management Training Program found that “the results…provide compelling near-term evidence of how the skills from the training are being used by participants to identify, prioritize, and solve day-to-day managerial problems” (14).

These gains in management competence may not have been possible without mentorship support. Post-training support is a necessary aspect of effective training design (15). IMPACT mentors provided guidance and support while participants were at the health centers, bridging the gap between classroom training and experiential learning. One participant shared that it was easier to apply what they learned in the classroom with the mentors' support and guidance. For future health workforce capacity building programs, funders and program implementers should consider mentorship an essential aspect to ensure participant understanding and skill application.

During baseline observational visits, we saw that health facilities largely did well in the areas of service delivery, health information system, and health financing. However, endpoint interviews captured how the participants used the training to improve factors outside of the indicators in the quantitative checklist, particularly in the areas of service delivery, community engagement, and self-efficacy. These factors are demonstrative of more comprehensive management and leadership capability.

The WHO's key components of a well-functioning health system can provide a useful guide for funders and program implementers in how they build and tailor their health workforce capacity building programs, as well as how they measure the impact of these programs on the health system. We found the training positively contributed toward building four out of six of the WHO's building blocks for a well-functioning health system—leadership and governance, health information systems, human resources for health, and service delivery (Table 6) (8). We found no evidence that capacity improved around health financing and essential medical products and technologies.

TABLE 6

Table 6. IMPACT Cambodia participants' contributions to related WHO key components of a well-functioning health system.

For leadership and governance, participants' creation of plans, and more importantly, staff involvement in creating those plans, can help improve health outcomes in the complex, ever-changing environment of the health system (16). Yet, effective leadership and governance are only possible with accurate information on the health challenges and context in which the health system operates (8). To that end, by identifying and analyzing the geographical locations within their catchment areas where residents who utilized the least amount of health services lived, then developing targeted plans for community outreach aimed at encouraging those community members to visit the health center, participants will be able to improve access to care and the quality of services provided.

Leadership, governance and health information systems are cross-cutting components that provide the foundation for all the other building blocks, yet human resources for health is a key input in a well-functioning health system (17). A strong health workforce responds to the needs and expectations of the community, and is fair and efficient in achieving health outcomes with available resources (8). Through conducting formal performance reviews with staff, using a more collaborative leadership approach, encouraging staff to share ideas about decisions that affected the health center, recognizing positive performance and providing encouragement and motivation to staff, participants are creating enabling work environments for the health workforce, which encourage them to focus on the needs and expectations of the community.

Health systems can only be successful in producing health outcomes if they deliver the necessary services. Service delivery crucially depends on the development of standards and guidance to ensure access and quality of care (8). Through developing and utilizing SOPs, instituting patient triage systems, improving WASH practices at the health centers, and creating and displaying a map directing patients where to access health services, participants are ensuring access, greater satisfaction, and higher quality of care for their patients and the community.

While the training did not appear to have an impact on the two remaining building blocks of a health system (health financing and essential medical products and technologies), this could be due to decentralization in the Cambodian health system. Implementation of health programs is the responsibility of provincial health departments rather than the Ministry of Health. Due to decentralization, provincial health departments are the focal point for allocation of resources to operational health districts and health centers (18). Therefore, we believe expanding public health management training to health center staff nationwide, particularly to provincial health department managers, would be beneficial for Cambodia's health system.

One strength of this evaluation is the utilization of a mixed methods approach, which permitted a triangulation of results that discretely quantitative or qualitative research approaches alone would not have achieved. A single methodology would have been inadequate to examine the impact of the training program in a complex environment such as a health system (19). While evaluation findings are not generalizable (20, 21), the mixed method approach is replicable and may prove beneficial for other countries or funders to evaluate their health workforce capacity building programs. Yet we agree with Yeager and Bertrand that evaluations of health system strengthening programs, particularly ones focused on management and leadership, require a significant investment of resources, which may be a deterrent for some funders or program implementers (5).

This evaluation has several limitations. First, self-report data may lead to a misestimation of abilities when compared with objective assessments. This may be compounded by issues of comprehension and memory recall. To address this response bias, we used the observational visit checklist as an objective assessment at baseline and endpoint to compare with the interview data. Second, these evaluation results are not generalizable outside of this program in Cambodia, but do demonstrate the short-term results of the implementation of such a program (20, 21). Third, since we used human translation to translate collected data from Khmer to English for analysis, it is possible that context was lost in translation. Fourth, we did not validate participants' interview responses with independent sources, so it's possible our own personal experiences may have resulted in methodological bias. To address this bias, we've attempted to clearly and accurately present participants' experiences in all cases (22). Lastly, the evaluation was constrained by time and our inability to assess the long-term impact of the training on graduates and the health outcomes in their communities. A long-term study that compares health centers with IMPACT graduates to those without IMPACT graduates and utilizes data from Cambodia's Health Equity and Quality Improvement Program could potentially measure the cascading effect of training participation on health outcomes.

The evaluation findings may have several practical implications for funders and program implementers, particularly those engaged in building and tailoring health workforce capacity building programs. In terms of program design, purposeful mentorship may be instrumental to ensuring participant understanding and skill application. In addition, trainings focused on competency development for frontline health managers and providers can be highly effective in skill-building and behavior change in health facility operations. Further, focused training of this kind may lead to demonstrations of enhanced leadership, self-efficacy, cooperation amongst staff, as well as improved service delivery and community engagement.

The evaluation approach may also provide lessons for global health capacity-building programs. Rigorous evaluations of such programs do require a significant investment of resources. However, this investment is critically important to understand the possible effects of these programs. Moreover, they are important to building the evidence-base around workforce capacity development efforts, particularly in the areas of health system management and leadership where there has been limited study.

We hope this evaluation encourages further research into the impact of skilled public health management on health systems, not only in low- and middle-income countries, but also in well-resourced nations. Although the Global Health Security Agenda identifies improved public health management capacity as essential for achieving its goals (23), many health systems still do not invest in or prioritize developing public health management capacity (24). The COVID-19 pandemic has elevated the urgency of this work, particularly in the managerial areas of coordination, community engagement and trust, planning, and adaptive leadership (25). Thus, continued investment and evaluation of workforce development efforts of this kind may be critical to addressing underlying health system issues and improving public health outcomes.

Data Availability Statement

The de-identified data that supports the findings of the study are available on request from the corresponding author.

Ethics Statement

The studies involving human participants were reviewed and approved by U.S. Centers for Disease Control and Prevention Center for Global Health's Institutional Review Board and the Cambodia National Ethics Committee for Health Research. The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author Contributions

KH, PW, PI, CC, MK, VL, and EW equally involved in the conceptualization and design of the program. CH led the data collection, analysis, and interpretation. CH, KH, PW, and CC drafted the manuscript. CH, PI, MK, VL, and EW provided critical revisions to the manuscript. All authors were involved in designing the evaluation of the program and approved the final version of the manuscript.

Funding

This publication was supported by Grant or Cooperative Agreement numbes 5UGH001873 and 1NU2HGH000044-01-00, funded by the U.S. Centers for Disease Control and Prevention. This project was supported by grant numbers 5NU2GGH000990-05-00 and 6NU2GGH002124-01-01, funded by the Korea International Cooperation Agency. This project was supported in part by an appointment to the Research Participation Program at the U.S. Centers for Disease Control and Prevention administered by the Oak Ridge Institute for Science and Education through an interagency agreement between the U.S. Department of Energy and the U.S. Centers for Disease Control and Prevention.

Author Disclaimer

The views expressed in the submitted article are solely the responsibility of the authors and do not necessarily represent the official views of the U.S. Centers for Disease Control and Prevention, the U.S. Department of Health and Human Services, the Korea International Cooperation Agency, the Task Force for Global Health, Inc., or Training Programs in Epidemiology and Public Health Interventions Network.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We thank the Cambodian National Institute of Public Health, the Cambodian Ministry of Health, and the Korea International Cooperation Agency for their expertise and support throughout all aspects of the IMPACT program and evaluation.

References

1. Kumah E, Ankomah S, Antwi F. The role of first-line managers in healthcare change management: a Ghanaian context. Int J Biosci Healthc Technol Manag. (2016) 6:20–33.

Google Scholar

2. Cambodia Ministry of Health. Operational Guidelines on Minimum Package of Activities, Version 10 (2017).

3. Willacy E, Bratton S. On management matters: why we must improve public health management through action comment on “management matters: a leverage point for health systems strengthening in global health.” Int J Health Policy Manag. (2015) 5:63–5. doi: 10.15171/ijhpm.2015.174

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Bradley EH, Taylor LA, Cuellar CJ. Management matters: a leverage point for health systems strengthening in global health. Int J Health Policy Manag. (2015) 4:411–5. doi: 10.15171/ijhpm.2015.101

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Yeager VA, Bertrand J. Putting Management Capacity Building at the Forefront of Health Systems Strengthening; Comment on “Management Matters: A Leverage Point for Health Systems Strengthening in Global Health”. Rochester, NY: Social Science Research Network (2016). Report No.: ID 2711067. Available online at: https://papers.ssrn.com/abstract=2711067 (accessed November 29, 2021).

PubMed Abstract | Google Scholar

6. Hatt L, Johns B, Connor C, Meline M, Kukla M, Moat K. Impact of Health Systems Strengthinging on Health. Bethesda, MD: Health Finance & Governance Project, Abt Associates (2015). Available online at: https://www.hfgproject.org/wp-content/uploads/2016/03/Impact-of-Health-Systems-Strengthening-on-Health-7-24-1.pdf

7. Adam T, Hsu J, de Savigny D, Lavis JN, Røttingen J-A, Bennett S. Evaluating health systems strengthening interventions in low-income and middle-income countries: are we asking the right questions? Health Policy Plan. (2012) 27(Suppl. 4):iv9–19. doi: 10.1093/heapol/czs086

PubMed Abstract | CrossRef Full Text | Google Scholar

8. World Health Organization. Key Components of a Well Functioning Health System. (2010). Available online at: https://www.who.int/healthsystems/EN_HSSkeycomponents.pdf?ua=1 (accessed March 25, 2020).

9. Public Health Foundation. Competency Assessments for Public Health Professionals. (2014). Available online at: http://www.phf.org/resourcestools/Pages/Competency_Assessments_For_Public_Health_Professionals.aspx

10. Edgar M, Mayer JP, Scharff DP. Construct validity of the core competencies for public health professionals. J Public Health Manag Pract. (2009) 15:E7–16. doi: 10.1097/PHH.0b013e31819c1452

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Boone HN Jr, Boone DA. Analyzing likert data. J Ext. (2012) 50:1–5.

Google Scholar

12. USAID Center for Development Information Evaluation. Performance Monitoring & Evaluation TIPS Using Direct Observation Techniques. USAID Center for Development Information and Evaluation (1996). Available online at: https://web.archive.org/web/20130302141450/http://transition.usaid.gov/policy/evalweb/documents/TIPS-UsingDirectObservationTechniques.pdf

Google Scholar

13. Patton MQ. Qualitative Research and Evaluation Methods. 4th ed. Thousand Oaks, CA: SAGE Publications, Inc. (2015).

Google Scholar

14. Sucaldito NL, Tayag EA, Roces MCR, Malison MD, Robie BD, Howze EH. The Philippines Field Management Training Program (FMTP): strengthening management capacity in a decentralized public health system. Int J Public Health. (2014) 59:897–903. doi: 10.1007/s00038-014-0603-5

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Thalheimer W. Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form. Work Learning Press. (2016). Available online at: http://site.ebrary.com/id/11209760 (accessed May 6, 2020).

16. Dennis C. Strategic planning—a health system operational perspective. J Hosp Manag Health Policy. (2019) 3:3–32. doi: 10.21037/jhmhp.2019.10.03

CrossRef Full Text

17. World Health Organization editor. Monitoring the Building Blocks of Health Systems: A Handbook of Indicators and Their Measurement Strategies. Geneva: World Health Organization (2010). p. 92.

Google Scholar

18. Men B, Grundy J, Cane J, Rasmey LC, An NS, Soeung SC, et al. Key issues relating to decentralization at the provincial level of health management in Cambodia. Int J Health Plann Manage. (2005) 20:3–19. doi: 10.1002/hpm.793

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health. (2015) 42:533–44. doi: 10.1007/s10488-013-0528-y

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Levin-Rozalis M. Evaluation and research: differences and similarities. Can J Program Eval Rev Can Eval Programme. (2003) 18:1–31.

Google Scholar

21. Patton MQ. Utilization-Focused Evaluation. 4th ed. Thousand Oaks, CA: SAGE Publications, Inc (2008). p. 688.

Google Scholar

22. Noble H, Smith J. Issues of validity and reliability in qualitative research. Evid Based Nurs. (2015) 18:34–5. doi: 10.1136/eb-2015-102054

PubMed Abstract | CrossRef Full Text | Google Scholar

23. National Academies of Sciences. Global Health Risk Framework: Governance for Global Health: Workshop Summary. (2016). Available online at: https://www.nap.edu/catalog/21854/global-health-risk-framework-governance-for-global-health-workshop-summary (accessed March 25, 2020).

Google Scholar

24. Fraser M, Castrucci B, Harper E. Public health leadership and management in the era of public health 3.0. J Public Health Manag Pract. (2017) 23:90–2. doi: 10.1097/PHH.0000000000000527

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Ahern S, Loh E. Leadership during the COVID-19 pandemic: building and sustaining trust in times of uncertainty. BMJ Leader. (2020) 5:266–9. doi: 10.1136/leader-2020-000271

PubMed Abstract | CrossRef Full Text | Google Scholar