The Effect of Using Different Competence Frameworks to Audit the Content of a Masters Program in Public Health

Objectives: (1) To quantify the effect of using different public health competence frameworks to audit the curriculum of an online distance learning MPH program, and (2) to measure variation in the outcomes of the audit depending on which competence framework is used.

Study Design: Retrospective audit.

Methods: We compared the teaching content of an online distance learning MPH program against each competence listed in different public health competence frameworks relevant to an MPH. We then compared the number of competences covered in each module in the program’s teaching curriculum and in the program overall, for each of the competence frameworks used in this audit.

Results: A comprehensive search of the literature identified two competence frameworks specific to MPH programs and two for public health professional/specialty training. The number of individual competences in each framework were 32 for the taught aspects of the UK Faculty of Public Health Specialist Training Program, 117 for the American Association of Public Health, 282 for the exam curriculum of the UK Faculty of Public Health Part A exam, and 393 for the European Core Competencies for MPH Education. This gave a total of 824 competences included in the audit. Overall, the online MPH program covered 88–96% of the competences depending on the specific framework used. This fell when the audit focused on just the three mandatory modules in the program, and the variation between the different competence frameworks was much larger.

Conclusion: Using different competence frameworks to audit the curriculum of an MPH program can give different indications of its quality, especially as it fails to capture teaching considered to be relevant, yet not included in an existing competence framework. The strengths and weaknesses of using competence frameworks to audit the content of an MPH program have largely been ignored. These debates are vital given that external organizations responsible for accreditation specify a particular competence framework to be used. Our study found that each of four different competence frameworks suggested different levels of quality in our teaching program, at least in terms of the competences included in the curriculum. Relying on just one established framework missed some aspects of the curriculum included in other frameworks used in this study. Conversely, each framework included items not covered by the others. Thus, levels of agreement with the content of our MPH and established areas of competence were, in part, dependent on the competence framework used to compare its’ content. While not entirely a surprising finding, this study makes an important point and makes explicit the challenges of selecting an appropriate competence framework to inform MPH programs, and especially one which recruits students from around the world.

Introduction

The Institute of Medicine and the American Public Health Association (1) recommended the use of competence frameworks to inform the content of public health teaching programs. This reflected concerns about variation in the content of public health professional training programs (2), and a growing trend to competence based education in other health professions (3–5). This has resulted in an increase in the number and scope of competence frameworks for public health training and professional practice (1, 6), along with different interpretations of such terms (7). In this paper, we have defined competence to mean a “complex set of measurable behaviors made up of knowledge, skills, and attitudes that can be shown to predict and measure effective performance” (8).

One of the ways in which different competence frameworks for MPH programs are being used is as part of internal and external accreditation processes. Indeed, there is no standardized way within (6, 9), or between countries for assuring the quality of MPH programs. This becomes more challenging as online distance learning has opened the way for students to be registered with a program from another country. Integrating students around the world to an MPH program has the potential to further help prepare graduates to then effectively work across a challenging and diverse global context. It does raise questions about the teaching curriculum, and how to maximize opportunities from a more global student-economy. Part of this needs to be informed by an appropriate quality assurance strategy, which is responsive to a global educational environment, and in meeting the needs of students regardless of where they live and study.

The current study stems from work at The University of Manchester, which has run an online distance learning masters program for 12 years. Each year, this program typically recruits around 70 students from the UK and other European countries, and around 30–40 students from countries outside of the European Union (10). To graduate with an MPH, our students need to pass three mandatory modules (Evidence Based Practice, Fundamentals of Epidemiology, and Introduction to Biostatistics) and a further five modules, which they select from a choice of 17 optional modules (available at the time of study). Each module reflects 10 weeks of teaching, around 150 study hours, and completion of two, written assignments. This is followed by a 12,000–15,000 word dissertation (11, 12), often relating to students’ vocational experience and future career direction (12). Previous research based on the Manchester MPH found that a diverse virtual student classroom enhanced students’ total learning experience (11). Yet from an educator’s perspective, it creates challenges in ensuring a curriculum which is suitable for all of the MPH students, regardless of where they live, and how this then ought to be quality assured, and by whom. These important questions have not been considered in the public health literature and warrant further investigation.

Aims

The aim of the current study was to inform the further development and delivery of a high quality online distance learning MPH program. Its primary objective was to complete an audit of the existing curriculum, across the 20 teaching modules, using an appropriate competence framework. Its secondary objective was to quantify the effect on the perceived content of the curriculum when compared using a different competence framework. In particular, we wanted to determine how these varied depending on the optional modules taken by the students.

Materials and Methods

We reviewed the literature to identify suitable competence frameworks for professional public health education. We wanted to review our course against competence frameworks that reflected countries from which the majority of our students were recruited. The online MPH at University of Manchester includes a broad curriculum covering core aspects relevant to public health professionals. As such, we excluded competence frameworks that focused entirely on a specific sub-set of competences [e.g., epidemiology (13) or informatics (14)].

Each item of competence listed in the frameworks was copied into an Excel spreadsheet. This was sent to all module instructors who were given information about how to review the content of their modules against the competences listed in the spreadsheet. A simple coding frame was designed to capture information about the coverage of teaching material in each course module relating to the competences: (1) fully covered, (2) partly covered, (3) comparably covered, and (4) not covered at all. The code “comparably covered” could be used when the direct competence was not covered, but for which comparable teaching material was included. The focus of the primary audit was to identify competences that were not covered at all in any of the teaching material. Therefore, we regrouped the original four categories into two dichotomous variables: (1) competence covered in some way, or (2) competence not covered at all. The module instructors used their expertise to reach a judgment as to which of these four categories was most applicable to each of the competence items presented. The methods for data collection were piloted including the face validity of the competence frameworks in their presented format. Inter-rater reliability was explored with the principal investigator cross checking the results from one of the core modules.

It is difficult to identify any typical groups of students across the program: it is affected by whether they are full or part time, and which optional modules they select, among other things. Consequently, we carried out an analysis using two dummy or model student’s that we created ourselves based on two distinctive possible pathways through the programme. Thus, one student pathway consisted of more quantitative and/or epidemiology focused optional modules. The second pathway focused on more narrative and health promotion focused optional modules. Therefore, our two model students for exploratory purposes with this audit were assumed to consist of:

Student One

The three mandatory modules (Evidence Based Practice, Fundamentals of Epidemiology & Introduction to Biostatistics) plus five optional modules, which were Communicable Disease Control, Advanced Epidemiology, Evidence Based Public Health, Impact Information Evaluation, and Health Economics.

Student Two

The three mandatory modules (Evidence Based Practice, Fundamentals of Epidemiology & Introduction to Biostatistics) plus five optional modules, which were Health Promotion Theory Methods, Management, Primary Healthcare, Qualitative Research Methods, and Health Promotion and Prevention.

Analysis

The analysis examined the proportion of competences deemed to be covered in the teaching content of each module, for each of the competence frameworks. This then allowed the calculation of the number of competence items covered in the three mandatory modules (Evidence Based Practice, Fundamentals of Epidemiology & Introduction to Biostatistics) and for the total number of competences covered across all of the twenty units available on the program. A sub analysis then focused on the outcome when applied to a potential learning experience using two model students.

Results

Competence Frameworks

The literature review identified two competence frameworks specific to MPH education. The Association of Public Health in America competence framework (ASPH) consisted of 117 competence items (15), and the European Core Competencies for MPH Education (ECCMPHE) had 393 competences listed (8). We also included two frameworks adapted for the audit, based on the UK Faculty of Public Health Part A membership exam (FPH) with 282 competences (16), and the educational/knowledge domains from the UK Specialist Registrars Training in Public Health (STP) with 32 listed items (16) (Table 1). Thus in total, the audit covered 824 competence items that were assessed against the curriculum for each of the 20 course modules available on the entire MPH program.

TABLE 1

Table 1. The number of competence items covered in each of the competence frameworks.

Audit of Three Mandatory Modules

The first stage of the analysis was restricted to the three mandatory modules in the online MPH program. The proportion of competences that were identified as been covered (fully, partly or comparably) in the teaching material for each of the four competence frameworks were 50% (n = 16) for the STP, 30% (118) for the ECCMPHE, 29% (n = 34) for ASPH, and 23% (n = 66) for the FPH (Table 2).

TABLE 2

Table 2. The number (%) of competences included in the curriculum for the three mandatory course modules (Evidence Based Practice, Introduction to Biostatistics, and Fundamentals of Epidemiology).

Audit of the Three Mandatory Modules Plus all of the Possible 17 Optional Modules

The second stage of the analysis focused on the content across the entire teaching program (i.e., the three mandatory modules and all of the 17 optional modules (although in practice, a student would take the three mandatory and five optional modules). In this analysis, the proportion of competences that were identified as been covered (fully, partly or comparably) in the teaching material for each of the four competence frameworks were 96% (n = 377) for the ECCMPHE, 95% (n = 267) for the FPH, 91% (n = 29) for the STP, and 88% (n = 103) for the ASPH (Table 3).

TABLE 3

Table 3. The number (%) of competences included in the curriculum across all 20 course modules available.

Audit of Two Example Student Pathways

The final analysis used two model/possible student pathways through the program to explore the effect of using the different competence frameworks on the outcomes from the audit (Table 4).

TABLE 4

Table 4. The number (%) of competences included in the curriculum for two proxy student pathways through the course.

For Student One (the quantitative/epidemiology focused pathway), the curricula covered 72% (n = 23) of competences for the STP framework, 64% (n = 252) for the ECCMPHE, 48% (n = 57) for the ASPH, and 45% (n = 128) for the FPH. For Student Two, this ranged from 84% (n = 27) for the STP, 77% (n = 90) for the ASPH, 66% (n = 186) for the FPH, and 65% (n = 254) for the ECCMPHE (Table 4).

Discussion

All MPH programs need to be fit for purpose, and have the potential to educate students to become effective public health professionals once they graduate. As such, the audit described in the current study is just one of a number of quality assurance initiatives we implement across the teaching program. We found that a detailed audit of each of the 20 modules on an online distance learning MPH was a time consuming process. Each course unit had to assess the content of their module against 824 individual competences. Despite such a task, many of our course leaders reported that it helped them reflect on the content of their program, and to then see links in teaching areas across the different modules. It then led to important discussions about what if any changes needed to be made to the content of the course, including consideration of developing further modules.

One of the main issues when we are to repeat this audit is about deciding on the appropriateness of using just one competence framework. This is the ideal scenario in an effort to reduce time to complete the audit, and we would suggest that this is the most likely option taken by other educationalists who might be/have embarking on such an audit. But the findings of our exploratory audit highlight some of the difficulties in selecting the most ideal framework to use for this purpose. Admittedly, the audit only included four different frameworks, yet this had 824 individual competences to assess the curriculum against, though fewer if it had been restricted to the two frameworks specifically developed for MPH programs. Despite this, we still found important variations in the perceived quality of our program depending on which of the four frameworks was used. We did not find any competence frameworks that were specific to public health in low- and middle-income countries. Of course, MPH programs include many generic skills that can be applied in a wide range of contexts. At the same time, low- and middle-income countries experience particular public health challenges, which require specific approaches for investigation and effective intervention. Their populations remain in desperate need of suitably trained health professionals (19, 20). Such a disconcerting observation has been reported in the recent past (21), and it is encouraging at least to see some further investigation into this topic (22, 23). We also need to ensure that we use competence frameworks relevant to other developing countries beyond north America and the European Union. This will include countries like China, which have dramatically increased the number of students coming to the UK (or studying online) to study for a degree (24).

Clearly, our audit highlighted potential gaps in the content of our MPH program, regardless of which competence framework it was compared against. This was not a surprising finding because the program was never designed to meet all of the competences outlined in any single framework. However, the audit did identify relatively small additions and changes that course leaders made, and provided justification for developing two new modules on the program. At the same time, the audit could not capture all of the teaching included in the program but which is not a competence on any of the frameworks used in this study. These additional areas of teaching are one of the great strengths of this online MPH, providing an extensive learning experience for a wider set of circumstances, and one which we believe is closely aligned with students previous experience, and future aspirations. These strengths are one of the reasons the program was been consistently rated as exemplary by senior external reviewers, along with evidence of extremely satisfied students.

We remain committed to proving an enhanced learning experience for our MPH programs, and that to do otherwise would be a disservice to students and to the populations they then serve. In recent times, we have observed growing pressures for the external accreditation of public health education and even public health departments. This is a complex and challenging debate, although aims such as increasing public accountability while delivering cost-effective interventions are ones we agree are laudable. One concern that is reflected in our current study relates to the methods used to assess an MPH as part of an accreditation process. Thus, we found that in terms of competences, the content of a teaching curriculum could vary in potentially important ways, depending on the competence framework used for its comparison. We stress the importance of this finding, especially when some organizations stipulate a particular competence framework to be used (e.g., Council on Education for Public Health, http://ceph.org/ Accesssed 5 December, 2014). Thus, a curriculum that aims to meet the content of a specific competence framework might be missing out other relevance competences included in a different framework.

A weakness of the current study is that we cannot rule out the possibility of measurement error, with course leaders interpreting the competences in different ways, and with the added challenges of determining what teaching something might actually mean. We accept that we used a relatively basic coding structure and that we might have under or over estimated the content compared with each of the competence frameworks. However, we anticipate that these biases would remain in the same direction across each of the four competence frameworks used in our audit. It is certainly true that our audit helped to stimulate debate across the academic team, and identify potential gaps in the programs content. While there are debates about the value of competence audits/mapping in higher education (25, 26), overall we found this was a positive, albeit time consuming process and it is doubtful that we would use all four frameworks again in future. We also experienced some practical difficulties in that the frameworks were presented in different ways, with lack of clarity if something was an actual competence or a subheading/heading, and with some confusion about the terminology. All of this can easily be rectified, and we encourage any competence frameworks to be presented in a standard format (27, 28) and downloaded as an editable spreadsheet reflecting that used for the Part A exam curriculum of the UK Faculty of Public Health (17). We had a positive response after raising our concerns with a leading author of the ECCMPHE (Birt C, 2014, personal communication).

Conclusion

Few specific competence frameworks exist that can be used to audit the teaching curriculum of an MPH. While time consuming, we found that this process was a positive experience for individual course leaders and the academic team. Our study highlights the challenges and effects of selecting any single competence framework against which the curriculum is assessed. This is greater when recruiting students from other countries, and especially when designing on online distance learning program with students studying concurrently from all over the world. It is important that students, program directors, and organizations responsible for external quality assurance and accreditation are aware of the strengths and weaknesses of these and other approaches for quality assurance. We remain concerned about the apparent lack of specific competence frameworks relevant to MPH education for low- and middle-income countries and others outside of North America and the European Union. At the same time, we would be skeptical of one international competence framework to guide the teaching curriculum of MPH programs around the world, and welcome some level of heterogeneity and flexibility at any one time. We will endeavor to proceed with caution, especially while the evidence to support this approach is wanting.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We would like to thank Emeritus Professor Richard Heller, Dr. Chris Birt, and Judith Clegg for comments on earlier revisions and general support for this project. Ethical approval was not required. No external funding was received and there are no competing interests.

References

1. Gebbie K, Rosenstock L, Hernandez L. Who Will Keep the Public Healthy? Educating Public Health Professionals for the 21st Century. Washington, DC. (2003). Available from: http://www.nap.edu/catalog.php?record_id=10542

Google Scholar

2. Fineberg H, Green G, Ware J, Anderson B. Changing public health training needs: professional education and the paradigm of public health. Annu Rev Public Health (1994) 15:237–57. doi: 10.1146/annurev.pu.15.050194.001321

CrossRef Full Text | Google Scholar

3. Horton S. Introduction – the competency movement: its origins and impact on the public sector. Int J Public Sector Manag (2000) 13(4):306–18. doi:10.1108/09513550010350283

CrossRef Full Text | Google Scholar

4. Swider S, Krothe J, Reyes D, Cravetz M. The quad council practice competencies for public health nursing. Public Health Nurs (2013) 30(6):519–36. doi:10.1111/phn.12090

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Bernabé E, Ludeña MA, Beltrán-Neira RJ. Self-perceived public health competency among recent dental graduates. Int Dent Educ (2006) 70(5):571–9.

PubMed Abstract | Google Scholar

6. Public Health Foundation. Council on Linkages: Core Competencies for Public Health Professionals. Washington, DC. (2010). 22 p. Available from: http://www.phf.org/resourcestools/Documents/Core_Competencies_for_Public_Health_Professionals_2010May.pdf

Google Scholar

7. Epstein RM. Defining and assessing professional competence. JAMA (2002) 287(2):226. doi:10.1001/jama.287.2.226

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Birt C, Foldspang A. Publication No.6: European Core Competencies for MPH Education (ECCMPHE). Brussels. (2011). Available from: http://www.wfpha.org/tl_files/images/NewsletterDecember2011/AndersCompetencesno6-1.pdf

Google Scholar

9. Otok R, Levin I, Sitko S, Flahault A. European accreditation of public health education. Public Health Rev (2011) 33(1):30–8.

Google Scholar

10. University of Manchester. Master in Public Health. (2014). Available from: manchester.ac.uk/mph

Google Scholar

11. Gemmell I, Harrison R, Clegg J, Reed K. Internationalisation in post graduate education: student views on the impact of learning alongside students from other countries on an online distance learning master of public health programme. Innov Educ Teach Int (2015) 52(2):137–47. doi:10.1080/14703297.2014.881264

CrossRef Full Text | Google Scholar

12. Harrison R, Gemmell I, Reed K. Student satisfaction with a web-based dissertation course: findings from an international distance learning Masters programme in public health. Int Rev Res Open Dist Learn (2014) 15(1):182–202.

Google Scholar

13. Moser M, Rahiah K, Ibrahim M. Epidemiology core competencies for master of public health students. Public Health Rep (2008) 123(Suppl1):59–66.

PubMed Abstract | Google Scholar

14. Carroll PWO. Informatics Competencies for Public Health Professionals. Washington: Public Health (2002).

Google Scholar

15. ASPH. Master’s Degree in Public Health Core Competency Model. Version 2.3. (2006). 40 p. Available from: http://www.aspph.org/wp-content/uploads/2014/04/Version2.31_FINAL.pdf

Google Scholar

16. UK Faculty of Public Health. (2014). Available from: fph.org.uk

Google Scholar

17. Faculty of Public Health. Part A Exam Content. (2014). Available from: http://www.fph.org.uk/part_a_exam

Google Scholar

18. Faculty of Public Health. Public Health Specialty Training Curriculum. (2010). 131 p. Available from: http://www.fph.org.uk/uploads/2010MASTERPHCurriculum.pdf

Google Scholar

19. Crisp N, Chen L. Global supply of health professionals. N Engl J Med (2014) 370(10):950–7. doi:10.1056/NEJMra1111610

CrossRef Full Text | Google Scholar

20. Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet (2010) 376(9756):1923–58. doi:10.1016/S0140-6736(10)61854-5

CrossRef Full Text | Google Scholar

21. Reynolds F, Heller RF. Peoples-uni: developing public health competences – lessons from a pilot course module. Int J Emerg Technol Learn (2008) 3(Special Issue 1):60–5.

Google Scholar

22. Cole DC, Davison C, Hanson L, Jackson SF, Page A, Lencuch R, et al. Being global in public health practice and research: complementary competencies are needed. Can J Public Health (2011) 102(5):394–7.

PubMed Abstract | Google Scholar

23. Zwanikken PA, Alexander L, Huong NT, Qian X, Valladares LM, Mohamed NA, et al. Validation of public health competencies and impact variables for low- and middle-income countries. BMC Public Health (2014) 14:55. doi:10.1186/1471-2458-14-55

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Wen S-H, Ma X-Q, Liu L-J, Ren W-M, Xu F, He X. A survey on core competencies of Master of public health students in China. Public Health (2013) 127(10):964–6. doi:10.1016/j.puhe.2013.04.028

CrossRef Full Text | Google Scholar

25. Wang C-L. Mapping or tracing? Rethinking curriculum mapping in higher education. Stud High Educ (2014):1–10. doi:10.1080/03075079.2014.899343

CrossRef Full Text | Google Scholar

26. Ervin L, Carter B, Robinson P. Curriculum mapping: not as straightforward as it sounds. J Vocat Educ Train (2013) 65(3):309–18. doi:10.1080/13636820.2013.819559

CrossRef Full Text | Google Scholar

27. Parry SB. Just what is a competency? (and why should you care? Training (1998) 35(6):58–62.

Google Scholar

28. Genat B, Robinson P. New competencies for public health graduates: a useful tool for course design. Aust N Z J Public Health (2010) 34(5):513–6. doi:10.1111/j.1753-6405.2010.00599.x

PubMed Abstract | CrossRef Full Text | Google Scholar