External assessment of medical education quality: indicative model development considering paradox of skill

This study proposes an approach to the external evaluation of medical education programs' quality based on a combination of indicators, including international rankings, external stakeholders' input, and independent agencies' assessments. We modify the success equation with a detailed consideration of the skill component and its decomposition into internal and external quality assurance elements along with authority. We carried out a bibliometric analysis regarding the problem of medical education quality assessment in the context of achieving sustainable development goals. We described the calculation model of external quality assessment indicators through the algorithms of independent education quality assurance agencies' activity and rating indicators shown in the modified Mauboussin's equation. The model considers the economic component (the consequence of achievement) of skill, which is expressed in raising funds from external sources to implement educational and scientific activities. The proposed algorithm for assessing the educational program quality can be applied to benchmark educational program components, complete educational programs within the subject area, and the educational institution for different areas. We propose a “financial” model for educational program quality based on the analysis results. The model makes it possible to determine the need for additional focused funding of the educational program based on the individual analysis of the external evaluation criteria of the achievement level. This study analyzes the accreditation results of more than 110 educational programs in 2020 and 8 months of 2021 within the direction 22 “Medicine” (according to the national classification of fields of knowledge) (state and private Ukrainian medical universities).

This study proposes an approach to the external evaluation of medical education programs' quality based on a combination of indicators, including international rankings, external stakeholders' input, and independent agencies' assessments. We modify the success equation with a detailed consideration of the skill component and its decomposition into internal and external quality assurance elements along with authority. We carried out a bibliometric analysis regarding the problem of medical education quality assessment in the context of achieving sustainable development goals. We described the calculation model of external quality assessment indicators through the algorithms of independent education quality assurance agencies' activity and rating indicators shown in the modified Mauboussin's equation. The model considers the economic component (the consequence of achievement) of skill, which is expressed in raising funds from external sources to implement educational and scientific activities. The proposed algorithm for assessing the educational program quality can be applied to benchmark educational program components, complete educational programs within the subject area, and the educational institution for di erent areas. We propose a "financial" model for educational program quality based on the analysis results. The model makes it possible to determine the need for additional focused funding of the educational program based on the individual analysis of the external evaluation criteria of the achievement level. This study analyzes the accreditation results of more than educational programs in and months of within the direction "Medicine" (according to the national classification of fields of knowledge) (state and private Ukrainian medical universities). KEYWORDS medical education, education quality, education funding, education evaluation, bibliometric analysis, sustainable development goals

. Introduction
The quality of medical education is a popular area of study by scientists in various applications: -at the national level, with proposals for the development of strategies for improving educational programs (1); -at the regional level, taking into account the specifics of medical programs (2, 3); -in terms of institutional accreditation in general (4); -communication with practitioners and employers (5,6); -discussions about the importance and influence of individual indicators on the quality of medical educational programs (7-9); -the role of stakeholders in ensuring the quality of medical education (10, 11), etc.
As the analysis of the data presented above shows, assessing the quality of education (especially external, which is implemented by independent agencies) is carried out qualitatively (the presence or absence of indicators according to various criteria). The lack of a quantitative assessment complicates the evaluation process to some degree. A tool that can quantify at least some of the indicators is needed.
External accreditation of educational programs by the National Agency for Higher Education Quality Assurance is currently the only tool for assessing the quality of medical education in universities under the Ukrainian Ministry of Education and Science and the Ministry of Health.
State regulation of educational activities (12,13) to meet the Sustainable Development Goals (14, 15) can only partially satisfy the requirements of educational services consumers. Innovation strategies (16,17), providing leadership in the implementation of breakthrough technologies (18, 19), choosing the most effective forms of the educational process (20,21), creating a successful socio-economic model of the university (22)(23)(24), promoting the university brand on the market of educational and scientific services (19, 25), evaluation of educational program quality by stakeholders (26,27), and the openness of information about the educational program (28,29)-all these characteristics are the subject of examination during the implementation of external education quality assessment.
This study demonstrates that quality assurance in education is a multifactorial "experiment". All components of the socio-economic state of the region are essential, especially the organization of training in different periods, the influence of external factors on the demand for educational programs, etc. (30-52).
The research is based on the example of data analysis in the field of "Medicine"; the research results can be further applied to other educational areas.
The data of bibliometric analysis of the keyword "medical education" states that various aspects of quality assurance are an integral part of the medical education analysis (Figure 1, tool-VOSviewer, analyzed 109,000 documents in all areas, selected the most cited 2,000 for the period 2016-2020, the minimum number of mentions of keywords: 25, excluding keywords related to specialized medical terminology). Interestingly, a significant number of keywords presented in Figure 1 are related to Sustainable Development Goals (Figure 2, tool-VOSviewer, analyzed 8,000 documents in the field of "medicine" for the period 2016-2020, the minimum number of mentions of keywords: 50, excluding keywords related to specialized medical terminology), which determines the additional relevance of this study.
Furthermore, the analysis was performed on data related to life expectancy and healthy life expectancy indices and indicators of the Universitas 21 rating leaders. The analysis involved constructing a table of relevant indicators and examining the top 10 countries in the ranking of Universitas 21 and the first 36 countries in the ranking and their respective indicators in the Life Expectancy Index 2020 and the WHO Healthy Life Expectancy Index 2018. The analysis aimed to explore the relationship between these variables and determine whether there is a significant correlation or associ ation between them. We utilized a simple linear regression model and interpretation of its coefficients to illustrate the interconnection between said indices. For the first round of analysis, we selected the Life Expectancy Index 2020 rating as the dependent variable, with the national educational Universitas 21 ranking as an independent variable. The second round of analysis accounted for data from the WHO Healthy Life Expectancy Index 2018 as a dependent variable and Universitas 21 ranking as an independent variable. Success (S) in achieving results in any activity depends on two components: skill (M) and luck (L). This approach is reflected in Michael Mauboussin's success equation (53), which is a weighted linear function of skill and luck: where the value a reflects the relative weight of skill in the range from 0 to 1, and as a result, b = 1 -a. Suppose we relay this approach to assessing education quality, in that case, we encounter the term "luck" as a very limited set, which is reduced to success not in achieving the quality of the educational program, but luck regarding the contingent involved in the current educational program. Furthermore, all the success, which is directly related to the educational program quality, can be measured by skill, and a value in (1) is around one. Thus, we can only talk partly about the paradox of skill in the case of the "education quality" dynamic system. For the system "education quality", the term "skill" in (1) has three components: 1. internal quality assurance of the educational program q.i. 2. mobility in improving the educational program in response to the challenges mo. 3. external evaluation by stakeholders and independent agencies q.e.
However, the success of the university's educational activities depends not only on skill and (at a minimum) luck but also on a third parameter-the authority of the university (R), which is determined, inter alia, by world rankings. Thus, the equation of Mauboussin's success can be upgraded to the following form: Frontiers in Public Health frontiersin.org . /fpubh. . where a + b + c = 1.
Considering components of the skill defined above, Eq.
(2) will take on the form External evaluation of the education quality by stakeholders and independent agencies is one of the most effective ways to determine the competitiveness level of an educational program. This tool is effective in cases when quality assessment indicators are clearly defined.
A multi-factor model for assessing the quality of medical education is proposed, which is based on the view from the "inside" (quality indicators of the components of the educational program and the quality assurance system) and "from the outside" (indicators of the success of the educational program according to the evaluation methods of the Ministry of Education and Science, the National Agency for Higher Education Quality Assurance).

. Materials and methods
Since rating agencies, in addition to general university rankings, also implement university rankings by area (which actually assess educational program(s) quality within a field), additional mechanisms for educational program benchmarking are emerging.
Given the similar nature of the external and rating university evaluation, the success in Eq. (2) for the university can be interpreted as follows: where The success of the educational program is also determined by Eq. (4). However, given the insignificance of luck's contribution to educational program quality, equation (5) takes the form In this study, we try to describe the component EXT in (7), considering both the rating agencies' requirements for education quality assessment by area and experience of external accreditation of educational programs by the National Agency for Higher Education Quality Assurance (Ukraine) in 2020-2021. We focus on determining the indicators QE and R because the weight factors a and c may vary for a particular evaluation period.

Results
The presence of Ukrainian institutions providing higher medical education in the world rankings is currently a future matter. Advances on the path to the entry of Ukrainian universities to the branch medical ratings are possible due to benchmarking the main rating indicators and understanding the landscape and depth of their content. In addition, one should consider that industry rankings do not assess specific programs but the whole subject area. This fact determines the additional relevance of developing procedures aimed at the evaluation of educational programs. An attempt to evaluate medical education and the institutions that provide it was made in the ranking of the top 100 faculties of domestic universities (Top best faculties of Ukraine according to Forbes, 2021) and the ranking of the top 200 universities in Ukraine (HEI rating "TOP-200 Ukraine 2021", 2021). Indirectly, these ratings can be used to assess educational programs' quality, but they have a low level of detail in the assessment as their main drawback.
Assessment of educational institutions' quality is also carried out according to the scientific component (54). However, it is more likely to be named an outcome of quality education, expressed in the level of implementation of research results. Such attestation is currently implemented by the Ministry of Education and Science of Ukraine for subordinated universities, which does not allow us to compare the results with the indicators of universities subordinated to the Ministry of Health of Ukraine. In addition, this tool does not allow the allocation of a separate sector "medicine, " as it evaluates the activities of universities in the field of "Biology and Health." Based on the assessment of certain university performance indicators, the Ministry of Education and Science of Ukraine has developed a methodology for allocating state budget expenditures (55). Since 1 January 2021, this method is also applicable to universities subordinated to the Ministry of Health of Ukraine. Among other indicators, this technique uses the indicator of international recognition and the indicator of graduate employment, which are part of the second term of Eq. (7).
Using the data of ratings of the Life Expectancy Index 2020, the WHO Healthy Life Expectancy Index 2018, and Universitas 21: Ranking of National Higher Education Systems 2020, we constructed a table of relevant indicators of Universitas 21 rating leaders related to life expectancy and healthy life expectancy. Table 1 presents the top 10 countries in the ranking of Universitas 21 plus Ukraine. For a more detailed analysis of the relationship between the ratings, the first 36 countries of the Universitas 21 ranking and their respective indicators in the Life Expectancy Index 2020 and the WHO Healthy Life Expectancy Index 2018 were considered. Figure 3 demonstrates the results of the life expectancy assessment depending on the national level of higher education. The coefficient at x in the linear regression equation can be interpreted as follows: with an increase of the Universitas 21 index by one unit, the value of the Life Expectancy Index increases .
/fpubh. .    Table 2 presents European universities ranking in the "Medicine" field according to QS World University Rankings. The university's rank is determined by the value of "Overall Score". It is important to consider that the positions of compared universities in the ranking may differ significantly despite slight differences in the value of the "Overall Score." In the general case, the assessment of university performance according to the QS methodology (56) is based on six key factors:  This study has an interest in analyzing the data related to the EXT component in (6) in the field of "Medicine". Therefore, we use QS World University Rankings by Subject, which takes into account the following indicators: • Academic Reputation • Employer Reputation • Citations per paper • H-index Indicators "Academic Reputation" and "Employer Reputation" are based on the data obtained during surveys of academic institutions' employees and employers, respectively. Indicators "Citations per paper" and "H-index" are used to measure the scientific productivity of the university by subject. The principles and reasons for using these indicators are described in detail in (57). The weights of mentioned indicators differ by area, hence within the "Medicine" subject, the following set is used: Academic Reputation, 40%; Employer Reputation, 10%; Citations per Paper, 25%; and H-index, 25%. For universities training health professionals, one can see that priority is given to reputation in academia and scientific productivity, while reputation among employers clearly recedes into the background.
We propose using QS World University Rankings by Subject in the model discussed in this study: Academic Reputation and Employer Reputation are included in the component cA of Eq. (6), and Citations per Paper and H-index are accounted for by the component aQE. Weight factors for these indicators should be consistent with the weight factors of the other components in terms of Eq. (7). However, the QS Rankings approach provides a general idea of prioritization in university performance indicators in the "Medicine" subject framework.
The methodology for external quality assurance of educational programs incorporated by the National Agency for Higher Education Quality Assurance has the following features [the description is based on the regulatory framework of the National Agency (58)]: Accreditation is carried out by nine (for the bachelor and master level of higher education) or 10 (for the phd level of higher education) criteria: • Criterion 1. Project and goals of the educational program.
• Criterion 2. Structure and content of the educational program.
• Criterion 3. Access to educational programs and education results acknowledgment.
• Criterion 4. Learning and teaching within the educational program.
• Criterion 7. The learning environment and material resources.
• Criterion 8. Internal quality assurance of the educational program.
• Criterion 10. Learning through research (for PhD level, not considered in the current study).
Each criterion evaluates the educational program and educational activities within the program according to the evaluation scale, which covers four levels of compliance.
• Level "A": the educational program and activities within the program fully meet the assessed criteria and have an innovative/exemplary character.
• Level "B": the educational program and educational activities within the program generally meet the assessed criterion with shortcomings that are not significant.
• Level "E": the educational program and/or educational activities within the program generally do not meet the assessed criterion, but identified shortcomings can be eliminated within 1 year; • Level "F": the educational program and/or educational activities within the program do not meet the assessed criteria.
The identified shortcomings are fundamental and/or cannot be eliminated within 1 year.
The educational program can receive one of four grades: • Accredited and marked as exemplary (A).
The following conditions for determining accreditation grade are proposed based on the number of corresponding criteria grades: F : E > 2 and/or F > 0. (11) This study analyzes the accreditation results of more than 110 educational programs in 2020 and 8 months of 2021 within the direction 22 "Medicine" (according to the national classification of fields of knowledge) (state and private medical universities).
The above evaluation criteria have been considered in previous studies (59)(60)(61)(62)(63)(64)(65) as those that have a significant impact on the quality of educational programs in the process of external expertise. Table 3 is constructed on the basis of the abovementioned 110 programs' accreditation data. For each educational program and corresponding evaluation criteria, we present the ratio "Success/Innovation". Success (k s ) is the percentage of grades A (n A ) and B (n B ) of the total accreditations number in the program for each area (N) and relevant criteria. Innovation (k i ) is the percentage of grades A (n A ) of the total accreditations number in the program for each area (N) and relevant criteria.
The criteria were combined into meaning-based complexes for a more effective and simple assessment of the success and innovation of educational activities within the educational program and the whole direction. "Content and Potential" (CP) complex combines criteria 1, 2, and 7; "Algorithms" (A) complex combines criteria 3 and 5; and "Personalities" complex combines criteria 4, 6, 8, and 9. Further evaluation of these complexes allows for a comprehensive assessment of both the educational program and the set of programs. Table 4 contains the Success/Innovation ratio calculations for the described criteria complexes for educational programs of 22 "Medicine" direction. Success (k s ) and innovation (k i ) are included in the QE component of Eq. (7), as they pose a convenient and accessible tool for external quality analysis. Equation (7) in explicit form, adapted to assess educational activities quality is: The component QE presenting an external quality evaluation of educational activities consists of the following components: b is the component determined by the basic funding from the Ministry of Education and Science. In the case of university-level assessment, this component is formed for the whole institution. for F e ≫ b is limited by 2.
3 j=1 k ij k sj is the component formed by the sum of Success/Innovation ratios for each direction and the relevant criteria complexes. In the case under consideration, the criteria are divided into three complexes, so the maximum value of the sum is 3.
University authority level R depends on two components, the first of which (r gen ) characterizes the university's position in the rankings (international and/or national) as a whole institution. The second component r loc characterizes the direction's position in ratings by area. The maximum value for both components is 1, while the second component allows accounting for situations when the rating by area does not exist or the university is not included in such rating. In these situations, it is recommended to set an indicator n m=1 r ocm individually in each evaluation process for internal authority level assessment and to use the recommended parameters for external evaluations:

. Discussion
To improve the quality of educational activities, it is proposed to allocate contact points for the application of additional funding within each set of evaluation criteria: 1. "content and potential" complex (criteria 1, 2, and 7) • Bonuses for educational program project working groups.
• Forming of additional budget for the engagement of external specialists in designated areas.
. /fpubh. .  Program Complex CP (C , C , and C ) Complex A (C and C ) Complex P (C , C , C , and C ) • Constant refreshing of material assets, technical learning instruments, purchase of additional literature and/or online services subscriptions, etc.
2. "algorithms" complex (criteria 3 and 5) • Additional funding is aimed at building a pipeline of constant communication with external stakeholders to synchronize learning results and gained competencies with the required expertise in the field.
• Funding for building and facilitating internal systems of academic integrity in accordance with the national regulatory framework and best international practices.
3. "personalities" complex (criteria 4, 6, 8, and 9) • Allocating the budget aimed to form a bonus system for learning process stakeholders and an academic mobility system independent from international programs.
• Ensuring the "lifelong learning study" concept by funding constant professional growth of human resources associated with the learning process.
• Funding of internal quality assurance system for educational activities and synchronizing it with best practices in the field.
• Establishing open catalogs of educational offers, programs, and resources along with expenditures for creating and supporting media resources.
Funding of proposed contact points must be performed according to sources defined by the first component of Eq. (14). Each component in the numerator of the ratio ln F e ln F b can ensure funding distribution via separate directions requiring financial support as mentioned above.
We propose to overlook the pipeline of funding sources allocation for the university considering perspective marking of meeting criteria as exemplary for each external quality assurance of the educational program: 1. Identifying shortcomings for defined criteria. 2. Considering the possibility of increasing criterion completion level due to additional funding. 3. Selecting financial source(s) present in the university which can fund (in the framework of the current regulatory base) chosen activities aimed at criterion completion level increase.
. /fpubh. . 4. Funding of certain activities or development of infrastructure through chosen financing source. Figure 5 illustrates the definition of funding sources for university activity. In this approach, we propose to assume the perfect meeting of external quality assurance criteria considering criteria 1-9 for educational programs. Here, F1-F8 are funding sources for university activities: F1 and F2 are incomes from domestic and foreign students, F3 and F4 are incomes from scientific activities from domestic and foreign clients, F5 and F6 are incomes from grant-related activities from domestic and foreign donors, F7 is an income from additional educational offers, and F8 is an income from additional non-educational offers. Criteria grades and percentages of funding sources are presented on a pie diagram as a demonstration.
This approach not only diversifies expenditure sources but also can act as a catalyst for the diversification of university income sources through various types of educational, research, and production activities. Diversification of income sources becomes urgent since each source allows funding specific activities following the income conditions (estimates defining future expenses). Increasing revenue streams of different origins covers a broader range of tasks that need funding. For example, we propose the algorithm for selecting funding sources for the "Content and Potential" complex (criteria 1, 2, 7) as shown in Figure 6.
The proposed method of external quality assessment has advantages over analogs, which are given in the literature review: 1. the methodology takes into account independent indicators obtained by third parties. 2. the methodology can evaluate the indicators numerically. 3. the methodology allows for a retrospective analysis of progress (positive or negative) in ensuring the quality of the educational program.

. Conclusion
External accreditation and ratings are the most effective tools for quality assessment. However, ratings often neglect the field of study with rare exceptions when the rating has a direction selection option. In addition, some rankings require data input from the FIGURE Algorithm for selecting funding source for the "Content and Potential": is the necessary funding volume, and Fi is the university funding sources (number of sources corresponds to Figure ). university, which universities do not always want to provide and hence are dropped from the rankings. In this case, the rating cannot be considered a full and independent assessment of the university but can take part in the overall assessment tool construction. Therefore, the external assessment of education quality through accreditation expertise comes to the fore.
Mauboussin's success equation requires modification for educational program quality assessment to evaluate the luck level at the design stage and add a component of external assessment by stakeholders that can be done through indicators of educational ratings. The paradox of skill, in which, other things being equal, the one who took advantage of the coincidence is more successful, in this case, can be interpreted as an opportunity to minimize risks. Risks here are feedback from external stakeholders and competitive or tender principles for obtaining funding from external sources.
The proposed approach to the formation of the external quality evaluation algorithm of educational activities EXT covers both the indicators that form the component of the external evaluation by stakeholders and independent agencies and the reputation component R formed by the university authority. We considered boundary cases for all algorithm components and provided examples of success and innovation calculation for the 22 "Medicine" subject. The discussed analysis method of external evaluation results proposed in the study can be effectively scaled to obtain conclusions at three levels: program, university (within the benchmarking of general university components of the educational program), and national (within the benchmarking of similar programs).
The methodology presented in this study allows us to assess the educational program quality both qualitatively and quantitatively.
According to the proposed method, an important calculation result is the determination of external quality assessment criteria requiring additional funding. It is possible to establish the relationship between current financing from external sources (excluding basic funding, which is also obtained in the competition for indicators defined by the relevant formula) and the state of implementing certain external evaluation criteria and drawing conclusions about strengthening activity in specific areas.

Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Funding
This study was carried out under the projects Economic and mathematical modeling and forecasting, development of methodological and methodological bases for creating a roadmap for reforming the health care system in Ukraine, considering behavioral, social, economic and legal determinants (the basic funding from the Ministry of Education and Science, order from 16.04.2021, No. 434, contract /24-2021) and Changes in the approach to the creation of companies' distribution management concepts influenced by the effects of social and economic crises caused by the global pandemic and increased security risks (funding from the Scientific Grant Agency of the Ministry of Education, Science, Research and Sport of the Slovak Republic and Slovak Academy of Science, project VEGA 1/0392/23). This research was supported by the Program Cooperatio Health Sciences (Charles University in Prague).