New Metrics for Cross-Country Comparison of Scientific Impact

1 Programa de Engenharia Biomédica, Coordenação dos Programas de Pós-Graduação em Engenharia (COPPE), Universidade Federal do Rio de Janeiro, Rio de Janeiro, Brazil, 2 Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, Ministério da Educação, Brasilia, Brazil, 3 Faculdade de Medicina, Universidade de Brasília, Brasilia, Brazil, 4 Departamento de Biologia Celular, Universidade de Brasília, Brasilia, Brazil


INTRODUCTION
Comparing scientific impact among countries is fundamental for the development of effective strategies for science financing and assessment (Vinkler, 1986;King, 2004;Hermes-Lima et al., 2007a;Wagner et al., 2018). Such comparisons allow the definition of funding strategies based on inputs and outputs of the Research and Development (R&D) system and the identification of countries and institutions with the best performance (Freeman, 1977;Leydesdorff et al., 2019). These top performer countries, in turn, can be emulated by less-developed nations.
Usually, the number of scientific papers and the number of citations of a country are the main variables used for these comparisons. These metrics are frequently combined in a single indicator, such as the Citations Per Publication (CPP) (Lehmann et al., 2003;Moed, 2005;Waltman and van Eck, 2009;Vinkler, 2010) or the well-known h-index (Hirsch, 2005;Egghe, 2006;Glänzel, 2006;Hermes-Lima et al., 2007b;Franceschini and Maisano, 2010). However, each index has its caveats, and identifying simple, meaningful metrics for scientific productivity evaluation at country level has been a challenge (King, 2004;Leydesdorff et al., 2019). The limitations of the h-index have been vastly discussed in the literature (Bornmann and Daniel, 2005;Waltman and van Eck, 2012); and inter-country comparisons using the CPP have problems related to the time frame of an analysis. Thus, early publications from a given country will have more citations (therefore resulting in a higher CPP) than later publications, simply because papers published earlier had more time to be cited.
For instance, Belgium was the country with the highest research impact in Medicine in 2019 (Scimago data), with a CPP value of 1.38 (12,768 publications; 17,595 citations), whereas its CPP value for 2016 is 17.73 (11,766 publications; 208,622 citations) and for 2013, 30.91 (22 times higher than in 2019). This limitation leads to the question: How can one avoid this caveat and meaningfully compare research impact among countries in a year-per-year basis?
In this essay we present two simple methods to standardize CPP values along time, thus allowing for comparisons among countries in a time-independent fashion. Therefore, two metrics are presented: The Relative CPP and the Impact-Relative Rank Score (IRRS). Both of these procedures are based on the principle of normalizing the metric according to maximumminimum values in the common set of observational units (countries) used for comparison.
this maximum value, one can then calculate the percentage of the CPPs for the remaining countries: The Relative CPPs in the data set, for each country. The second procedure, the IRRS, takes the additional step of ranking countries according to their Relative CPPs, after what their rank order is used to calculate a score in a 0 to 10 scale (Equation 1). In this scale, the value "zero" is assigned to the last-ranked country and "10" to the first placed one (the "benchmark country").

EXAMPLES
In general, the inclusion of countries with a modest number of publications per year (such as small islands) leads to atypical CPP values, distorting a metric and thus reducing its efficacy. In order to minimize this effect, a cut-off value is commonly used for country selection, preferably resulting in the inclusion of more than 95% of all publications in a base year. Therefore, below, the approach is illustrated for the field of Medicine, with a cut-off of 2,000 papers/ year (for the 2019 dataset, this threshold resulted in 52 countries representing 95.7% of the 1.21 million available publications). At first, the ranking of the 52 selected countries according to their CPPs in Medicine was obtained from the Scimago platform (Scimago Lab, 2019). In this rank, Belgium was the benchmark country for 2019, with CPP 1.38; and the Netherlands was in the second position (CPP 1.28). Belgium was also the benchmark As mentioned, taking into account the CPP value of the benchmark country, it is possible, then, to calculate the impact of other countries relatively to it in each year. For example, medical United Kingdom papers had CPP 1.07 in 2019 and CPP 4.71 in 2018. Therefore, according to the Relative CPP procedure described above, the relative impact of the United Kingdom in those years was 77.5 and 79.3% respectively -the United Kingdom Relative CPPs. Figure 1A, presents the 2011-2019 Relative CPPs for the Medical area in eight countries, selected as to give a

DISCUSSION
This article described two new metrics for cross-country evaluation of scientific productivity that, while still using the widely available "number of papers -number of citations" data, improve over the much used CPP and h-index metrics. The metrics presented here are simple, objective and consistent along time, and their use would allow for meaningful and informative comparisons of the performance of countries. As an example, a remarkable improvement in Chilean relative performance could be detected here, and, on the other hand, the decrease in the United States and Russian performances could also be seen in a visually simple and direct way.
In the field of Scientometrics, measurement of impact in specific subject areas is influenced by frequency of publication, length of reference lists, and number of co-authors (Elsevier, 2018). Medicine, for instance, has a high frequency of publications in comparison with other 26 subject areas, according to the Scopus All Science Journal Classification. Moreover, nation publication patterns are highly heterogeneous both in their numbers, area, and impact, and, therefore, since the number of scientific publications has markedly increased in the last decades, comparisons with the world impact average (although common) may not represent the real change in a country's scientific output. In addition, the world impact average also changes substantially within fields, relatively to the benchmark country. For example, for 2018, its value is 69% in Medicine 54% in Chemistry, 51% in Engineering, and 41% in Dentistry (Table 1).
Although essentially reporting the same information, the Relative CPP and the IRRS allow for different views of country performance in a dataset. Thus, IRRS indicates how countries "move" in a rank-list while the Relative CPP shows how much is needed for a country to get to "first place"that is, how policy decisions are impacting their science sectors. Therefore, the ranking procedure (IRRS) is more sensitive to small changes, while the Relative CPP allows one to more globally visualize the extent of the difference between a country and the top ranker in the field.
The main limitation of the proposed indicators is the mentioned common problem among cross-country impact performance metrics: Relative performance evaluation can change contingent on a specific set of countries analyzed, and the inclusion of smallimpact countries can greatly distort an analysis. In summary, we presented here two metrics for cross-country comparison of scientific performance: The Relative CPP and the IRRS, and we hope that these will allow for a more informative discussion of government policies and their impact over the world scientific output.

AUTHOR CONTRIBUTIONS
RA, LB, DM, and MH contributed to the study design, data analysis, and manuscript drafting.

FUNDING
Part of this research was funded by a Ministry of Education/ CAPES grant (Code: 001), to whom we thank.