The Slavery of the h-index—Measuring the Unmeasurable

Last year we “celebrated” the 10th anniversary of the invention of the h-index (also known as the Hirsch factor; Hirsch, 2005), an indicator created by Jorge E. Hirsch, that attempts to measure the achievements of a research scientist. However, it not only appears that h-index has taken on a life of its own but also that the popularity of this formula currently surpasses the initial idea for its use envisioned by the inventor. Originally introduced as a simple characterization of the scientific output of a researcher (Hirsch, 2005), the h-index has come to be uncritically regarded as a “magic tool” that is applied to measure what is unmeasurable—the quality of science. As a result, it has become a “must have” indicator when applying for funds or a new position. Surprisingly, many decision-makers apparently are not fully aware of what it represents. According to its inventor, the h-index is the number of papers coauthored by the investigator with at least h citations each (Hirsch, 2005). That is, to be the proud owner of a high h-index, it is not sufficient to have authored many articles or for some of them to have been extensively cited. Rather, such an achievement must satisfy both issues: a considerable number of articles must be highly cited, which is reflected by ranking them according to the number of citations they have collected, and finding the one where the position on the list equals or is less than the number of the citations it garnered. 
 
The pros and cons of the h-index 
The initial idea of Hirsch was to discriminate the investigators who are persistently productive from those who experienced an isolated auspicious moment in their scientific life, and who currently only cut coupons from their popularity roll. Nevertheless, it assumes that researcher A, who published a breakthrough story that was extensively cited, should deserve less respect than researcher B who publishes often and regularly; however, the outcome of the latter's work has not contributed yet to any remarkable discovery. A good example is the inventor of the RNA isolation method, Piotr Chomczynski. He has in total over 65 000 citations to which he contributed almost exclusively (92.9% of all citations) with one single paper regarding the method he introduced in 1987 (Chomczynski and Sacchi, 1987). His current h-index is 23, relatively low for such a prodigious number of citations. Yet, could we imagine working with RNA over the past number of years without possessing this simple technique that is now the principle of virtually every commercial protocol related to RNA extraction? Notwithstanding, it might be argued—depreciating that discovery—that because this method is so simple, someone else would have been discovered it sooner or later. In response, let me cite my former mentor, professor Gunther Schutz, who would disprove similar arguments with a simple statement: If you say this is so trivial, why was it not you who discovered PCR? Breakthrough discoveries are not solely dependent on sophisticated science. 
 
Another problem with the h-index is the impossibility of comparing the investigators during different stages of their careers (even assuming comparisons among those representing the same field, which is another ambiguous factor). There is a certain correlation between the age of an investigator and h-index. Clearly and in any case, some of articles will accumulate citations and this number will increase over the time since they were first published. However, even the comparison between investigators at a similar career stage may often be misleading, particularly among young post-docs whose careers have just begun. We must be honest and acknowledge the fact that it is a rare occurrence that right after completing a PhD, such an investigator is able to independently decide about own career development. Mostly, the scientific achievements at this stage are primarily a derivative of the power of the PhD mentor and the reputation of the hosting institution. 
 
Another issue contributing to h-index limitations is that many research groups have different regulations regarding authorship. It is assumed that a researcher's name will be added to the authorship list only after considerable contribution has been made to the published work. However, what occurs fairly often is that being a “middle man” on the listing does not necessary reflect the significant contribution and, worth to be emphasized, the h-index does not differentiate between article authors who hold the most valuable first and last authorship position and those wherein the author's name appears as one among perhaps even 100 authors, as occurs with articles containing vast meta-analyses of clinical data. 
 
Although, some of these concerns were raised in the discussion part of the initial paper published by Hirsch (2005), they were overshadowed by the enormous popularity of this tool and its indiscriminate application. Despite of many critical yet unofficial discussions about the h-index, its limitations, and perhaps dangerous influence on science, the topic—if only tackled by publications—is usually narrowed to the pursuit of more and more sophisticated bibliometrics and proposals of new indicators (Bharathi, 2013; Biswal, 2013; Diaz et al., 2016; Wurtz and Schmidt, 2016). The critical voices seem to be less represented, nevertheless there are of course existing articles pointing out that past achievements of the scientists may not necessarily be correlated with future success and that all such rankings need context which means that the best method to gain an impression of the quality is still simply reading the papers (Wendl, 2007; von Bohlen Und Halbach, 2011). Perhaps the combination of looking into the context of the particular paper and the journal reputation gives the best assumption in this matter, however even this approach can be biased toward personal preferences and requires considerable amount of time. 
 
Last but not least, when examining the reasons why the h-index is not trustworthy, we should not neglect issues regarding fraud. Because the h-index does not discriminate self-citations, it is not difficult to predict that even the investigators who are poorly cited by others but who publish prodigiously, citing mostly themselves, will easily increase their h-index in the long run. Moreover, some evidence of misuse has been reported involving artificially pumped h-indices based on i.e., regular cross-citations between good friends (Kotov, 2010).


INTRODUCTION
Last year we "celebrated" the 10th anniversary of the invention of the h-index (also known as the Hirsch factor; Hirsch, 2005), an indicator created by Jorge E. Hirsch, that attempts to measure the achievements of a research scientist. However, it not only appears that h-index has taken on a life of its own but also that the popularity of this formula currently surpasses the initial idea for its use envisioned by the inventor. Originally introduced as a simple characterization of the scientific output of a researcher (Hirsch, 2005), the h-index has come to be uncritically regarded as a "magic tool" that is applied to measure what is unmeasurable-the quality of science. As a result, it has become a "must have" indicator when applying for funds or a new position. Surprisingly, many decision-makers apparently are not fully aware of what it represents. According to its inventor, the h-index is the number of papers coauthored by the investigator with at least h citations each (Hirsch, 2005). That is, to be the proud owner of a high h-index, it is not sufficient to have authored many articles or for some of them to have been extensively cited. Rather, such an achievement must satisfy both issues: a considerable number of articles must be highly cited, which is reflected by ranking them according to the number of citations they have collected, and finding the one where the position on the list equals or is less than the number of the citations it garnered.

The Pros and Cons of the h-index
The initial idea of Hirsch was to discriminate the investigators who are persistently productive from those who experienced an isolated auspicious moment in their scientific life, and who currently only cut coupons from their popularity roll. Nevertheless, it assumes that researcher A, who published a breakthrough story that was extensively cited, should deserve less respect than researcher B who publishes often and regularly; however, the outcome of the latter's work has not contributed yet to any remarkable discovery. A good example is the inventor of the RNA isolation method, Piotr Chomczynski. He has in total over 65 000 citations to which he contributed almost exclusively (92.9% of all citations) with one single paper regarding the method he introduced in 1987 ( Chomczynski and Sacchi, 1987). His current h-index is 23, relatively low for such a prodigious number of citations. Yet, could we imagine working with RNA over the past number of years without possessing this simple technique that is now the principle of virtually every commercial protocol related to RNA extraction? Notwithstanding, it might be argued-depreciating that discovery-that because this method is so simple, someone else would have been discovered it sooner or later. In response, let me cite my former mentor, professor Günther Schütz, who would disprove similar arguments with a simple statement: If you say this is so trivial, why was it not you who discovered PCR? Breakthrough discoveries are not solely dependent on sophisticated science.
Another problem with the h-index is the impossibility of comparing the investigators during different stages of their careers (even assuming comparisons among those representing the same field, which is another ambiguous factor). There is a certain correlation between the age of an investigator and h-index. Clearly and in any case, some of articles will accumulate citations and this number will increase over the time since they were first published. However, even the comparison between investigators at a similar career stage may often be misleading, particularly among young post-docs whose careers have just begun. We must be honest and acknowledge the fact that it is a rare occurrence that right after completing a PhD, such an investigator is able to independently decide about own career development. Mostly, the scientific achievements at this stage are primarily a derivative of the power of the PhD mentor and the reputation of the hosting institution.
Another issue contributing to h-index limitations is that many research groups have different regulations regarding authorship. It is assumed that a researcher's name will be added to the authorship list only after considerable contribution has been made to the published work. However, what occurs fairly often is that being a "middle man" on the listing does not necessary reflect the significant contribution and, worth to be emphasized, the h-index does not differentiate between article authors who hold the most valuable first and last authorship position and those wherein the author's name appears as one among perhaps even 100 authors, as occurs with articles containing vast meta-analyses of clinical data.
Although, some of these concerns were raised in the discussion part of the initial paper published by Hirsch (2005), they were overshadowed by the enormous popularity of this tool and its indiscriminate application. Despite of many critical yet unofficial discussions about the h-index, its limitations, and perhaps dangerous influence on science, the topic-if only tackled by publications-is usually narrowed to the pursuit of more and more sophisticated bibliometrics and proposals of new indicators (Bharathi, 2013;Biswal, 2013;Díaz et al., 2016;Würtz and Schmidt, 2016). The critical voices seem to be less represented, nevertheless there are of course existing articles pointing out that past achievements of the scientists may not necessarily be correlated with future success and that all such rankings need context which means that the best method to gain an impression of the quality is still simply reading the papers (Wendl, 2007;von Bohlen Und Halbach, 2011). Perhaps the combination of looking into the context of the particular paper and the journal reputation gives the best assumption in this matter, however even this approach can be biased toward personal preferences and requires considerable amount of time.
Last but not least, when examining the reasons why the h-index is not trustworthy, we should not neglect issues regarding fraud. Because the h-index does not discriminate self-citations, it is not difficult to predict that even the investigators who are poorly cited by others but who publish prodigiously, citing mostly themselves, will easily increase their h-index in the long run. Moreover, some evidence of misuse has been reported involving artificially pumped h-indices based on i.e., regular cross-citations between good friends (Kotov, 2010).

The h-indices of Nobel Prize Winners
Great discoveries require time for experiments. Furthermore, quality investigational pursuits require time for planning, critical thought, discussion, repetition. This process cannot be conducted in haste and under pressure to simply publish all that has been collected. Isaak Newton was said to think twice and to examine all of his discoveries time after time until he finally decided to publish them. Currently, with such an approach, he would most likely have been released nowadays from the university due to the lack of progress in his research. Seemingly, this example perhaps would be difficult to prove. However, consider instead scenarios that are closer to reality-the Nobel Prize laureates in Physiology & Medicine, an unbiased selection of preeminent scientists in life sciences. Do they actually possess h-indices above 100 or thereabouts? Indeed and not surprisingly, many of them do. However, there are also laureates who would have inevitably encountered roadblocks had their h-index been the decisive factor in their nomination. If we consider the last 25 laureates from the period of the recent 10 years, the outcomes are somewhat surprising. The h-index values vary among these esteemed professors within the range of 24-139 ( Figure 1A), although no one would rightfully propose that one of these laureates should be regarded as a 5-fold better scientist than another. Clearly, this may again have reflected the time when the Nobel-worthy discovery was made and simply the age of the laureate, another problematic issue raised in one of recent commentaries in Nature journal (Fortunato, 2014). Nonetheless, even when comparing the laureates by their h-indices at the same age of 35 years, the average age of a young post-doc (Figure 1B), or the year before the laureates' famous discoveries were first announced in top-flight journals (Figure 1C), no correlation exists and moreover, examples may be found of investigators who had virtually no bibliometric track records. This is also true among relatively young Nobel laureates (those born >1960) whose careers were certainly influenced by the IT era from the very beginning (Figures 1A-C, green bars). It could be surmised that in many cases the moderate h-indices at this stage are simply correlated to the persistence in pursuing the hypothesis that ultimately was proven, thereby yielding a scientific breakthrough.

A Lesson from Poland
We have had a unique experience in Poland, arising from the Communist period of our history, when all work efforts were required to be quantified by numbers, that were the most important, even though the outcomes of that work were far from expected. That was a time during which, e.g., a car factory would proudly announce that it had produced the 1-millionth vehicle, although it completely disregarded that it was barely possible to start the engine in most of the cars or to exit a parking spot because of a variety of mechanical failures.

FIGURE 1 | Continued
Barry J. Marshall, BJM (Marshall and Warren, 1984); J. Robin Warren, JRW (Marshall and Warren, 1984 Poland is now enjoying a good period in its history. As a result of maintaining the gross domestic product in the black, despite worldwide economic crises, the monies spent on research are continuously increasing. Moreover, several attempts and programs are underway that aim to reorganize science in Poland to be more competitive and efficient. However, all these efforts may backfire if we remain stuck to the easy and old-fashioned way of evaluating our work. Unquestionably, we scientists must somehow be evaluated as our job is based on spending public monies and such work should be accomplished in an efficient manner. However, the critical evaluation of science requires a great deal of effort and cannot be done using a simple comparison of numbers. As stated by the Hirsch himself, the single number can give only the rough approximation to an individual's researcher profile with many other factors contributing as well (Hirsch, 2005). The overall assessment is also affected by an average number of citations per paper, the quality of journals where the work was published, and goes far beyond the publication track records including other scientific achievements i.e., invited lectures, international experience (post-doctoral fellowships, sabbaticals, collaborations), project managements.
Another issue is the system of evaluation built upon assumption that future success of the project depends on the previous achievements of the applicant. Thus, it remains tempting for some of the grant reviewers to assess the applicants solely through the prism of their respective h-indices, and such a relentless pursuit of bibliometric factors may bring global science to its knees, leaving no room for independent research that may lead to the elucidation of fundamental biological quandaries.

AUTHOR NOTE
The author is a research associate at the Department of Brain Biochemistry, Institute of Pharmacology, Polish Academy of Sciences (IF PAS), Krakow, Poland. A former post-doc (2005)(2006)(2007)(2008) at the German Cancer Research Center (DKFZ), Heidelberg, Germany. The PI on 2 research grants funded by the Polish National Science Center (NCN).

AUTHOR CONTRIBUTIONS
GK designed, calculated data, and wrote the paper. No other person was involved.

ACKNOWLEDGMENTS
The content of this article is solely the responsibility of the author and does not necessarily represent the official view of the affiliated organization.