Are Scientists Nearsighted Gamblers? The Misleading Nature of Impact Factors

Despite a “Cambrian” explosion in the number of citation metrics used (Van Noorden, 2010), the impact factor (IF) of a journal remains a decisive factor of choice when publishing your ultimate research results and evaluating research productivity. Most other citation metrics correlate with the IF and there is little doubt that they reflect the overall impact of different journals. However, there is good reason to be more cautious about IF judgments.

First, the distribution of the number of citations per paper (NCPP) within a journal is heavily skewed. A few highly cited papers often account for a significant amount of the total citation count of a journal (25% of the papers in Nature account for 89% of the IF “Not-so-deep impact,” 2005) and a recent report highlighted that even a single article can dramatically bias the IF of a small journal (Dimitrov et al., 2010). The mean NCPP, as captured with the IF, should therefore never be used. A more appropriate measure is the median NCPP. Figure 1 (left) plots the median of the total NCPP against the mean, for three potential publication outlets for psychologists; Psychological Review (IF2009 = 9.1), Nature (IF2009 = 34.5), and Psychological Science (IF2009 = 5.1), for different years (data compiled from ISI Web of Knowledge). Nature follows a distinct trend when compared to specialist journals: the median seems independent from its mean. This apparent dissociation results from the skew of the citation distribution observed in Nature, in which up to 35–40% of the published articles are never cited. Moreover, when using a robust metric for skewed distributions, Psychological Science's median is about seven times higher than Nature's even though its IF is about seven times lower than Nature's. This discrepancy even holds for specialized journals with very low impact factors; despite possessing an IF nearly 35 times lower than Nature's, the Journal of Child Language's median is higher than Nature's (data not shown).

Figure 1

A second interesting property of the IF is that it focuses on citations of recently published articles only. For example, the 2009 IF of a journal considers the number of citations in 2009 to articles published in 2007 and 2008 only. However, the citations’ time course differs dramatically from one journal to another. Figure 1 (right) reports the mean NCPP per year, as a function of the number of years since publication. Once again, Nature possesses a distinct trend in its citation profile – the number of citations peaks 2 years after publication – whereas specialist journals have a steady increase in the NCPP. An article in Psychological Review will see its influence grow with age and would be outperformed by Nature when the IF monitors only short-lived citation patterns.

From this perspective, the IF, commonly accepted as golden standard for performance metrics seems to reward high-risk strategies (after all your Nature article has only slightly over 50% chance of being ever cited! ), and short-lived outbursts. Are scientists then nearsighted gamblers?