On the stability of citation-based journal rankings

Abstract

© 2015 Elsevier Ltd. The article presents a large-scale comparison of journal rankings based on seven impact measures: Impact Factor (2- and 5-year), SJR, IPP, SNIP, H index, and Article Influence Score. Three aspects of ranking stability in the 2007-2014 period were analyzed: temporal, cross-discipline, and cross-indicator. Impact measures based on five-year citation windows enable more stable journal rankings over time. Journal rankings based on the source-normalized indicator (SNIP) have the largest cross-discipline stability. Journals in the fields of social sciences and humanities have lower temporal and cross-discipline ranking stability compared to those in "hard" sciences. Although correlation coefficients indicate relatively high agreement among the rankings based on different indicators, variations in quartile and percentile ranks suggest different conclusions. WoS journals almost linearly improve their ranking positions in Scopus lists, while many high-impact journals covered by Scopus are not available in WoS. An important element of the ranking stability is the discriminability of impact measures. Beyond the segregation between the top and bottom ranked journals, our assessment of "quality" relies in most cases on a rather arguable assumption that a couple of citations more or less is making a big difference.