Computing a consensus journal meta-ranking using paired comparisons and adaptive lasso estimators

In a \publish-or-perish culture", the ranking of scientic journals plays a central role in assessing performance in the current research environment. With a wide range of existing methods and approaches to deriving journal rankings, meta-rankings have gained popularity as a means of aggregating dierent information sources. In this paper, we propose a method to create a consensus meta-ranking using heterogeneous journal rankings. Using a parametric model for paired comparison data we estimate quality scores for 58 journals in the OR/MS community, which together with a shrinkage procedure allows for the identication of clusters of journals with similar quality. The use of paired comparisons provides a exible framework for deriving a consensus score while eliminating the problem of data missingness.

[1]  John Mingers,et al.  Ranking journals in business and management: a statistical analysis of the Harzing data set , 2007, Eur. J. Inf. Syst..

[2]  John G. Kemeny,et al.  Mathematical models in the social sciences , 1964 .

[3]  Clyde W. Holsapple,et al.  Behavior-based analysis of knowledge dissemination channels in operations management , 2010 .

[4]  E. J. Emond,et al.  A new rank correlation coefficient with application to the consensus ranking problem , 2002 .

[5]  Kurt Hornik,et al.  Validation of Credit Rating Systems Using Multi-Rater Information , 2006 .

[6]  P. McCullagh.  Analysis of Ordinal Categorical Data , 1985 .

[7]  Kurt Hornik,et al.  How to derive consensus among various marketing journal rankings , 2014 .

[8]  B. Frey,et al.  Do Rankings Reflect Research Quality? , 2008, SSRN Electronic Journal.

[9]  Jack R. Meredith,et al.  Knowledge dissemination in operations management: Published perceptions versus academic reality , 2011 .

[10]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS , 1952 .

[11]  Cristiano Varin,et al.  The ranking lasso and its application to sport tournaments , 2012, 1301.2954.

[12]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS THE METHOD OF PAIRED COMPARISONS , 1952 .

[13]  Rudolf R. Sinkovics,et al.  Towards a Consolidation of Worldwide Journal Rankings — A Classification Using Random Forests and Aggregate Rating via Data Envelopment Analysis , 2014 .

[14]  Andrew Lim,et al.  Evaluating OR/MS Journals via PageRank , 2011, Interfaces.

[15]  B. Efron.  Better Bootstrap Confidence Intervals , 1987 .

[16]  S. Barman,et al.  Discipline note Perceived relevance and quality of POM journals: a decade later , 2001 .

[17]  Josephine E. Olson,et al.  Top-25-Business-School Professors Rate Journals in Operations Management and Related Fields , 2005, Interfaces.

[18]  N. Adler,et al.  When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings , 2009 .

[19]  R. J. Tersine,et al.  An empirical assessment of the perceived relevance and quality of POM-related journals by academicians , 1991 .

[20]  Jeremy B. Fox,et al.  A multi-method evaluation of journals in the decision and management sciences by US academics , 2000 .

[21]  Clyde W. Holsapple,et al.  An empirical assessment of influences on POM research , 1996 .

[22]  Michael Rowlinson,et al.  Journal Rankings in Business and Management and the 2001 Research Assessment Exercise in the UK , 2004 .

[23]  Hugh Willmott,et al.  Journal list fetishism and the perversion of scholarship: reactivity and the ABS list , 2011 .

[24]  Charles G. Petersen,et al.  Journal ranking analyses of operations management research , 2011 .

[25]  Timothy D. Fry,et al.  Outlets for operations management research: a DEA assessment of journal quality and rankings , 2013 .

[26]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[27]  Alexander Serenko,et al.  Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence , 2011, J. Informetrics.

[28]  Robert J. Vokurka,et al.  The relative importance of journals used in operations management research A citation analysis , 1996 .

[29]  R. A. Bradley,et al.  Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , 1952 .

[30]  Jian Ma,et al.  Quality Assessment : An Integrated Subjective and Objective Approach , 2001 .

[31]  W. Cook,et al.  Aggregating Incomplete Lists of Journal Rankings: An Application to Academic Accounting Journals* , 2010 .