Efficient Ranking from Pairwise Comparisons

The ranking of n objects based on pairwise comparisons is a core machine learning problem, arising in recommender systems, ad placement, player ranking, biological applications and others. In many practical situations the true pairwise comparisons cannot be actively measured, but a subset of all n(n-1)/2 comparisons is passively and noisily observed. Optimization algorithms (e.g., the SVM) could be used to predict a ranking with fixed expected Kendall tau distance, while achieving an Ω (n) lower bound on the corresponding sample complexity. However, due to their centralized structure they are difficult to extend to online or distributed settings. In this paper we show that much simpler algorithms can match the same Ω (n) lower bound in expectation. Furthermore, if an average of O(n log(n)) binary comparisons are measured, then one algorithm recovers the true ranking in a uniform sense, while the other predicts the ranking more accurately near the top than the bottom. We discuss extensions to online and distributed ranking, with benefits over traditional alternatives.

[1]  R. Graham,et al.  Spearman's Footrule as a Measure of Disarray , 1977 .

[2]  Eli Upfal,et al.  Computing with Noisy Information , 1994, SIAM J. Comput..

[3]  Yoram Singer,et al.  An Efficient Boosting Algorithm for Combining Preferences by , 2013 .

[4]  Thore Graepel,et al.  Large Margin Rank Boundaries for Ordinal Regression , 2000 .

[5]  Gert Cauwenberghs,et al.  Incremental and Decremental Support Vector Machine Learning , 2000, NIPS.

[6]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[7]  Gábor Lugosi,et al.  Introduction to Statistical Learning Theory , 2004, Advanced Lectures on Machine Learning.

[8]  Yoram Singer,et al.  Log-Linear Models for Label Ranking , 2003, NIPS.

[9]  Igor Durdanovic,et al.  Parallel Support Vector Machines: The Cascade SVM , 2004, NIPS.

[10]  Jason E. Fulman,et al.  Stein's method, Jack measure, and the Metropolis algorithm , 2003, Journal of combinatorial theory. Series A.

[11]  D. Roth,et al.  A study of the bipartite ranking problem in machine learning , 2005 .

[12]  Joachim Giesen,et al.  Approximate Sorting , 2006, LATIN.

[13]  Thorsten Joachims,et al.  Training linear SVMs in linear time , 2006, KDD '06.

[14]  Atri Rudra,et al.  Ordering by weighted number of wins gives a good ranking for weighted tournaments , 2006, SODA '06.

[15]  Mark Braverman,et al.  Noisy sorting without resampling , 2007, SODA '08.

[16]  Tamir Hazan,et al.  A Parallel Decomposition Solver for SVM: Distributed dual ascend using Fenchel Duality , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[17]  Mark Braverman,et al.  Sorting from Noisy Information , 2009, ArXiv.

[18]  Joachim Giesen,et al.  Approximate Sorting , 2006, Fundam. Informaticae.

[19]  Cynthia Rudin,et al.  The P-Norm Push: A Simple Convex Ranking Algorithm that Concentrates at the Top of the List , 2009, J. Mach. Learn. Res..

[20]  Ioannis Mitliagkas,et al.  User rankings from comparisons: Learning permutations in high dimensions , 2011, 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[21]  Nir Ailon,et al.  A New Active Learning Scheme with Applications to Learning to Rank from Pairwise Preferences , 2011, ArXiv.

[22]  Shivani Agarwal,et al.  The Infinite Push: A New Support Vector Ranking Algorithm that Directly Optimizes Accuracy at the Absolute Top of the List , 2011, SDM.

[23]  Nir Ailon,et al.  Ranking from pairs and triplets: information quality, evaluation methods and query complexity , 2011, WSDM '11.

[24]  David F. Gleich,et al.  Rank aggregation via nuclear norm minimization , 2011, KDD.

[25]  Robert D. Nowak,et al.  Active Ranking using Pairwise Comparisons , 2011, NIPS.

[26]  Devavrat Shah,et al.  Ranking: Compare, don't score , 2011, 2011 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[27]  Nir Ailon,et al.  An Active Learning Algorithm for Ranking from Pairwise Preferences with an Almost Optimal Query Complexity , 2010, J. Mach. Learn. Res..

[28]  Devavrat Shah,et al.  Iterative ranking from pair-wise comparisons , 2012, NIPS.