Learning to rank by aggregating expert preferences

We present a general treatment of the problem of aggregating preferences from several experts into a consensus ranking, in the context where information about a target ranking is available. Specifically, we describe how such problems can be converted into a standard learning-to-rank one on which existing learning solutions can be invoked. This transformation allows us to optimize the aggregating function for any target IR metric, such as Normalized Discounted Cumulative Gain, or Expected Reciprocal Rank. When applied to crowdsourcing and meta-search benchmarks, our new algorithm improves on state-of-the-art preference aggregation methods.

[1]  Charles L. A. Clarke,et al.  Reciprocal rank fusion outperforms condorcet and individual rank learning methods , 2009, SIGIR.

[2]  Tao Qin,et al.  Supervised rank aggregation , 2007, WWW '07.

[3]  Thomas Hofmann,et al.  Learning to Rank with Nonsmooth Cost Functions , 2006, NIPS.

[4]  Yuan Yao,et al.  Statistical ranking and combinatorial Hodge theory , 2008, Math. Program..

[5]  Tao Qin,et al.  A New Probabilistic Model for Rank Aggregation , 2010, NIPS.

[6]  R. A. Bradley,et al.  Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , 1952 .

[7]  A. Elo The rating of chessplayers, past and present , 1978 .

[8]  Javed A. Aslam,et al.  Condorcet fusion for improved retrieval , 2002, CIKM '02.

[9]  David F. Gleich,et al.  Rank aggregation via nuclear norm minimization , 2011, KDD.

[10]  Gabriella Kazai,et al.  MSRC at TREC 2011 Crowdsourcing Track , 2011, TREC.

[11]  John Guiver,et al.  Bayesian inference for Plackett-Luce ranking models , 2009, ICML '09.

[12]  Jaana Kekäläinen,et al.  IR evaluation methods for retrieving highly relevant documents , 2000, SIGIR '00.

[13]  Javed A. Aslam,et al.  Models for metasearch , 2001, SIGIR '01.

[14]  C. L. Mallows NON-NULL RANKING MODELS. I , 1957 .

[15]  Jeff A. Bilmes,et al.  Consensus ranking under the exponential model , 2007, UAI.

[16]  Craig Boutilier,et al.  Learning Mallows Models with Pairwise Preferences , 2011, ICML.

[17]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[18]  Hsuan-Tien Lin,et al.  An Ensemble Ranking Solution for the Yahoo ! Learning to Rank Challenge , 2010 .

[19]  John D. Lafferty,et al.  Cranking: Combining Rankings Using Conditional Probability Models on Permutations , 2002, ICML.

[20]  Tao Qin,et al.  LETOR: Benchmark Dataset for Research on Learning to Rank for Information Retrieval , 2007 .

[21]  Christopher J. C. Burges,et al.  From RankNet to LambdaRank to LambdaMART: An Overview , 2010 .

[22]  R. Plackett The Analysis of Permutations , 1975 .

[23]  R. Luce,et al.  Individual Choice Behavior: A Theoretical Analysis. , 1960 .

[24]  Dan Roth,et al.  Unsupervised rank aggregation with distance-based models , 2008, ICML '08.

[25]  L. Thurstone The method of paired comparisons for social values , 1927 .

[26]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS , 1952 .

[27]  Alexander Dekhtyar,et al.  Information Retrieval , 2018, Lecture Notes in Computer Science.

[28]  Tom Minka,et al.  TrueSkill Through Time: Revisiting the History of Chess , 2007, NIPS.

[29]  H. A. David,et al.  The method of paired comparisons , 1966 .

[30]  R. Duncan Luce,et al.  Individual Choice Behavior: A Theoretical Analysis , 1979 .