Multidimensional Learning from Crowds: Usefulness and Application of Expertise Detection

Learning from crowds is a classification problem where the provided training instances are labeled by multiple (usually conflicting) annotators. In different scenarios of this problem, straightforward strategies show an astonishing performance. In this paper, we characterize the crowd scenarios where these basic strategies show a good behavior. As a consequence, this study allows to identify those scenarios where non‐basic methods for combining the multiple labels are expected to obtain better results. In this context, we extend the learning from crowds paradigm to the multidimensional (MD) classification domain. Measuring the quality of the annotators, the presented EM‐based method overcomes the lack of a fully reliable labeling for learning MD Bayesian network classifiers: As the expertise is identified and the contribution of the relevant annotators promoted, the model parameters are optimized. The good performance of our proposal is demonstrated throughout different sets of experiments.

[1]  Zhi-Hua Zhou,et al.  Multi-Label Learning with Weak Label , 2010, AAAI.

[2]  Iñaki Inza,et al.  Learning from Crowds in Multi-dimensional Classification Domains , 2013, CAEPIA.

[3]  Janyce Wiebe,et al.  Development and Use of a Gold-Standard Data Set for Subjectivity Classifications , 1999, ACL.

[4]  Pietro Perona,et al.  The Multidimensional Wisdom of Crowds , 2010, NIPS.

[5]  S. Sathiya Keerthi,et al.  Regularized Structured Output Learning with Partial Labels , 2012, SDM.

[6]  Thierry Denoeux,et al.  Evidential Multi-Label Classification Approach to Learning from Data with Imprecise Labels , 2010, IPMU.

[7]  Brendan T. O'Connor,et al.  Cheap and Fast – But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks , 2008, EMNLP.

[8]  Pietro Perona,et al.  Inferring Ground Truth from Subjective Labelling of Venus Images , 1994, NIPS.

[9]  Xindong Wu,et al.  Eliminating Class Noise in Large Datasets , 2003, ICML.

[10]  Arthur M. Breipohl,et al.  An Error Correcting Procedure for Learning with an Imperfect Teacher , 1971, IEEE Trans. Syst. Man Cybern..

[11]  David Heckerman,et al.  A Tutorial on Learning with Bayesian Networks , 1998, Learning in Graphical Models.

[12]  Zoran Obradovic,et al.  Learning by aggregating experts and filtering novices: a solution to crowdsourcing problems in bioinformatics , 2013, BMC Bioinformatics.

[13]  Janez Demsar,et al.  Statistical Comparisons of Classifiers over Multiple Data Sets , 2006, J. Mach. Learn. Res..

[14]  Javier R. Movellan,et al.  Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise , 2009, NIPS.

[15]  Min-Ling Zhang,et al.  A Review on Multi-Label Learning Algorithms , 2014, IEEE Transactions on Knowledge and Data Engineering.

[16]  S. García,et al.  An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for all Pairwise Comparisons , 2008 .

[17]  Concha Bielza,et al.  Bayesian network modeling of the consensus between experts: An application to neuron classification , 2014 .

[18]  Rong Jin,et al.  Learning with Multiple Labels , 2002, NIPS.

[19]  A. P. Dawid,et al.  Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm , 1979 .

[20]  Gerardo Hermosillo,et al.  Learning From Crowds , 2010, J. Mach. Learn. Res..

[21]  Friedhelm Schwenker,et al.  Partially supervised learning for pattern recognition , 2014, Pattern Recognit. Lett..

[22]  Gábor Lugosi,et al.  Learning with an unreliable teacher , 1992, Pattern Recognit..

[23]  José Antonio Lozano,et al.  Using Multidimensional Bayesian Network Classifiers to Assist the Treatment of Multiple Sclerosis , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[24]  Ben Taskar,et al.  Learning from Partial Labels , 2011, J. Mach. Learn. Res..

[25]  Ohad Shamir,et al.  Good learners for evil teachers , 2009, ICML '09.

[26]  Rong Jin,et al.  Multi-label learning with incomplete class assignments , 2011, CVPR 2011.

[27]  Concha Bielza,et al.  Multi-dimensional classification with Bayesian networks , 2011, Int. J. Approx. Reason..

[28]  Panagiotis G. Ipeirotis,et al.  Get another label? improving data quality and data mining using multiple, noisy labelers , 2008, KDD.

[29]  Dana M. Caudle,et al.  To tag or not to tag? , 2009, Libr. Hi Tech.

[30]  Avrim Blum,et al.  Veritas: Combining Expert Opinions without Labeled Data , 2008, 2008 20th IEEE International Conference on Tools with Artificial Intelligence.

[31]  Nir Friedman,et al.  Learning Belief Networks in the Presence of Missing Values and Hidden Variables , 1997, ICML.

[32]  Milos Hauskrecht,et al.  Learning classification models from multiple experts , 2013, J. Biomed. Informatics.

[33]  Ohad Shamir,et al.  Vox Populi: Collecting High-Quality Labels from a Crowd , 2009, COLT.

[34]  Carla E. Brodley,et al.  Identifying Mislabeled Training Data , 1999, J. Artif. Intell. Res..

[35]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .