A survey on multi‐output regression

In recent years, a plethora of approaches have been proposed to deal with the increasingly challenging task of multi‐output regression. This study provides a survey on state‐of‐the‐art multi‐output regression methods, that are categorized as problem transformation and algorithm adaptation methods. In addition, we present the mostly used performance evaluation measures, publicly available data sets for multi‐output regression real‐world problems, as well as open‐source software frameworks. WIREs Data Mining Knowl Discov 2015, 5:216–233. doi: 10.1002/widm.1157

[1]  Neil D. Lawrence,et al.  Kernels for Vector-Valued Functions: a Review , 2011, Found. Trends Mach. Learn..

[2]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[3]  Michelangelo Ceci,et al.  Semi-supervised Learning for Multi-target Regression , 2014, NFMCP.

[4]  Saso Dzeroski,et al.  Constraint Based Induction of Multi-objective Regression Trees , 2005, KDID.

[5]  S. Džeroski,et al.  Using single- and multi-target regression trees and ensembles to model a compound index of vegetation condition , 2009 .

[6]  J. Vanschoren,et al.  Scientific Workflow Management with ADAMS , 2012, ECML/PKDD.

[7]  Ying Liu,et al.  Real time prediction for converter gas tank levels based on multi-output least square support vector regressor , 2012 .

[8]  Grigorios Tsoumakas,et al.  MULAN: A Java Library for Multi-Label Learning , 2011, J. Mach. Learn. Res..

[9]  David H. Wolpert,et al.  Stacked generalization , 1992, Neural Networks.

[10]  S. Džeroski,et al.  Using multi-objective classification to model communities of soil microarthropods , 2006 .

[11]  Massimiliano Pontil,et al.  Regularized multi--task learning , 2004, KDD.

[12]  Yong Yu,et al.  Multi-output regression on the output manifold , 2009, Pattern Recognit..

[13]  G. De’ath MULTIVARIATE REGRESSION TREES: A NEW TECHNIQUE FOR MODELING SPECIES–ENVIRONMENT RELATIONSHIPS , 2002 .

[14]  Charles A. Micchelli,et al.  Learning Multiple Tasks with Kernel Methods , 2005, J. Mach. Learn. Res..

[15]  Saso Dzeroski,et al.  Predicting Chemical Parameters of River Water Quality from Bioindicator Data , 2000, Applied Intelligence.

[16]  Michelangelo Ceci,et al.  Top-down induction of model trees with regression and splitting nodes , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[18]  Geoff Holmes,et al.  Classifier chains for multi-label classification , 2009, Machine Learning.

[19]  Saso Dzeroski,et al.  Estimating vegetation height and canopy cover from remotely sensed data with machine learning , 2010, Ecol. Informatics.

[20]  Shai Ben-David,et al.  Exploiting Task Relatedness for Mulitple Task Learning , 2003, COLT.

[21]  Luis Alonso,et al.  Multioutput Support Vector Regression for Remote Sensing Biophysical Parameter Estimation , 2011, IEEE Geoscience and Remote Sensing Letters.

[22]  J. Chilès,et al.  Geostatistics: Modeling Spatial Uncertainty , 1999 .

[23]  Leo Breiman,et al.  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) , 2001, Statistical Science.

[24]  Mark Brudnak Vector-Valued Support Vector Regression , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[25]  J. Friedman Stochastic gradient boosting , 2002 .

[26]  Lin Li,et al.  Multi-output least-squares support vector regression machines , 2013, Pattern Recognit. Lett..

[27]  Leo Breiman,et al.  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) , 2001 .

[28]  Grigorios Tsoumakas,et al.  Multi-target Regression via Random Linear Target Combinations , 2014, ECML/PKDD.

[29]  J. Zidek,et al.  Multivariate regression analysis and canonical variates , 1980 .

[30]  Min-Ling Zhang,et al.  A Review on Multi-Label Learning Algorithms , 2014, IEEE Transactions on Knowledge and Data Engineering.

[31]  Alexander J. Smola,et al.  Support Vector Regression Machines , 1996, NIPS.

[32]  Timo Similä,et al.  Input selection and shrinkage in multiresponse linear regression , 2007, Comput. Stat. Data Anal..

[33]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[34]  Albert Bifet,et al.  Massive Online Analysis , 2009 .

[35]  Timo Aho,et al.  Rule Ensembles for Multi-target Regression , 2009, 2009 Ninth IEEE International Conference on Data Mining.

[36]  Saso Dzeroski,et al.  Incremental multi-target model trees for data streams , 2011, SAC.

[37]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[38]  Riccardo Poli,et al.  Particle swarm optimization , 1995, Swarm Intelligence.

[39]  Ivan Bratko,et al.  First Order Regression , 1997, Machine Learning.

[40]  Fernando Pérez-Cruz,et al.  SVM multiregression for nonlinear channel estimation in multiple-input multiple-output systems , 2004, IEEE Transactions on Signal Processing.

[41]  Wei-Yin Loh,et al.  Classification and regression trees , 2011, WIREs Data Mining Knowl. Discov..

[42]  Geoff Holmes,et al.  MOA: Massive Online Analysis , 2010, J. Mach. Learn. Res..

[43]  Saso Dzeroski,et al.  Stepwise Induction of Multi-target Model Trees , 2007, ECML.

[44]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[45]  Jon Y. Hardeberg,et al.  Multi- and Single-output Support Vector Regression for Spectral Reflectance Recovery , 2012, 2012 Eighth International Conference on Signal Image Technology and Internet Based Systems.

[46]  Pang-Ning Tan,et al.  Position Preserving Multi-Output Prediction , 2013, ECML/PKDD.

[47]  Michelangelo Ceci,et al.  Network regression with predictive clustering trees , 2011, Data Mining and Knowledge Discovery.

[48]  Ali Jalali,et al.  A Dirty Model for Multi-task Learning , 2010, NIPS.

[49]  Andre F. Marquand,et al.  Full Bayesian multi-task learning for multi-output brain decoding and accommodating missing data , 2014, 2014 International Workshop on Pattern Recognition in Neuroimaging.

[50]  A. Izenman Reduced-rank regression for the multivariate linear model , 1975 .

[51]  Concha Bielza,et al.  Multi-dimensional classification with Bayesian networks , 2011, Int. J. Approx. Reason..

[52]  Glenn De ' ath,et al.  MULTIVARIATE REGRESSION TREES: A NEW TECHNIQUE FOR MODELING SPECIES-ENVIRONMENT RELATIONSHIPS , 2002 .

[53]  Charles A. Micchelli,et al.  On Learning Vector-Valued Functions , 2005, Neural Computation.

[54]  Saso Dzeroski,et al.  An extensive experimental comparison of methods for multi-label learning , 2012, Pattern Recognit..

[55]  Trevor Hastie,et al.  Regularization Paths for Generalized Linear Models via Coordinate Descent. , 2010, Journal of statistical software.

[56]  Saso Dzeroski,et al.  Tree ensembles for predicting structured outputs , 2013, Pattern Recognit..

[57]  J. Friedman,et al.  Predicting Multivariate Responses in Multiple Linear Regression , 1997 .

[58]  Y. Haitovsky On multivariate ridge regression , 1987 .

[59]  Arthur E. Hoerl,et al.  Ridge Regression: Biased Estimation for Nonorthogonal Problems , 2000, Technometrics.

[60]  Grigorios Tsoumakas,et al.  Multi-Label Classification: An Overview , 2007, Int. J. Data Warehous. Min..

[61]  Vladimir Cherkassky,et al.  SVM+ regression and multi-task learning , 2009, 2009 International Joint Conference on Neural Networks.

[62]  Sunita Sarawagi,et al.  Discriminative Methods for Multi-labeled Classification , 2004, PAKDD.

[63]  Alison J. Burnham,et al.  LATENT VARIABLE MULTIVARIATE REGRESSION MODELING , 1999 .

[64]  Jonathan Baxter,et al.  A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling , 1997, Machine Learning.

[65]  Christopher M. Bishop,et al.  Classification and regression , 1997 .

[66]  J. Zidek,et al.  Adaptive Multivariate Ridge Regression , 1980 .

[67]  Donato Malerba,et al.  Leveraging the power of local spatial autocorrelation in geophysical interpolative clustering , 2014, Data Mining and Knowledge Discovery.

[68]  Shinichi Morishita,et al.  On Classification and Regression , 1998, Discovery Science.

[69]  Grigorios Tsoumakas,et al.  Multi-target regression via input space expansion: treating targets as inputs , 2012, Machine Learning.

[70]  Tapio Elomaa,et al.  Multi-target regression with rule ensembles , 2012, J. Mach. Learn. Res..

[71]  Wei Zhang,et al.  Multi-output LS-SVR machine in extended feature space , 2012, 2012 IEEE International Conference on Computational Intelligence for Measurement Systems and Applications (CIMSA) Proceedings.

[72]  Lorenzo Rosasco,et al.  Multi-output learning via spectral filtering , 2012, Machine Learning.