Multi-objective Ensemble Construction , Learning and Evolution

An ensemble of learning machines has been theoretically and empirically shown to generalise better than single learners. Diversity and accuracy are two key properties that ensemble members should possess in order for this generalisation principle to hold. Viewing these properties as objectives, we take the position of rendering multi-objective evolutionary algorithms as effective solution concepts to the problem of ensemble construction and learning.

[1]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[2]  Bernhard Sendhoff,et al.  Neural network regularization and ensembling using multi-objective evolutionary algorithms , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[3]  Bev Littlewood,et al.  Conceptual Modeling of Coincident Failures in Multiversion Software , 1989, IEEE Trans. Software Eng..

[4]  Kagan Tumer,et al.  Analysis of decision boundaries in linearly combined neural classifiers , 1996, Pattern Recognit..

[5]  Jeffrey Horn,et al.  Multiobjective Optimization Using the Niched Pareto Genetic Algorithm , 1993 .

[6]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[7]  Alan S. Perelson,et al.  Using Genetic Algorithms to Explore Pattern Recognition in the Immune System , 1993, Evolutionary Computation.

[8]  Xin Yao,et al.  Ensemble Learning Using Multi-Objective Evolutionary Algorithms , 2006, J. Math. Model. Algorithms.

[9]  Derek Partridge,et al.  Diversity between Neural Networks and Decision Trees for Building Multiple Classifier Systems , 2000, Multiple Classifier Systems.

[10]  William B. Yates,et al.  Use of methodological diversity to improve neural network generalisation , 2005, Neural Computing & Applications.

[11]  Naonori Ueda,et al.  Generalization error of ensemble estimators , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[12]  Huanhuan Chen,et al.  Trade-Off Between Diversity and Accuracy in Ensemble Generation , 2006, Multi-Objective Machine Learning.

[13]  H. Abbass,et al.  PDE: a Pareto-frontier differential evolution approach for multi-objective optimization problems , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[14]  Ronald L. Rivest,et al.  Training a 3-node neural network is NP-complete , 1988, COLT '88.

[15]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[16]  Xin Yao,et al.  Ensemble learning via negative correlation , 1999, Neural Networks.

[17]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[18]  Gavin Brown,et al.  Diversity in neural network ensembles , 2004 .

[19]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[20]  Xin Yao,et al.  Evolving hybrid ensembles of learning machines for better generalisation , 2006, Neurocomputing.

[21]  Hussein A. Abbass,et al.  Speeding Up Backpropagation Using Multiobjective Evolutionary Algorithms , 2003, Neural Computation.

[22]  M. Anastasio,et al.  Multiobjective genetic optimization of diagnostic classifiers with implications for generating receiver operating characteristic curves , 1999, IEEE Transactions on Medical Imaging.

[23]  Ricardo H. C. Takahashi,et al.  Improving generalization of MLPs with multi-objective optimization , 2000, Neurocomputing.

[24]  Thomas G. Dietterich.  Machine-Learning Research Four Current Directions , 1997 .

[25]  Hussein A. Abbass,et al.  A Memetic Pareto Evolutionary Approach to Artificial Neural Networks , 2001, Australian Joint Conference on Artificial Intelligence.

[26]  Xin Yao,et al.  Every Niching Method has its Niche: Fitness Sharing and Implicit Sharing Compared , 1996, PPSN.

[27]  Hussein A. Abbass,et al.  Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[28]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[29]  Xin Yao,et al.  Learning and Evolution by Minimization of Mutual Information , 2002, PPSN.

[30]  Yianni Attikiouzel,et al.  A novel multicriteria optimization algorithm for the structure determination of multilayer feedforward neural networks , 1996 .

[31]  David J. Spiegelhalter,et al.  Machine Learning, Neural and Statistical Classification , 2009 .

[32]  Giorgio Valentini,et al.  Ensembles of Learning Machines , 2002, WIRN.

[33]  Xin Yao,et al.  Evolutionary framework for the construction of diverse hybrid ensembles , 2005, ESANN.

[34]  Xin Yao,et al.  Using Negative Correlation to Evolve Fault-Tolerant Circuits , 2003, ICES.

[35]  Xin Yao,et al.  Evolving Neural Network Ensembles by Minimization of Mutual Information , 2004, Int. J. Hybrid Intell. Syst..

[36]  C. Metz.  Basic principles of ROC analysis. , 1978, Seminars in nuclear medicine.

[37]  Xin Yao,et al.  DIVACE: Diverse and Accurate Ensemble Learning Algorithm , 2004, IDEAL.

[38]  David W. Opitz,et al.  Generating Accurate and Diverse Members of a Neural-Network Ensemble , 1995, NIPS.