Evolving artificial neural network ensembles

Using a coordinated group of simple solvers to tackle a complex problem is not an entirely new idea. Its root could be traced back hundreds of years ago when ancient Chinese suggested a team approach to problem solving. For a long time, engineers have used the divide-and-conquer strategy to decompose a complex problem into simpler sub-problems and then solve them by a group of solvers. However, knowing the best way to divide a complex problem into simpler ones relies heavily on the available domain knowledge. It is often a manual process by an experienced engineer. There have been few automatic divide-and-conquer methods reported in the literature. Fortunately, evolutionary computation provides some of the interesting avenues to automatic divide-and-conquer methods. An in-depth study of such methods reveals that there is a deep underlying connection between evolutionary computation and ANN ensembles. Ideas in one area can be usefully transferred into another in producing effective algorithms. For example, using speciation to create and maintain diversity had inspired the development of negative correlation learning for ANN ensembles, and an in-depth study of diversity in ensembles. This paper will review some of the recent work in evolutionary approaches to designing ANN ensembles.

[1]  Catherine Blake,et al.  UCI Repository of machine learning databases , 1998 .

[2]  C. Cowan,et al.  Adaptive Filters and Equalisers , 1988 .

[3]  David W. Opitz,et al.  Generating Accurate and Diverse Members of a Neural-Network Ensemble , 1995, NIPS.

[4]  Lars Kai Hansen,et al.  Neural Network Ensembles , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Xin Yao,et al.  Every Niching Method has its Niche: Fitness Sharing and Implicit Sharing Compared , 1996, PPSN.

[6]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[7]  Lutz Prechelt,et al.  PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms , 1994 .

[8]  Xin Yao,et al.  An analysis of diversity measures , 2006, Machine Learning.

[9]  David B. Fogel,et al.  CONTINUOUS EVOLUTIONARY PROGRAMMING: ANALYSIS AND EXPERIMENTS , 1995 .

[10]  Xin Yao,et al.  The Evolution of Connectionist Networks , 1994 .

[11]  H. Abbass The self-adaptive Pareto differential evolution algorithm , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[12]  Terry Dartnall,et al.  Artificial Intelligence and Creativity , 1994 .

[13]  Lutz Prechelt,et al.  Some notes on neural learning algorithm benchmarking , 1995, Neurocomputing.

[14]  Yoonseon Song,et al.  Evolutionary programming using the Levy probability distribution , 1999 .

[15]  Dušan Petrovački,et al.  Evolutional development of a multilevel neural network , 1993, Neural Networks.

[16]  Sherif Hashem,et al.  Optimal Linear Combinations of Neural Networks , 1997, Neural Networks.

[17]  Ferdinand Hergert,et al.  Improving model selection by nonconvergent methods , 1993, Neural Networks.

[18]  Xin Yao,et al.  An empirical study of genetic operators in genetic algorithms , 1993, Microprocess. Microprogramming.

[19]  Hussein A. Abbass Pareto Neuro-Ensembles , 2003, Australian Conference on Artificial Intelligence.

[20]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[21]  Xin Yao,et al.  Ensemble structure of evolutionary artificial neural networks , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[22]  Herbert F. Jelinek,et al.  Complex systems : from local interactions to global phenomena , 1996 .

[23]  R. K. Ursem Multi-objective Optimization using Evolutionary Algorithms , 2009 .

[24]  Xin Yao,et al.  Evolutionary Artificial Neural Networks , 1993, Int. J. Neural Syst..

[25]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[26]  Kurt Hornik,et al.  Learning in linear neural networks: a survey , 1995, IEEE Trans. Neural Networks.

[27]  Xin Yao,et al.  A Preliminary Study on Designing Artiicial Neural Networks Using Co-evolution , 1995 .

[28]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[29]  Xin Yao,et al.  Making use of population information in evolutionary artificial neural networks , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[30]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[31]  Xin Yao,et al.  Ensemble Learning Using Multi-Objective Evolutionary Algorithms , 2006, J. Math. Model. Algorithms.

[32]  M. Perrone Improving regression estimation: Averaging methods for variance reduction with extensions to general convex measure optimization , 1993 .

[33]  James T. Kwok,et al.  Constructive algorithms for structure learning in feedforward neural networks for regression problems , 1997, IEEE Trans. Neural Networks.

[34]  Shun-ichi Amari,et al.  Mutual information of sparsely coded associative memory with self-control and ternary neurons , 2000, Neural Networks.

[35]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[36]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[37]  D. Opitz,et al.  Popular Ensemble Methods: An Empirical Study , 1999, J. Artif. Intell. Res..

[38]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .

[39]  Amanda J. C. Sharkey,et al.  On Combining Artificial Neural Nets , 1996, Connect. Sci..

[40]  Hussein A. Abbass,et al.  Speeding Up Backpropagation Using Multiobjective Evolutionary Algorithms , 2003, Neural Computation.

[41]  Peter J. B. Hancock,et al.  Genetic algorithms and permutation problems: a comparison of recombination operators for neural net structure specification , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[42]  Gavin Brown,et al.  Negative Correlation Learning and the Ambiguity Family of Ensemble Methods , 2003, Multiple Classifier Systems.

[43]  Xin Yao,et al.  Speciation as automatic categorical modularization , 1997, IEEE Trans. Evol. Comput..

[44]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[45]  Xin Yao,et al.  A constructive algorithm for training cooperative neural network ensembles , 2003, IEEE Trans. Neural Networks.

[46]  J. MacQueen Some methods for classification and analysis of multivariate observations , 1967 .

[47]  Rainer Storn,et al.  Minimizing the real functions of the ICEC'96 contest by differential evolution , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[48]  James T. Kwok,et al.  Objective functions for training new hidden units in constructive neural networks , 1997, IEEE Trans. Neural Networks.

[49]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[50]  Xin Yao,et al.  Multi-network evolutionary systems and automatic decomposition of complex problems , 2006, Int. J. Gen. Syst..

[51]  Xin Yao,et al.  Ensemble learning via negative correlation , 1999, Neural Networks.

[52]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[53]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[54]  Xin Yao,et al.  Diversity creation methods: a survey and categorisation , 2004, Inf. Fusion.

[55]  Thomas G. Dietterich Machine-Learning Research Four Current Directions , 1997 .

[56]  Thomas Bäck,et al.  Evolutionary computation: Toward a new philosophy of machine intelligence , 1997, Complex..

[57]  X. Yao,et al.  How to Make Best Use of Evolutionary Learning , 1996 .

[58]  Samir W. Mahfoud Niching methods for genetic algorithms , 1996 .

[59]  H. Abbass,et al.  PDE: a Pareto-frontier differential evolution approach for multi-objective optimization problems , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[60]  A. Krogh,et al.  Statistical mechanics of ensemble learning , 1997 .

[61]  Hussein A. Abbass,et al.  Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[62]  Kalyanmoy Deb,et al.  MULTI-OBJECTIVE FUNCTION OPTIMIZATION USING NON-DOMINATED SORTING GENETIC ALGORITHMS , 1994 .

[63]  Xin Yao,et al.  Evolutionary programming made faster , 1999, IEEE Trans. Evol. Comput..