G-Prop-II: global optimization of multilayer perceptrons using GAs

A general problem in model selection is to obtain the right parameters that make a model fit observed data. For a multilayer perceptron (MLP) trained with backpropagation (BP), this means finding the right hidden layer size, appropriate initial weights and learning parameters. The paper proposes a method (G-Prop-II) that attempts to solve that problem by combining a genetic algorithm (GA) and BP to train MLPs with a single hidden layer. The GA selects the initial weights and the learning rate of the network, and changes the number of neurons in the hidden layer through the application of specific genetic operators. G-Prop-II combines the advantages of the global search performed by the GA over the MLP parameter space and the local search of the BP algorithm. The application of the G-Prop-II algorithm to several real world and benchmark problems shows that MLPs evolved using G-Prop-II are smaller and achieve a higher level of generalization than other perceptron training algorithms, such as QuickPropagation or RPROP, and other evolutive algorithms, such as G-LVQ. It also shows some improvement over previous versions of the algorithm.

[1]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[2]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[3]  Jie Zhang,et al.  A Sequential Learning Approach for Single Hidden Layer Neural Networks , 1998, Neural Networks.

[4]  Vasant Honavar,et al.  Evolutionary Design of Neural Architectures -- A Preliminary Taxonomy and Guide to Literature , 1995 .

[5]  David White,et al.  GANNet: A Genetic Algorithm for Optimizing Topology and Weights in Neural Network Design , 1993, IWANN.

[6]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[7]  Gregory J. Wolff,et al.  Optimal Brain Surgeon: Extensions and performance comparisons , 1993, NIPS 1993.

[8]  Xin Yao,et al.  Evolving modular neural networks which generalise well , 1997, Proceedings of 1997 IEEE International Conference on Evolutionary Computation (ICEC '97).

[9]  Ivanoe De Falco,et al.  Evolutionary Neural Networks for Nonlinear Dynamics Modeling , 1998, PPSN.

[10]  Rajesh Parekh,et al.  Constructive Neural Network Learning Algorithms for Multi-Category Real-Valued Pattern Classification , 1997 .

[11]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[12]  Tariq Samad,et al.  Towards the Genetic Synthesisof Neural Networks , 1989, ICGA.

[13]  L. Darrell Whitley,et al.  Genetic algorithms and neural networks: optimizing connections and connectivity , 1990, Parallel Comput..

[14]  Scott E. Fahlman,et al.  An empirical study of learning speed in back-propagation networks , 1988 .

[15]  Anna Maria Fanelli,et al.  A Method of Pruning Layered Feed-Forward Neural Networks , 1993, IWANN.

[16]  Ethem Alpaydin,et al.  GAL: Networks That Grow When They Learn and Shrink When They Forget , 1994, Int. J. Pattern Recognit. Artif. Intell..

[17]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[18]  Xin Yao,et al.  Towards designing artificial neural networks by evolution , 1998 .

[19]  Werner Kinnebrock,et al.  Accelerating the standard backpropagation method using a genetic approach , 1994, Neurocomputing.

[20]  Geoffrey E. Hinton,et al.  How Learning Can Guide Evolution , 1996, Complex Syst..

[21]  Martin A. Riedmiller,et al.  A direct adaptive method for faster backpropagation learning: the RPROP algorithm , 1993, IEEE International Conference on Neural Networks.

[22]  David E. Rumelhart,et al.  Generalization by Weight-Elimination with Application to Forecasting , 1990, NIPS.

[23]  Michael Georgiopoulos,et al.  Coupling weight elimination with genetic algorithms to reduce network size and preserve generalization , 1997, Neurocomputing.

[24]  Hean-Lee Poh,et al.  Analysis of Pruning in Backpropagation Networks for Artificial and Real Worls Mapping Problems , 1995, IWANN.

[25]  Ernesto Tarantino,et al.  Optimizing Neural Networks for Time Series Prediction , 1999 .

[26]  Juan Julián Merelo Guervós,et al.  SA-Prop: Optimization of Multilayer Perceptron Parameters Using Simulated Annealing , 1999, IWANN.

[27]  Juan Julián Merelo Guervós,et al.  Optimization of a Competitive Learning Neural Network by Genetic Algorithms , 1993, IWANN.

[28]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.

[29]  Panos A. Ligomenides,et al.  GANNET: a genetic algorithm for searching topology and weight spaces in neural network design. The first step in finding a neural network solution , 1993 .

[30]  Lars Kai Hansen,et al.  Neural classifier construction using regularization, pruning and test error estimation , 1998, Neural Networks.

[31]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[32]  Martin P. DeSimio,et al.  MLP iterative construction algorithm , 1997, Defense, Security, and Sensing.

[33]  J. Baldwin.  A New Factor in Evolution , 1896, The American Naturalist.

[34]  O. Mangasarian,et al.  Pattern Recognition Via Linear Programming: Theory and Application to Medical Diagnosis , 1989 .

[35]  J M Carazo,et al.  Six molecules of SV40 large T antigen assemble in a propeller-shaped particle around a channel. , 1997, Journal of molecular biology.

[36]  Lutz Prechelt,et al.  PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms , 1994 .

[37]  Ignacio Bellido,et al.  Backpropagation Growing Networks: Towards Local Minima Elimination , 1991, IWANN.

[38]  Vassilios Petridis,et al.  A hybrid genetic algorithm for training neural networks , 1992 .

[39]  Jenq-Neng Hwang,et al.  The cascade-correlation learning: a projection pursuit learning perspective , 1996, IEEE Trans. Neural Networks.