Combinatorial evolution of regression nodes in feedforward neural networks

A number of techniques exist with which neural network architectures such as multilayer perceptrons and radial basis function networks can be trained. These include backpropagation, k-means clustering and evolutionary algorithms. The latter method is particularly useful as it is able to avoid local optima in the search space and can optimise parameters for which no gradient information exists. Unfortunately, only moderately sized networks can be trained by this method, owing to the fact that evolutionary optimisation is very computationally intensive. In this paper a novel algorithm (CERN) is therefore proposed which uses a special form of combinatorial search to optimise groups of neural nodes. Oriented, ellipsoidal basis nodes optimised with CERN achieved significantly better accuracy with fewer nodes than spherical basis nodes optimised by k-means clustering. Multilayer perceptrons optimised by CERN were found to be as accurate as those trained by advanced gradient descent techniques. CERN was also found to be significantly more efficient than a conventional evolutionary algorithm that does not use a combinatorial search.

[1]  Avijit Saha,et al.  Oriented Non-Radial Basis Functions for Image Coding and Analysis , 1990, NIPS.

[2]  Michael Conrad,et al.  Combining evolution with credit apportionment: A new learning algorithm for neural nets , 1994, Neural Networks.

[3]  R. Storn,et al.  Differential evolution a simple and efficient adaptive scheme for global optimization over continu , 1997 .

[4]  Bernd Fritzke,et al.  Growing cell structures--A self-organizing network for unsupervised and supervised learning , 1994, Neural Networks.

[5]  David S. Broomhead,et al.  Multivariable Functional Interpolation and Adaptive Networks , 1988, Complex Syst..

[6]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[7]  Rosalind W. Picard,et al.  On the efficiency of the orthogonal least squares training method for radial basis function networks , 1996, IEEE Trans. Neural Networks.

[8]  Stefan Bornholdt,et al.  General asymmetric neural networks and structure design by genetic algorithms: a learning rule for temporal patterns , 1992, Proceedings of IEEE Systems Man and Cybernetics Conference - SMC.

[9]  Kalyanmoy Deb,et al.  An Investigation of Niche and Species Formation in Genetic Function Optimization , 1989, ICGA.

[10]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[11]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[12]  Y Lu,et al.  A Sequential Learning Scheme for Function Approximation Using Minimal Radial Basis Function Neural Networks , 1997, Neural Computation.

[13]  J. Nadal,et al.  Learning in feedforward layered networks: the tiling algorithm , 1989 .

[14]  J. Saffery,et al.  Using stereographic projection as a preprocessing technique for Upstart , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[15]  Sheng Chen,et al.  Practical identification of NARMAX models using radial basis functions , 1990 .

[16]  L. Darrell Whitley,et al.  Genetic algorithms and neural networks: optimizing connections and connectivity , 1990, Parallel Comput..

[17]  Stephen A. Billings,et al.  Radial basis function network configuration using genetic algorithms , 1995, Neural Networks.

[18]  D. B. Fogel,et al.  Using evolutionary programing to create neural networks that are capable of playing tic-tac-toe , 1993, IEEE International Conference on Neural Networks.

[19]  Patrick K. Simpson,et al.  Fuzzy min-max neural networks. I. Classification , 1992, IEEE Trans. Neural Networks.

[20]  Shang-Liang Chen,et al.  Orthogonal least squares learning algorithm for radial basis function networks , 1991, IEEE Trans. Neural Networks.

[21]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[22]  Hans-Michael Voigt,et al.  Task Decomposition and Correlations in Growing Artificial Neural Networks , 1994 .

[23]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[24]  E. Fiesler,et al.  Comparative Bibliography of Ontogenic Neural Networks , 1994 .

[25]  Peter J. B. Hancock,et al.  Genetic algorithms and permutation problems: a comparison of recombination operators for neural net structure specification , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[26]  Bruce A. Whitehead,et al.  Evolving space-filling curves to distribute radial basis functions over an input space , 1994, IEEE Trans. Neural Networks.

[27]  D. Broomhead,et al.  Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks , 1988 .

[28]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[29]  John E. Moody,et al.  Fast Learning in Multi-Resolution Hierarchies , 1988, NIPS.

[30]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[31]  M. J. D. Powell,et al.  Radial basis functions for multivariable interpolation: a review , 1987 .

[32]  E. Littmann.  Generalization Abilities of Cascade Network Architectures , 1992 .

[33]  R. Scott Crowder,et al.  Predicting the Mackey-Glass Timeseries With Cascade-Correlation Learning , 1990 .

[34]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[35]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[36]  J. Rissanen,et al.  Modeling By Shortest Data Description* , 1978, Autom..

[37]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[38]  Vittorio Maniezzo,et al.  Genetic evolution of the topology and weight distribution of neural networks , 1994, IEEE Trans. Neural Networks.

[39]  George A. F. Seber,et al.  Linear regression analysis , 1977 .

[40]  Tomas Hrycej.  Modular learning in neural networks - a modularized approach to neural network classification , 1992, Sixth-Generation computer technology series.

[41]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[42]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[43]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[44]  Bruce A. Whitehead,et al.  Cooperative-competitive genetic evolution of radial basis function centers and widths for time series prediction , 1996, IEEE Trans. Neural Networks.

[45]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[46]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[47]  R. Palmer,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[48]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[49]  P. K. Simpson.  Fuzzy Min-Max Neural Networks-Part 1 : Classification , 1992 .

[50]  Dušan Petrovački,et al.  Evolutional development of a multilevel neural network , 1993, Neural Networks.

[51]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[52]  Wolfram Schiffmann,et al.  Performance Evaluation of Evolutionarily Created Neural Network Topologies , 1990, PPSN.

[53]  Helge Ritter,et al.  Cascade network architectures , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[54]  David S. Watkins,et al.  Fundamentals of matrix computations , 1991 .

[55]  Robert W. Wilson,et al.  Regressions by Leaps and Bounds , 2000, Technometrics.