Genetic neural networks on MIMD computers

This work draws together two natural metaphors: "genetic algorithms" draw inspiration from natural evolution to attempt to provide a robust, efficient search technique, and "neural networks" form a crude model of information processing in brains which offer the prospect of training computers to solve tasks by example. It is very natural to think of applying the adaptive techniques of genetic algorithms to the problem of searching the space of neural networks, but doing so is extremely hard, and provides the motivation for this work. It is argued that the key determinant of the success of any particular genetic algorithm is the interaction between the underlying correlations in the search space, the representation of the space adopted, and the genetic operators used to manipulate the representatives of elements in the search space. It is further argued that genetic algorithms as usually formulated are not ideally suited to "training" neural networks, and that in order to make significant progress in this area a broadening of the standard "schema" analysis is required. Such a generalisation, based on the notion of imposing suitable nested sets of equivalence relations over arbitrary search spaces, is proposed and developed. "Design principles" to help construct genetic algorithms for arbitrary problems are suggested in the context of such knowledge of the regularities in the search space as are known. The techniques developed are applied to a number of problem domains and yield some new insights. Issues of linkage and convergence are also relevant to the application of adaptive genetic techniques to neural network problems. Studies of these are presented. Existing attempts to apply genetic algorithms are also reviewed in the light of the non-standard analysis developed, and the prospects for further progress are discussed. In recognition of the fact that much of this work was carried Out on a large, medium-grained, reconfigurable parallel computer, a study of connection strategies for such machines is also presented.

[1]  J. Nadal,et al.  Learning in feedforward layered networks: the tiling algorithm , 1989 .

[2]  David E. Goldberg,et al.  Alleles, loci and the traveling salesman problem , 1985 .

[3]  D. E. Goldberg,et al.  Genetic Algorithms in Search, Optimization & Machine Learning , 1989 .

[4]  John J. Grefenstette,et al.  Genetic Algorithms for the Traveling Salesman Problem , 1985, ICGA.

[5]  N. Dodd,et al.  Optimisation of network structure using genetic techniques , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[6]  Tariq Samad,et al.  Towards the Genetic Synthesisof Neural Networks , 1989, ICGA.

[7]  James E. Baker,et al.  Reducing Bias and Inefficienry in the Selection Algorithm , 1987, ICGA.

[8]  R. Dawkins The Extended Phenotype , 1982 .

[9]  Heinz Mühlenbein,et al.  Parallel Genetic Algorithms, Population Genetics, and Combinatorial Optimization , 1989, Parallelism, Learning, Evolution.

[10]  John J. Grefenstette,et al.  Optimization of Control Parameters for Genetic Algorithms , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[11]  Eugene L. Lawler,et al.  The travelling salesman problem (A guided tour of combinatorial optimisation ), edited by E. L. Lawler, J. K. Lenstra, A. H. G. Rinnooy Kan and D. B. Shmoys. Pp 465. £39·95. 1985. ISBN 0-471-90413-9 (Wiley) , 1986, The Mathematical Gazette.

[12]  L. Darrell Whitley,et al.  The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials is Best , 1989, ICGA.

[13]  Peter M. Todd,et al.  Designing Neural Networks using Genetic Algorithms , 1989, ICGA.

[14]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[15]  R. Dawkins,et al.  Replicator selection and the extended phenotype. , 2010, Zeitschrift fur Tierpsychologie.

[16]  Eugene L. Lawler,et al.  The Traveling Salesman Problem: A Guided Tour of Combinatorial Optimization , 1985 .

[17]  de Ng Dick Bruijn A combinatorial problem , 1946 .

[18]  J. David Schaffer,et al.  Representation and Hidden Bias: Gray vs. Binary Coding for Genetic Algorithms , 1988, ML.

[19]  Gilbert Syswerda,et al.  Uniform Crossover in Genetic Algorithms , 1989, ICGA.

[20]  David H. Sharp,et al.  Scaling, machine learning, and genetic neural nets , 1989 .

[21]  Rajarshi Das,et al.  A Study of Control Parameters Affecting Online Performance of Genetic Algorithms for Function Optimization , 1989, ICGA.

[22]  Lawrence Davis,et al.  Adapting Operator Probabilities in Genetic Algorithms , 1989, ICGA.

[23]  James E. Baker,et al.  Adaptive Selection Methods for Genetic Algorithms , 1985, International Conference on Genetic Algorithms.

[24]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[25]  Daniel Raymond Frantz,et al.  Nonlinearities in genetic adaptive search. , 1972 .

[26]  D. J. Cavicchio,et al.  Adaptive search using simulated evolution , 1970 .

[27]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[28]  C. G. Shaefer,et al.  The ARGOT Strategy: Adaptive Representation Genetic Optimizer Technique , 1987, ICGA.

[29]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[30]  John F. Kolen,et al.  Backpropagation is Sensitive to Initial Conditions , 1990, Complex Syst..

[31]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[32]  L. Darrell Whitley,et al.  Using Reproductive Evaluation to Improve Genetic Search and Heuristic Discovery , 1987, International Conference on Genetic Algorithms.

[33]  Larry J. Eshelman,et al.  Biases in the Crossover Landscape , 1989, ICGA.

[34]  R.J.F. Dow,et al.  Neural net pruning-why and how , 1988, IEEE 1988 International Conference on Neural Networks.

[35]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[36]  L. Darrell Whitley,et al.  Optimizing Neural Networks Using FasterMore Accurate Genetic Search , 1989, ICGA.

[37]  Larry J. Eshelman,et al.  Using genetic search to exploit the emergent behavior of neural networks , 1990 .

[38]  Dana S. Richards,et al.  Punctuated Equilibria: A Parallel Genetic Algorithm , 1987, ICGA.

[39]  Tom Tollenaere,et al.  SuperSAB: Fast adaptive back propagation with good scaling properties , 1990, Neural Networks.

[40]  Shen Lin Computer solutions of the traveling salesman problem , 1965 .

[41]  Kenneth Alan De Jong,et al.  An analysis of the behavior of a class of genetic adaptive systems. , 1975 .

[42]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .