Adjusting Weights in Artificial Neural Networks using Evolutionary Algorithms

Training artificial neural networks is a complex task of great practical importance. Besides classical ad-hoc algorithms such as backpropagation, this task can be approached by using Evolutionary Computation, a highly configurable and effective optimization paradigm. This chapter provides a brief overview of these techniques, and shows how they can be readily applied to the resolution of this problem. Three popular variants of Evolutionary Algorithms —Genetic Algorithms, Evolution Strategies and Estimation of Distribution Algorithms— are described and compared. This comparison is done on the basis of a benchmark comprising several standard classification problems of interest for neural networks. The experimental results confirm the general appropriateness of Evolutionary Computation for this problem. Evolution Strategies seem particularly proficient techniques in this optimization domain, and Estimation of Distribution Algorithms are also a competitive approach.

[1]  A. A. Mullin,et al.  Principles of neurodynamics , 1962 .

[2]  Ingo Rechenberg,et al.  Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution , 1973 .

[3]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[4]  H. P. Schwefel,et al.  Numerische Optimierung von Computermodellen mittels der Evo-lutionsstrategie , 1977 .

[5]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[6]  Geoffrey E. Hinton,et al.  Learning representations by back-propagation errors, nature , 1986 .

[7]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[8]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[9]  J. David Schaffer,et al.  Proceedings of the third international conference on Genetic algorithms , 1989 .

[10]  Thomas P. Caudell,et al.  Parametric Connectivity: Training of Constrained Networks using Genetic Algorithms , 1989, ICGA.

[11]  L. Darrell Whitley,et al.  Optimizing Neural Networks Using FasterMore Accurate Genetic Search , 1989, ICGA.

[12]  L. Darrell Whitley,et al.  Genetic algorithms and neural networks: optimizing connections and connectivity , 1990, Parallel Comput..

[13]  R. Eckmiller.  Advanced neural computers , 1990 .

[14]  Lawrence. Davis,et al.  Handbook Of Genetic Algorithms , 1990 .

[15]  Luís B. Almeida,et al.  Speeding up Backpropagation , 1990 .

[16]  L. Darrell Whitley,et al.  Delta Coding: An Iterative Search Strategy for Genetic Algorithms , 1991, ICGA.

[17]  Nostrand Reinhold,et al.  the utility of using the genetic algorithm approach on the problem of Davis, L. (1991), Handbook of Genetic Algorithms. Van Nostrand Reinhold, New York. , 1991 .

[18]  M. Kanehisa,et al.  A knowledge base for predicting protein localization sites in eukaryotic cells , 1992, Genomics.

[19]  Willfried Wienholt.  Minimizing the System Error in Feedforward Neural Networks with Evolution Strategy , 1993 .

[20]  L. Darrell Whitley,et al.  Adding Learning to the Cellular Development of Neural Networks: Evolution and the Baldwin Effect , 1993, Evolutionary Computation.

[21]  Hans-Georg Beyer,et al.  Toward a Theory of Evolution Strategies: Some Asymptotical Results from the (1,+ )-Theory , 1993, Evolutionary Computation.

[22]  Hans-Georg Beyer,et al.  Toward a Theory of Evolution Strategies: The (, )-Theory , 1994, Evolutionary Computation.

[23]  Hans-Georg Beyer,et al.  Toward a Theory of Evolution Strategies: On the Benefits of Sex the (/, ) Theory , 1995, Evolutionary Computation.

[24]  S. Baluja.  An Empirical Comparison of Seven Iterative and Evolutionary Function Optimization Heuristics , 1995 .

[25]  A. Dourado,et al.  Lime Kiln Fault Detection and Diagnosis by Neural Networks , 1995, ICANNGA.

[26]  Hans-Georg Beyer,et al.  Toward a Theory of Evolution Strategies: Self-Adaptation , 1995, Evolutionary Computation.

[27]  Tim Jones.  Evolutionary Algorithms, Fitness Landscapes and Search , 1995 .

[28]  Markus Höhfeld,et al.  Improving the Generalization Performance of Multi-Layer-Perceptrons with Population-Based Incremental Learning , 1996, PPSN.

[29]  Thomas Bäck,et al.  Evolutionary Algorithms in Theory and Practice , 1996 .

[30]  F. Herrera,et al.  Dynamic and heuristic fuzzy connectives‐based crossover operators for controlling the diversity and convergence of real‐coded genetic algorithms , 1996 .

[31]  H. Mühlenbein,et al.  From Recombination of Genes to the Estimation of Distributions I. Binary Parameters , 1996, PPSN.

[32]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[33]  L. D. Whitley.  A free lunch proof for Gray versus Binary encodings , 1999 .

[34]  Jorng-Tzong Horng,et al.  Incorporation family competition into Gaussian and Cauchy mutations to training neural networks using an evolutionary algorithm , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[35]  Sven Anderson,et al.  Training hidden Markov Models using population-based learning , 1999 .

[36]  Araceli Sanchis,et al.  Applying Evolution Strategies to Neural Networks Robot Controller , 1999, IWANN.

[37]  Pedro Ángel Castillo Valdivieso,et al.  G-Prop-II: global optimization of multilayer perceptrons using GAs , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[38]  José M. Molina,et al.  Neural networks robot controller trained with evolution strategies , 1999, Proceedings of the 1999 Congress on Evolutionary Computation-CEC99 (Cat. No. 99TH8406).

[39]  Pablo Moscato,et al.  Memetic algorithms: a short introduction , 1999 .

[40]  Byoung-Tak Zhang,et al.  Evolving neural trees for time series prediction using Bayesian evolutionary algorithms , 2000, 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No.00.

[41]  Zbigniew Michalewicz,et al.  Evolutionary Computation 1 , 2018 .

[42]  Marcus Gallagher,et al.  Multi-layer Perceptron Error Surfaces: Visualization, Structure and Modelling , 2000 .

[43]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[44]  Pedro Larrañaga,et al.  A Review on Estimation of Distribution Algorithms , 2002, Estimation of Distribution Algorithms.

[45]  D. B. Fogel,et al.  Evolving neural networks , 1990, Biological Cybernetics.