Differential Evolution Training Algorithm for Feed-Forward Neural Networks

An evolutionary optimization method over continuous search spaces, differential evolution, has recently been successfully applied to real world and artificial optimization problems and proposed also for neural network training. However, differential evolution has not been comprehensively studied in the context of training neural network weights, i.e., how useful is differential evolution in finding the global optimum for expense of convergence speed. In this study, differential evolution has been analyzed as a candidate global optimization method for feed-forward neural networks. In comparison to gradient based methods, differential evolution seems not to provide any distinct advantage in terms of learning rate or solution quality. Differential evolution can rather be used in validation of reached optima and in the development of regularization terms and non-conventional transfer functions that do not necessarily provide gradient information.

[1]  Min Liang,et al.  Fast learning algorithms for multi-layered feedforward neural network , 1994, Proceedings of National Aerospace and Electronics Conference (NAECON'94).

[2]  Steven Doyle,et al.  Automated mirror design using an evolution strategy , 1999 .

[3]  Manfred M. Fischer,et al.  A global search procedure for parameter estimation in neural spatial interaction modelling , 1999 .

[4]  C. Charalambous,et al.  Conjugate gradient algorithm for efficient training of artifi-cial neural networks , 1990 .

[5]  X. Yao.  Evolving Artificial Neural Networks , 1999 .

[6]  R. W. Derksen,et al.  Differential Evolution in Aerodynamic Optimization , 1999 .

[7]  A. Hamler,et al.  Analysis of iron loss in interior permanent magnet synchronous motor over a wide-speed range of constant output power operation , 2000 .

[8]  S. P. Day,et al.  A stochastic training technique for feed-forward neural networks , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[9]  Jarmo T. Alander,et al.  An Indexed Bibliography of Genetic Algorithms , 1995 .

[10]  W. Land,et al.  A new training algorithm for the general regression neural network , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[11]  R. Storn,et al.  Differential evolution a simple and efficient adaptive scheme for global optimization over continu , 1997 .

[12]  Feng-Sheng Wang,et al.  Multiobjective parameter estimation problems of fermentation processes using a high ethanol tolerance yeast , 2000 .

[13]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[14]  George D. Magoulas,et al.  Hybrid methods using evolutionary algorithms for on-line training , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[15]  Lutz Prechelt.  Some notes on neural learning algorithm benchmarking , 1995, Neurocomputing.

[16]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[17]  Chris Aldrich,et al.  Combinatorial evolution of regression nodes in feedforward neural networks , 1999, Neural Networks.

[18]  Ivan Zelinka,et al.  Mechanical engineering design optimization by differential evolution , 1999 .

[19]  Kay Hameyer,et al.  Optimization of radial active magnetic bearings using the finite element technique and the differential evolution algorithm , 2000 .

[20]  T.,et al.  Training Feedforward Networks with the Marquardt Algorithm , 2004 .

[21]  Riccardo Poli,et al.  New Ideas In Optimization , 1999 .

[22]  Lutz Prechelt.  PROBEN 1 - a set of benchmarks and benchmarking rules for neural network training algorithms , 1994 .

[23]  Martin F. Møller,et al.  Efficient Training of Feed-Forward Neural Networks , 1993 .

[24]  Nathalie Japkowicz,et al.  Adaptability of the backpropagation procedure , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).