Evolving recurrent perceptrons for time-series modeling

Evolutionary programming, a systematic multi-agent stochastic search technique, is used to generate recurrent perceptrons (nonlinear IIR filters). A hybrid optimization scheme is proposed that embeds a single-agent stochastic search technique, the method of Solis and Wets, into the evolutionary programming paradigm. The proposed hybrid optimization approach is further augmented by "blending" randomly selected parent vectors to create additional offspring. The first part of this work investigates the performance of the suggested hybrid stochastic search method. After demonstration on the Bohachevsky and Rosenbrock response surfaces, the hybrid stochastic optimization approach is applied in determining both the model order and the coefficients of recurrent perceptron time-series models. An information criterion is used to evaluate each recurrent perceptron structure as a candidate solution. It is speculated that the stochastic training method implemented in this study for training recurrent perceptrons can be used to train perceptron networks that have radically recurrent architectures.

[1]  Samuel H. Brooks.  A Discussion of Random Methods for Seeking Maxima , 1958 .

[2]  H. H. Rosenbrock,et al.  An Automatic Method for Finding the Greatest or Least Value of a Function , 1960, Comput. J..

[3]  Dean C. Karnopp,et al.  Random search techniques for optimization problems , 1963, at - Automatisierungstechnik.

[4]  Arthur Gelb,et al.  Applied Optimal Estimation , 1974 .

[5]  H. Akaike.  A new look at the statistical model identification , 1974 .

[6]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[7]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[8]  H. Tong,et al.  Threshold Autoregression, Limit Cycles and Cyclical Data , 1980 .

[9]  Singiresu S. Rao,et al.  Optimization Theory and Applications , 1980, IEEE Transactions on Systems, Man, and Cybernetics.

[10]  R. Kashyap.  Inconsistency of the AIC rule for estimating the order of autoregressive models , 1980 .

[11]  Roger J.-B. Wets,et al.  Minimization by Random Search Techniques , 1981, Math. Oper. Res..

[12]  J. Rissanen.  Stochastic Complexity and Modeling , 1986 .

[13]  M. E. Johnson,et al.  Generalized simulated annealing for function optimization , 1986 .

[14]  D A Pierre,et al.  Optimization Theory with Applications , 1986 .

[15]  M. B. Priestley,et al.  Non-linear and non-stationary time series analysis , 1990 .

[16]  Geoffrey E. Hinton,et al.  Phoneme recognition using time-delay neural networks , 1989, IEEE Trans. Acoust. Speech Signal Process..

[17]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[18]  Alex Waibel,et al.  Consonant recognition by modular construction of large phonemic time-delay neural networks , 1989, International Conference on Acoustics, Speech, and Signal Processing,.

[19]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[20]  Norio Baba,et al.  A new approach for finding the global minimum of error function of neural networks , 1989, Neural Networks.

[21]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[22]  S. Qian,et al.  Nonlinear adaptive networks: A little theory, a few applications , 1990 .

[23]  Ronald J. Williams,et al.  Adaptive state representation and estimation using recurrent connectionist networks , 1990 .

[24]  David E. Rumelhart,et al.  Predicting the Future: a Connectionist Approach , 1990, Int. J. Neural Syst..

[25]  Emile H. L. Aarts,et al.  Simulated annealing and Boltzmann machines - a stochastic approach to combinatorial optimization and neural computing , 1990, Wiley-Interscience series in discrete mathematics and optimization.

[26]  D. Haesloop,et al.  Neural networks for process identification , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[27]  P. S. Lewis,et al.  Function approximation and time series prediction with neural networks , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[28]  D. Fogel.  System Identification Through Simulated Evolution: A Machine Learning Approach to Modeling , 1991 .

[29]  Ah Chung Tsoi,et al.  FIR and IIR Synapses, a New Neural Network Architecture for Time Series Modeling , 1991, Neural Computation.

[30]  D. B. Fogel,et al.  Using evolutionary programming for modeling: an ocean acoustic example , 1992 .

[31]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[32]  David B. Fogel,et al.  Evolving artificial intelligence , 1992 .

[33]  Sukhan Lee,et al.  Supervised learning with Gaussian potentials , 1992 .

[34]  Geoffrey E. Hinton,et al.  Simplifying Neural Networks by Soft Weight-Sharing , 1992, Neural Computation.

[35]  Antonette M. Logar,et al.  A comparison of recurrent neural network learning algorithms , 1993, IEEE International Conference on Neural Networks.

[36]  Lars Kai Hansen,et al.  On design and evaluation of tapped-delay neural network architectures , 1993, IEEE International Conference on Neural Networks.

[37]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[38]  S. Haykin,et al.  A cascaded recurrent neural network for real-time nonlinear adaptive filtering , 1993, IEEE International Conference on Neural Networks.

[39]  V. Ramamurti,et al.  A hybrid technique to enhance the performance of recurrent neural networks for time series prediction , 1993, IEEE International Conference on Neural Networks.

[40]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[41]  David B. Fogel,et al.  CONTINUOUS EVOLUTIONARY PROGRAMMING: ANALYSIS AND EXPERIMENTS , 1995 .

[42]  Zbigniew Michalewicz,et al.  Genetic Algorithms + Data Structures = Evolution Programs , 1996, Springer Berlin Heidelberg.