Bifurcations in the learning of recurrent neural networks

Gradient descent algorithms in recurrent neural networks can have problems when the network dynamics experience bifurcations in the course of learning. The possible hazards caused by the bifurcations of the network dynamics and the learning equations are investigated. The roles of teacher forcing, preprogramming of network structures, and the approximate learning algorithms are discussed.<<ETX>>

[1]  H. Kalmus.  Biological Cybernetics , 1972, Nature.

[2]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[3]  D. Rumelhart.  Learning internal representations by back-propagating errors , 1986 .

[4]  Fernando J. Pineda,et al.  Dynamics and architecture for neural computation , 1988, J. Complex..

[5]  Barak A. Pearlmutter.  Learning State Space Trajectories in Recurrent Neural Networks , 1989, Neural Computation.

[6]  Ronald J. Williams,et al.  Experimental Analysis of the Real-time Recurrent Learning Algorithm , 1989 .

[7]  K. Doya,et al.  Memorizing oscillatory patterns in the analog neuron network , 1989, International 1989 Joint Conference on Neural Networks.

[8]  James L. McClelland,et al.  Finite State Automata and Simple Recurrent Networks , 1989, Neural Computation.

[9]  Kenji Doya,et al.  Adaptive neural oscillator using continuous-time back-propagation learning , 1989, Neural Networks.

[10]  Garrison W. Cottrell,et al.  Some experiments on learning stable network oscillations , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[11]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[12]  Terrence J. Sejnowski,et al.  A Dynamic Neural Network Model of Sensorimotor Transformations in the Leech , 1990, Neural Computation.

[13]  Masa-aki Sato,et al.  APOLONN brings us to the real world: learning nonlinear dynamics and fluctuations in nature , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[14]  David Zipser,et al.  Recurrent Network Model of the Neural Mechanism of Short-Term Active Memory , 1991, Neural Computation.

[15]  C. Lee Giles,et al.  Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks , 1992, Neural Computation.