Constructive learning of recurrent neural networks

It is difficult to determine the minimal neural network structure for a particular automaton. A large recurrent network in practice is very difficult to train. Constructive or destructive recurrent methods might offer a solution to this problem. It is proved that one current method, recurrent cascade correlation, has fundamental limitations in representation and thus in its learning capabilities. A preliminary approach to circumventing these limitations by devising a simple constructive training method that adds neurons during training while still preserving the powerful fully recurrent structure is given. Through simulations it is shown that such a method can learn many types of regular grammars which the recurrent cascade correlation method is unable to learn.<<ETX>>

[1]  C. L. Giles,et al.  Machine learning using higher order correlation networks , 1986 .

[2]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[3]  T. Ash,et al.  Dynamic node creation in backpropagation networks , 1989, International 1989 Joint Conference on Neural Networks.

[4]  Timur Ash,et al.  Dynamic node creation in backpropagation networks , 1989 .

[5]  James L. McClelland,et al.  Finite State Automata and Simple Recurrent Networks , 1989, Neural Computation.

[6]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[7]  Stephen Jose Hanson,et al.  Meiosis Networks , 1989, NIPS.

[8]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[9]  Jude Shavlik,et al.  Refinement ofApproximate Domain Theories by Knowledge-Based Neural Networks , 1990, AAAI.

[10]  Noga Alon,et al.  Efficient simulation of finite automata by neural nets , 1991, JACM.

[11]  Giovanni Soda,et al.  An unified approach for integrating explicit knowledge and learning by example in recurrent networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[12]  Hava T. Siegelmann,et al.  On the computational power of neural nets , 1992, COLT '92.

[13]  C. Lee Giles,et al.  Learning and Extracting Finite State Automata with Second-Order Recurrent Neural Networks , 1992, Neural Computation.

[14]  Raymond L. Watrous,et al.  Induction of Finite-State Languages Using Second-Order Recurrent Networks , 1992, Neural Computation.

[15]  C. L. Giles,et al.  Inserting rules into recurrent neural networks , 1992, Neural Networks for Signal Processing II Proceedings of the 1992 IEEE Workshop.