A learning result for continuous-time recurrent neural networks 1 1 Supported in part by US Air Forc

Abstract The following learning problem is considered, for continuous-time recurrent neural networks having sigmoidal activation functions. Given a “black box” representing an unknown system, measurements of output derivatives are collected, for a set of randomly generated inputs, and a network is used to approximate the observed behavior. It is shown that the number of inputs needed for reliable generalization (the sample complexity of the learning problem) is upper bounded by an expression that grows polynomially with the dimension of the network and logarithmically with the number of output derivatives being matched.

[1]  Paul W. Goldberg,et al.  Bounding the Vapnik-Chervonenkis Dimension of Concept Classes Parameterized by Real Numbers , 1993, COLT '93.

[2]  Eduardo D. Sontag,et al.  State observability in recurrent neural networks , 1993, Proceedings of 32nd IEEE Conference on Decision and Control.

[3]  Eduardo D. Sontag,et al.  Mathematical Control Theory: Deterministic Finite Dimensional Systems , 1990 .

[4]  Yoshua Bengio,et al.  Neural networks for speech and sequence recognition , 1996 .

[5]  Yun-Chung Chu,et al.  Bounds of the induced norm and model reduction errors for systems with repeated scalar nonlinearities , 1999, IEEE Trans. Autom. Control..

[6]  Marek Karpinski,et al.  Polynomial bounds for VC dimension of sigmoidal neural networks , 1995, STOC '95.

[7]  Rafal Zbikowski.  Lie Algebra of Recurrent Neural Networks and Identifiability , 1993, 1993 American Control Conference.

[8]  Richard S. Sutton,et al.  Neural networks for control , 1990 .

[9]  Mathukumalli Vidyasagar,et al.  A Theory of Learning and Generalization , 1997 .

[10]  Kenneth J. Hunt,et al.  Neural Adaptive Control Technology , 1996 .

[11]  Eduardo D. Sontag,et al.  For neural networks, function determines form , 1992, [1992] Proceedings of the 31st IEEE Conference on Decision and Control.

[12]  Eduardo D. Sontag,et al.  Neural Networks for Control , 1993 .

[13]  Allan Pinkus,et al.  Multilayer Feedforward Networks with a Non-Polynomial Activation Function Can Approximate Any Function , 1991, Neural Networks.

[14]  Eduardo Sontag,et al.  Turing computability with neural nets , 1991 .

[15]  Eduardo D. Sontag,et al.  NEURAL NETS AS SYSTEMS MODELS AND CONTROLLERS , 1992 .

[16]  Eduardo D. Sontag,et al.  Sample complexity for learning recurrent perceptron mappings , 1995, IEEE Trans. Inf. Theory.

[17]  Eduardo D. Sontag,et al.  Recurrent Neural Networks: Some Systems-Theoretic Aspects , 1997 .

[18]  Ah Chung Tsoi,et al.  FIR and IIR Synapses, a New Neural Network Architecture for Time Series Modeling , 1991, Neural Computation.

[19]  Eduardo D. Sontag,et al.  Complete controllability of continuous-time recurrent neural networks , 1997 .

[20]  Mathukumalli Vidyasagar,et al.  A Theory of Learning and Generalization: With Applications to Neural Networks and Control Systems , 1997 .

[21]  Eduardo D. Sontag,et al.  Vapnik-Chervonenkis Dimension of Recurrent Neural Networks , 1997, Discret. Appl. Math..

[22]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[23]  David Haussler,et al.  Decision Theoretic Generalizations of the PAC Model for Neural Net and Other Learning Applications , 1992, Inf. Comput..

[24]  Marek Karpinski,et al.  Polynomial Bounds for VC Dimension of Sigmoidal and General Pfaffian Neural Networks , 1997, J. Comput. Syst. Sci..