A signal processing framework based on dynamic neural networks with application to problems in adaptation, filtering, and classification

We present a coherent neural net based framework for solving various signal processing problems. It relies on the assertion that time-lagged recurrent networks possess the necessary representational capabilities to act as universal approximators of nonlinear dynamical systems. This applies to system identification, time-series prediction, nonlinear filtering, adaptive filtering, and temporal pattern classification. We address the development of models of nonlinear dynamical systems, in the form of time-lagged recurrent neural nets, which can be used without further training. We employ a weight update procedure based on the extended Kalman filter (EKF). Against the tendency for a net to forget earlier learning as it processes new examples, we develop a technique called multistream training. We demonstrate our framework by applying it to 4 problems. First, we show that a single time-lagged recurrent net can be trained to produce excellent one-time-step predictions for two different time series and also to be robust to severe errors in the input sequence. Second, we model stably a complex system containing significant process noise. The remaining two problems are drawn from real-world automotive applications. One involves input-output modeling of the dynamic behavior of a catalyst-sensor system which is exposed to an operating engine's exhaust stream, the other the real-time and continuous detection of engine misfire.

[1]  B. Anderson,et al.  Optimal Filtering , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[2]  S. Haykin,et al.  Adaptive Filter Theory , 1986 .

[3]  Sharad Singhal,et al.  Training Multilayer Perceptrons with the Extende Kalman Algorithm , 1988, NIPS.

[4]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[5]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[6]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[7]  Ronald J. Williams,et al.  Adaptive state representation and estimation using recurrent connectionist networks , 1990 .

[8]  Jing Peng,et al.  An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories , 1990, Neural Computation.

[9]  Peter R. Conwell,et al.  Fixed-weight networks can learn , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[10]  Lee A. Feldkamp,et al.  Decoupled extended Kalman filter training of feedforward layered networks , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[11]  N. E. Cotter,et al.  Learning algorithms and fixed dynamics , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[12]  Ronald J. Williams,et al.  Training recurrent networks using the extended Kalman filter , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[13]  Lee A. Feldkamp,et al.  Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks , 1994, IEEE Trans. Neural Networks.

[14]  T. Kailath,et al.  A state-space approach to adaptive RLS filtering , 1994, IEEE Signal Processing Magazine.

[15]  G. V. Puskorius,et al.  Training of robust neural controllers , 1994, Proceedings of 1994 33rd IEEE Conference on Decision and Control.

[16]  Yoshua Bengio,et al.  Learning long-term dependencies with gradient descent is difficult , 1994, IEEE Trans. Neural Networks.

[17]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[18]  Lee A. Feldkamp,et al.  Dynamic neural network methods applied to on-vehicle idle speed control , 1996, Proc. IEEE.

[19]  Lee A. Feldkamp,et al.  Adaptation from fixed weight dynamic networks , 1996, Proceedings of International Conference on Neural Networks (ICNN'96).

[20]  Lee A. Feldkamp,et al.  Adaptive Behavior from Fixed Weight Networks , 1997, Inf. Sci..

[21]  Johan A. K. Suykens,et al.  NLq theory: checking and imposing stability of recurrent neural networks for nonlinear modeling , 1997, IEEE Trans. Signal Process..

[22]  Lee A. Feldkamp,et al.  Extensions and enhancements of decoupled extended Kalman filter training , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[23]  Nazeeh Aranki,et al.  Custom VLSI ASIC for automotive applications with recurrent networks , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[24]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .