Applying LSTM to Time Series Predictable through Time-Window Approaches

Long Short-Term Memory (LSTM) is able to solve many time series tasks unsolvable by feed-forward networks using fixed size time windows. Here we find that LSTM''s superiority does {\em not} carry over to certain simpler time series prediction tasks solvable by time window approaches: the Mackey-Glass series and the Santa Fe FIR laser emission series (Set A). This suggests to use LSTM only when simpler traditional approaches fail.

[1]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[2]  Hübner,et al.  Dimensions and entropies of chaotic intensity pulsations in a single-mode far-infrared NH3 laser. , 1989, Physical review. A, General physics.

[3]  Jirí Benes,et al.  On neural networks , 1990, Kybernetika.

[4]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[5]  Alexander H. Waibel,et al.  Multi-State Time Delay Networks for Continuous Speech Recognition , 1991, NIPS.

[6]  Michael R. Davenport,et al.  Continuous-time temporal back-propagation with adaptable time delays , 1993, IEEE Trans. Neural Networks.

[7]  Thomas Martinetz,et al.  'Neural-gas' network for vector quantization and its application to time-series prediction , 1993, IEEE Trans. Neural Networks.

[8]  A. Weigend,et al.  Time Series Prediction: Forecasting the Future and Understanding the Past , 1994 .

[9]  José Carlos Príncipe,et al.  Dynamic Modelling of Chaotic Time Series with Neural Networks , 1994, NIPS.

[10]  Andreas S. Weigend,et al.  Time Series Prediction: Forecasting the Future and Understanding the Past , 1994 .

[11]  A. Weigend,et al.  Predictions with Confidence Intervals (Local Error Bars) ; CU-CS-724-94 , 1994 .

[12]  Peter Tiño,et al.  Learning long-term dependencies in NARX recurrent neural networks , 1996, IEEE Trans. Neural Networks.

[13]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[14]  Xin Yao,et al.  A new evolutionary system for evolving artificial neural networks , 1997, IEEE Trans. Neural Networks.

[15]  Jukka Heikkonen,et al.  Recurrent SOM with local linear models in time series prediction , 1998, ESANN.

[16]  Mauro Birattari,et al.  Adaptive memory based regression methods , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[17]  Ivanoe De Falco,et al.  Evolutionary Neural Networks for Nonlinear Dynamics Modeling , 1998, PPSN.

[18]  Mauro Birattari,et al.  Local Learning for Iterated Time-Series Prediction , 1999, ICML.

[19]  Jürgen Schmidhuber,et al.  Learning to Forget: Continual Prediction with LSTM , 2000, Neural Computation.

[20]  Local Modeling Optimization for Time Series Prediction , 2000 .

[21]  C. Lee Giles,et al.  Learning Chaotic Attractors by Neural Networks , 2000, Neural Computation.

[22]  Jürgen Schmidhuber,et al.  Recurrent nets that time and count , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[23]  Romuald Boné,et al.  A bounded exploration approach to constructive algorithms for recurrent neural networks , 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium.

[24]  James McNames.  Local model optimization for time series prediction , 2000, ESANN.

[25]  Jürgen Schmidhuber,et al.  LSTM recurrent networks learn simple context-free and context-sensitive languages , 2001, IEEE Trans. Neural Networks.