Recurrent Neural Processes

We extend Neural Processes (NPs) to sequential data through Recurrent NPs or RNPs, a family of conditional state space models. RNPs model the state space with Neural Processes. Given time series observed on fast real-world time scales but containing slow long-term variabilities, RNPs may derive appropriate slow latent time scales. They do so in an efficient manner by establishing conditional independence among subsequences of the time series. Our theoretically grounded framework for stochastic processes expands the applicability of NPs while retaining their benefits of flexibility, uncertainty estimation, and favorable runtime with respect to Gaussian Processes (GPs). We demonstrate that state spaces learned by RNPs benefit predictive performance on real-world time-series data and nonlinear system identification, even in the case of limited data availability.

[1]  B. D. Finetti La prévision : ses lois logiques, ses sources subjectives , 1937 .

[2]  J. Lamperti Stochastic processes : a survey of the mathematical theory , 1979 .

[3]  J. Urgen Schmidhuber Neural Sequence Chunkers , 1991 .

[4]  Jürgen Schmidhuber,et al.  Learning Complex, Extended Sequences Using the Principle of History Compression , 1992, Neural Computation.

[5]  Yoshua Bengio,et al.  Hierarchical Recurrent Neural Networks for Long-Term Dependencies , 1995, NIPS.

[6]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[7]  Torbjörn Wigren,et al.  Input-output data sets for development and benchmarking in nonlinear identification , 2010 .

[8]  Carl E. Rasmussen,et al.  State-Space Inference and Learning with Gaussian Processes , 2010, AISTATS.

[9]  Jürgen Schmidhuber,et al.  A Clockwork RNN , 2014, ICML.

[10]  Christian Osendorfer,et al.  Learning Stochastic Recurrent Networks , 2014, NIPS 2014.

[11]  Yoshua Bengio,et al.  A Recurrent Latent Variable Model for Sequential Data , 2015, NIPS.

[12]  Honglak Lee,et al.  Learning Structured Output Representation using Deep Conditional Generative Models , 2015, NIPS.

[13]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[14]  Ryan P. Adams,et al.  Composing graphical models with neural networks for structured representations and fast inference , 2016, NIPS.

[15]  Oriol Vinyals,et al.  Matching Networks for One Shot Learning , 2016, NIPS.

[16]  Ole Winther,et al.  A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning , 2017, NIPS.

[17]  Uri Shalit,et al.  Structured Inference Networks for Nonlinear State Space Models , 2016, AAAI.

[18]  Amos J. Storkey,et al.  Towards a Neural Statistician , 2016, ICLR.

[19]  Byron Boots,et al.  Predictive-State Decoders: Encoding the Future into Recurrent Networks , 2017, NIPS.

[20]  J. Schmidhuber,et al.  The Sacred Infrastructure for Computational Research , 2017, SciPy.

[21]  Byron Boots,et al.  Predictive State Recurrent Neural Networks , 2017, NIPS.

[22]  Tuan Anh Le,et al.  Empirical Evaluation of Neural Process Objectives , 2018 .

[23]  Byron Boots,et al.  Initialization matters: Orthogonal Predictive State Recurrent Neural Networks , 2018, ICLR.

[24]  Tim G. J. Rudner On the Connection between Neural Processes and Gaussian Processes with Deep Kernels , 2018 .

[25]  Joshua B. Tenenbaum,et al.  The Variational Homoencoder: Learning to learn high capacity generative models from few examples , 2018, UAI.

[26]  Matthias W. Seeger,et al.  Deep State Space Models for Time Series Forecasting , 2018, NeurIPS.

[27]  Alexei A. Efros,et al.  Time-Agnostic Prediction: Predicting Predictable Video Frames , 2018, ICLR.

[28]  Sungjin Ahn,et al.  Sequential Neural Processes , 2019, NeurIPS.

[29]  Yee Whye Teh,et al.  Meta-learning of Sequential Strategies , 2019, ArXiv.

[30]  Stefan Zohren,et al.  Recurrent Neural Filters: Learning Independent Bayesian Filtering Steps for Time Series Prediction , 2019, 2020 International Joint Conference on Neural Networks (IJCNN).