Deep recurrent Gaussian processes for outlier-robust system identification

Abstract Gaussian Processes (GP) comprise a powerful kernel-based machine learning paradigm which has recently attracted the attention of the nonlinear system identification community, specially due to its inherent Bayesian-style treatment of the uncertainty. However, since standard GP models assume a Gaussian distribution for the observation noise, i.e., a Gaussian likelihood, the learning and predictive capabilities of such models can be severely degraded when outliers are present in the data. In this paper, motivated by our previous work on GP learning with data containing outliers and recent advances in hierarchical (deep GPs) and recurrent GP (RGP) approaches, we introduce an outlier-robust recurrent GP model, the RGP- t . Our approach explicitly models the observation layer, which includes a heavy-tailed Student- t likelihood, and allows for a hierarchy of multiple transition layers to learn the system dynamics directly from estimation data contaminated by outliers. In addition, we modify the original variational framework of standard RGP in order to perform inference with the new RGP- t model. The proposed approach is comprehensively evaluated using six artificial benchmarks, within several outlier contamination levels, and two datasets related to process industry systems (pH neutralization and heat exchanger), whose estimation data undergo large contamination rates. The simulation results obtained by the RGP- t model indicates an impressive resilience to outliers and a superior capability to learn nonlinear dynamics directly from highly outlier-contaminated data in comparison to existing GP models.

[1]  Andreas C. Damianou,et al.  Deep Gaussian processes and variational propagation of uncertainty , 2015 .

[2]  Neil D. Lawrence,et al.  Latent Autoregressive Gaussian Processes Models for Robust System Identification , 2016 .

[3]  Giulio Bottegal,et al.  Outlier robust system identification: a Bayesian kernel-based approach , 2013, 1312.6317.

[4]  Neil D. Lawrence,et al.  Semi-described and semi-supervised learning with Gaussian processes , 2015, UAI.

[5]  Radford M. Neal.  Monte Carlo Implementation of Gaussian Process Models for Bayesian Regression and Classification , 1997, physics/9701026.

[6]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine Learning.

[7]  Nasser M. Nasrabadi,et al.  Pattern Recognition and Machine Learning , 2006, Technometrics.

[8]  O. Nelles.  Nonlinear System Identification: From Classical Approaches to Neural Networks and Fuzzy Models , 2000 .

[9]  Neil D. Lawrence,et al.  Deep Gaussian Processes , 2012, AISTATS.

[10]  Roger Woodard,et al.  Interpolation of Spatial Data: Some Theory for Kriging , 1999, Technometrics.

[11]  Benjamin Berger,et al.  Robust Gaussian Process Modelling for Engine Calibration , 2012 .

[12]  Neil D. Lawrence,et al.  Variational inference for Student-t models: Robust Bayesian interpolation and generalised component analysis , 2005, Neurocomputing.

[13]  G. Matheron.  The intrinsic random functions and their applications , 1973, Advances in Applied Probability.

[14]  K. Obermayer,et al.  Multiple-step ahead prediction for non linear dynamic systems: A Gaussian Process treatment with propagation of the uncertainty , 2003, NIPS 2003.

[15]  Geoffrey E. Hinton,et al.  Evaluation of Gaussian processes and other methods for non-linear regression , 1997 .

[16]  Kumpati S. Narendra,et al.  Identification and control of dynamical systems using neural networks , 1990, IEEE Trans. Neural Networks.

[17]  R. B. Gopaluni.  A particle filter approach to identification of nonlinear processes under missing observations , 2008 .

[18]  Neil D. Lawrence,et al.  Recurrent Gaussian Processes , 2015, ICLR.

[19]  Henrik Ohlsson,et al.  On the estimation of transfer functions, regularizations and Gaussian processes - Revisited , 2012, Autom..

[20]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[21]  Wolfram Burgard,et al.  Learning Non-stationary System Dynamics Online Using Gaussian Processes , 2010, DAGM-Symposium.

[22]  Aki Vehtari,et al.  Robust Gaussian Process Regression with a Student-t Likelihood , 2011, J. Mach. Learn. Res..

[23]  Charu C. Aggarwal,et al.  Outlier Detection for Temporal Data: A Survey , 2014, IEEE Transactions on Knowledge and Data Engineering.

[24]  Tom Minka,et al.  Expectation Propagation for approximate Bayesian inference , 2001, UAI.

[25]  Arnaud Doucet,et al.  On Particle Methods for Parameter Estimation in State-Space Models , 2014, 1412.8695.

[26]  Neil D. Lawrence,et al.  Bayesian Gaussian Process Latent Variable Model , 2010, AISTATS.

[27]  Weiyu Xu,et al.  System identification in the presence of outliers and random noises: A compressed sensing approach , 2014, Autom..

[28]  Johan Dahlin,et al.  Sequential Monte Carlo Methods for System Identification , 2015, 1503.06058.

[29]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[30]  Carl E. Rasmussen,et al.  Identification of Gaussian Process State-Space Models with Particle Stochastic Approximation EM , 2013, ArXiv.

[31]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[32]  Ronald K. Pearson,et al.  Outliers in process modeling and identification , 2002, IEEE Trans. Control. Syst. Technol..

[33]  V. Peterka.  BAYESIAN APPROACH TO SYSTEM IDENTIFICATION , 1981 .

[34]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[35]  Frank L. Lewis,et al.  Backlash compensation with filtered prediction in discrete time nonlinear systems by dynamic inversion using neural networks , 2000, Proceedings of the 39th IEEE Conference on Decision and Control (Cat. No.00CH37187).

[36]  P. Rousseeuw,et al.  Partitioning Around Medoids (Program PAM) , 2008 .

[37]  Sean B. Holden,et al.  Robust Regression with Twinned Gaussian Processes , 2007, NIPS.

[38]  David Barber,et al.  Bayesian reasoning and machine learning , 2012 .

[39]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[40]  Agathe Girard,et al.  Dynamic systems identification with Gaussian processes , 2005 .

[41]  P. L. Green.  Bayesian system identification of a nonlinear dynamical system using a novel variant of Simulated Annealing , 2015 .

[42]  Malte Kuß,et al.  Gaussian process models for robust regression, classification, and reinforcement learning , 2006 .

[43]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[44]  Carl E. Rasmussen,et al.  In Advances in Neural Information Processing Systems , 2011 .

[45]  S. Billings.  Nonlinear System Identification: NARMAX Methods in the Time, Frequency, and Spatio-Temporal Domains , 2013 .

[46]  Razvan Pascanu,et al.  How to Construct Deep Recurrent Neural Networks , 2013, ICLR.

[47]  Oliver Stegle,et al.  Gaussian Process Robust Regression for Noisy Heart Rate Data , 2008, IEEE Transactions on Biomedical Engineering.

[48]  Carl E. Rasmussen,et al.  Derivative Observations in Gaussian Process Models of Dynamic Systems , 2002, NIPS.

[49]  Biao Huang,et al.  Robust Gaussian process modeling using EM algorithm , 2016 .

[50]  Jus Kocijan,et al.  Dynamical systems identification using Gaussian process models with incorporated local models , 2011, Eng. Appl. Artif. Intell..

[51]  T. Johansen,et al.  On transient dynamics, off-equilibrium behaviour and identification in blended multiple model structures , 1999, 1999 European Control Conference (ECC).

[52]  Guilherme De A. Barreto,et al.  An Empirical Evaluation of Robust Gaussian Process Models for System Identification , 2015, IDEAL.

[53]  Carl E. Rasmussen,et al.  Variational Gaussian Process State-Space Models , 2014, NIPS.