Linear Latent Force Models Using Gaussian Processes

Purely data-driven approaches for machine learning present difficulties when data are scarce relative to the complexity of the model or when the model is forced to extrapolate. On the other hand, purely mechanistic approaches need to identify and specify all the interactions in the problem at hand (which may not be feasible) and still leave the issue of how to parameterize the system. In this paper, we present a hybrid approach using Gaussian processes and differential equations to combine data-driven modeling with a physical model of the system. We show how different, physically inspired, kernel functions can be developed through sensible, simple, mechanistic assumptions about the underlying system. The versatility of our approach is illustrated with three case studies from motion capture, computational biology, and geostatistics.

[1]  Venkataramanan Balakrishnan,et al.  System identification: theory for the user (second edition): Lennart Ljung; Prentice-Hall, Englewood Cliffs, NJ, 1999, ISBN 0-13-656695-2 , 2002, Autom..

[2]  Ronald P. Barry,et al.  Blackbox Kriging: Spatial Prediction Without Specifying Variogram Models , 1996 .

[3]  B. Øksendal Stochastic differential equations : an introduction with applications , 1987 .

[4]  Ted Chang,et al.  Introduction to Geostatistics: Applications in Hydrogeology , 2001, Technometrics.

[5]  Ronald P. Barry,et al.  Constructing and fitting models for cokriging and multivariable spatial prediction , 1998 .

[6]  Thore Graepel,et al.  Solving Noisy Linear Operator Equations by Gaussian Processes: Application to Ordinary and Partial Differential Equations , 2003, ICML.

[7]  Antti Honkela,et al.  Model-based method for transcription factor target identification with limited data , 2010, Proceedings of the National Academy of Sciences.

[8]  Uwe D. Hanebeck,et al.  Analytic moment-based Gaussian process filtering , 2009, ICML '09.

[9]  Mark J. Schervish,et al.  Nonstationary Covariance Functions for Gaussian Process Regression , 2003, NIPS.

[10]  Michalis K. Titsias,et al.  Variational Learning of Inducing Variables in Sparse Gaussian Processes , 2009, AISTATS.

[11]  J. Rougier Efficient Emulators for Multivariate Deterministic Functions , 2008 .

[12]  null null,et al.  Review of Geostatistics in Geohydrology. II: Applications , 1990 .

[13]  Agathe Girard,et al.  Dynamic systems identification with Gaussian processes , 2005 .

[14]  Jiguo Cao,et al.  Parameter estimation for differential equations: a generalized smoothing approach , 2007 .

[15]  A. Balakrishnan Applied Functional Analysis , 1976 .

[16]  Yonathan Bard,et al.  Nonlinear parameter estimation , 1974 .

[17]  L. Reichl A modern course in statistical physics , 1980 .

[18]  E. G. Vomvoris,et al.  A geostatistical approach to the inverse problem in groundwater modeling (steady state) and one‐dimensional simulations , 1983 .

[19]  Yee Whye Teh,et al.  Semiparametric latent factor models , 2005, AISTATS.

[20]  Neil D. Lawrence,et al.  Efficient Sampling for Gaussian Process Inference using Control Variables , 2008, NIPS.

[21]  G. Roach,et al.  Green's Functions , 1982 .

[22]  Marcus R. Frean,et al.  Dependent Gaussian Processes , 2004, NIPS.

[23]  D. Higdon Space and Space-Time Modeling using Process Convolutions , 2002 .

[24]  I. Stakgold Green's Functions and Boundary Value Problems , 1979 .

[25]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[26]  Timothy C. Coburn,et al.  Geostatistics for Natural Resources Evaluation , 2000, Technometrics.

[27]  Neil D. Lawrence,et al.  Efficient Multioutput Gaussian Processes through Variational Inducing Kernels , 2010, AISTATS.

[28]  A. Polyanin Handbook of Linear Partial Differential Equations for Engineers and Scientists , 2001 .

[29]  Noel A Cressie,et al.  Some topics in convolution-based spatial modeling , 2007 .

[30]  Sarvapali D. Ramchurn,et al.  Towards Real-Time Information Processing of Sensor Network Data Using Computationally Efficient Multi-output Gaussian Processes , 2008, 2008 International Conference on Information Processing in Sensor Networks (ipsn 2008).

[31]  J. Reinitz,et al.  Rapid preparation of a panel of polyclonal antibodies to Drosophila segmentation proteins , 1998, Development Genes and Evolution.

[32]  Lennart Ljung,et al.  System Identification: Theory for the User , 1987 .

[33]  P. James McLellan,et al.  Parameter estimation in continuous-time dynamic models using principal differential analysis , 2006, Comput. Chem. Eng..

[34]  Simo Särkkä,et al.  Sequential Inference for Latent Force Models , 2011, UAI.

[35]  C. Rasmussen,et al.  Gaussian Process Priors with Uncertain Inputs - Application to Multiple-Step Ahead Time Series Forecasting , 2002, NIPS.

[36]  Asce Hydraulics Division Review of geostatistics in geohydrology. I: Basic concepts. , 1990 .

[37]  Neil D. Lawrence,et al.  Sparse Convolved Gaussian Processes for Multi-output Regression , 2008, NIPS.

[38]  James O. Ramsay,et al.  Principal differential analysis : Data reduction by differential operators , 1996 .

[39]  Hans-Georg Müller,et al.  Functional Data Analysis , 2016 .

[40]  D Kosman,et al.  Automated assay of gene expression at cellular resolution. , 1998, Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing.

[41]  K. Shanmugan,et al.  Random Signals: Detection, Estimation and Data Analysis , 1988 .

[42]  Phillip Boyle,et al.  Gaussian Processes for Regression and Optimisation , 2007 .

[43]  Keith R. Thompson,et al.  Implementation of gaussian process models for non-linear system identification , 2009 .

[44]  Carl E. Rasmussen,et al.  State-Space Inference and Learning with Gaussian Processes , 2010, AISTATS.

[45]  Leon Glass,et al.  Reverse Engineering the Gap Gene Network of Drosophila melanogaster , 2006, PLoS Comput. Biol..

[46]  null null,et al.  Review of Geostatistics in Geohydrology. I: Basic Concepts , 1990 .

[47]  T. Brubaker,et al.  Nonlinear Parameter Estimation , 1979 .

[48]  Neil D. Lawrence,et al.  Latent Force Models , 2009, AISTATS.

[49]  Multivariate Geostatistics , 2004 .

[50]  Barak A. Pearlmutter,et al.  Filtered Gaussian Processes for Learning with Large Data-Sets , 2003, European Summer School on Multi-AgentControl.

[51]  T. J. Mitchell,et al.  Bayesian design and analysis of computer experiments: Use of derivatives in surface prediction , 1993 .

[52]  M. Barenco,et al.  Fitting ordinary differential equations to short time course data , 2008, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[53]  Neil D. Lawrence,et al.  Fast Forward Selection to Speed Up Sparse Gaussian Process Regression , 2003, AISTATS.

[54]  Max D. Morris,et al.  Asymptotically optimum experimental designs for prediction of deterministic functions given derivative information , 1994 .

[55]  Simo Särkkä,et al.  State-Space Inference for Non-Linear Latent Force Models with Application to Satellite Orbit Prediction , 2012, ICML.

[56]  Zoubin Ghahramani,et al.  Sparse Gaussian Processes using Pseudo-inputs , 2005, NIPS.

[57]  Barak A. Pearlmutter,et al.  Transformations of Gaussian Process Priors , 2004, Deterministic and Statistical Methods in Machine Learning.

[58]  Bernhard Schölkopf,et al.  Switched Latent Force Models for Movement Segmentation , 2010, NIPS.

[59]  Neil D. Lawrence,et al.  Kernels for Vector-Valued Functions: a Review , 2011, Found. Trends Mach. Learn..

[60]  Edwin V. Bonilla,et al.  Multi-task Gaussian Process Prediction , 2007, NIPS.

[61]  Dieter Fox,et al.  GP-UKF: Unscented kalman filters with Gaussian process prediction and observation models , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[62]  A. G. Butkovskiy,et al.  Characteristics of distributed-parameter systems , 1993 .

[63]  Manu,et al.  Characterization of the Drosophila segment determination morphome. , 2008, Developmental biology.

[64]  C. Anderson,et al.  Quantitative Methods for Current Environmental Issues , 2005 .

[65]  Robert Haining,et al.  Statistics for spatial data: by Noel Cressie, 1991, John Wiley & Sons, New York, 900 p., ISBN 0-471-84336-9, US $89.95 , 1993 .

[66]  Asce Hydraulics Division Review of geostatistics in geohydrology II. Applications. , 1990 .

[67]  Carl E. Rasmussen,et al.  Derivative Observations in Gaussian Process Models of Dynamic Systems , 2002, NIPS.

[68]  Neil D. Lawrence,et al.  Accelerating Bayesian Inference over Nonlinear Differential Equations with Gaussian Processes , 2008, NIPS.

[69]  Catherine A. Calder,et al.  A dynamic process convolution approach to modeling ambient particulate matter concentrations , 2008 .

[70]  Neil D. Lawrence,et al.  Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models , 2005, J. Mach. Learn. Res..

[71]  A. O'Hagan,et al.  Bayesian emulation of complex multi-output and dynamic computer models , 2010 .

[72]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[73]  Manfred Opper,et al.  Sparse Representation for Gaussian Process Models , 2000, NIPS.

[74]  Johannes Jaeger,et al.  Pattern formation and nuclear divisions are uncoupled in Drosophila segmentation: comparison of spatially discrete and continuous models , 2004 .

[75]  Neil D. Lawrence,et al.  Gaussian process modelling of latent chemical species: applications to inferring transcription factor activities , 2008, ECCB.

[76]  Roderick Murray-Smith,et al.  Learning with large data sets using filtered {G}aussian Process priors , 2005 .

[77]  J. Chilès,et al.  Geostatistics: Modeling Spatial Uncertainty , 1999 .

[78]  Carl E. Rasmussen,et al.  A Unifying View of Sparse Approximate Gaussian Process Regression , 2005, J. Mach. Learn. Res..

[79]  Neil D. Lawrence,et al.  Modelling transcriptional regulation using Gaussian Processes , 2006, NIPS.

[80]  Catherine A. Calder,et al.  Dynamic factor process convolution models for multivariate space–time data with application to air quality assessment , 2007, Environmental and Ecological Statistics.