Empirical evaluation of bayesian optimization in parametric tuning of chaotic systems

In this work, we consider the Bayesian optimization (BO) approach for parametric tuning of complex chaotic systems. Such problems arise, for instance, in tuning the sub-grid scale parameterizations in weather and climate models. For such problems, the tuning procedure is generally based on a performance metric which measures how well the tuned model fits the data. This tuning is often a computationally expensive task. We show that BO, as a tool for finding the extrema of computationally expensive objective functions, is suitable for such tuning tasks. In the experiments, we consider tuning parameters of two systems: a simplified atmospheric model and a low-dimensional chaotic system. We show that BO is able to tune parameters of both the systems with a low number of objective function evaluations.

[1]  Antti Solonen,et al.  An approach for tuning ensemble prediction systems , 2013 .

[2]  Heikki Haario,et al.  Efficient MCMC for Climate Model Parameter Estimation: Parallel Adaptive Chains and Early Rejection , 2012 .

[3]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[4]  Nando de Freitas,et al.  Portfolio Allocation for Bayesian Optimization , 2010, UAI.

[5]  Nikolaos V. Sahinidis,et al.  Derivative-free optimization: a review of algorithms and comparison of software implementations , 2013, J. Glob. Optim..

[6]  Michael A. Osborne,et al.  Gaussian Processes for Global Optimization , 2008 .

[7]  C. Nicolis,et al.  Lyapunov Vectors and Error Growth Patterns in a T21L3 Quasigeostrophic Model , 1997 .

[8]  Donald R. Jones,et al.  Efficient Global Optimization of Expensive Black-Box Functions , 1998, J. Glob. Optim..

[9]  J D Annan,et al.  Efficient estimation and ensemble generation in climate modelling , 2007, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[10]  Nikolaus Hansen,et al.  Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adaptation , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[11]  Harold J. Kushner,et al.  A New Method of Locating the Maximum Point of an Arbitrary Multipeak Curve in the Presence of Noise , 1964 .

[12]  Sudipto Banerjee,et al.  On Geodetic Distance Computations in Spatial Modeling , 2005, Biometrics.

[13]  Alexander Ilin,et al.  On closure parameter estimation in chaotic systems , 2012 .

[14]  Jonas Mockus,et al.  On Bayesian Methods for Seeking the Extremum , 1974, Optimization Techniques.

[15]  G. Evensen Data Assimilation: The Ensemble Kalman Filter , 2006 .

[16]  T. Gneiting Strictly and non-strictly positive definite functions on spheres , 2011, 1111.7077.

[17]  Robert Pincus,et al.  Parameter estimation using data assimilation in an atmospheric general circulation model: From a perfect toward the real world , 2013 .

[18]  Nando de Freitas,et al.  A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning , 2010, ArXiv.

[19]  Petros Koumoutsakos,et al.  A Method for Handling Uncertainty in Evolutionary Optimization With an Application to Feedback Control of Combustion , 2009, IEEE Transactions on Evolutionary Computation.

[20]  Phillip Boyle,et al.  Gaussian Processes for Regression and Optimisation , 2007 .

[21]  James C McWilliams,et al.  Considerations for parameter optimization and sensitivity in climate models , 2010, Proceedings of the National Academy of Sciences.

[22]  Nando de Freitas,et al.  A Bayesian interactive optimization approach to procedural animation design , 2010, SCA '10.

[23]  Y. Marzouk,et al.  A stochastic collocation approach to Bayesian inference in inverse problems , 2009 .

[24]  J. Mockus Bayesian Approach to Global Optimization: Theory and Applications , 1989 .

[25]  C. D. Perttunen,et al.  Lipschitzian optimization without the Lipschitz constant , 1993 .

[26]  Kevin Leyton-Brown,et al.  Bayesian Optimization With Censored Response Data , 2013, ArXiv.

[27]  Alexander Ilin,et al.  A dilemma of the uniqueness of weather and climate model closure parameters , 2013 .

[28]  N. Zheng,et al.  Global Optimization of Stochastic Black-Box Systems via Sequential Kriging Meta-Models , 2006, J. Glob. Optim..

[29]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[30]  Marcus R. Frean,et al.  Using Gaussian Processes to Optimize Expensive Functions , 2008, Australasian Conference on Artificial Intelligence.

[31]  Derek J. Pike,et al.  Empirical Model‐building and Response Surfaces. , 1988 .

[32]  D. Lizotte,et al.  An experimental methodology for response surface optimization methods , 2012, J. Glob. Optim..

[33]  D. Lizotte Practical bayesian optimization , 2008 .

[34]  Nikolaus Hansen,et al.  Completely Derandomized Self-Adaptation in Evolution Strategies , 2001, Evolutionary Computation.

[35]  Andrew Keats,et al.  Artificial neural network assisted Bayesian calibration of climate models , 2012, Climate Dynamics.

[36]  Donald R. Jones,et al.  A Taxonomy of Global Optimization Methods Based on Response Surfaces , 2001, J. Glob. Optim..

[37]  D. Wilks Effects of stochastic parametrizations in the Lorenz '96 system , 2005 .

[38]  Erkki Oja,et al.  Estimation of ECHAM5 climate model closure parameters with adaptive MCMC , 2010 .