Heavy-Tailed Process Priors for Selective Shrinkage

Heavy-tailed distributions are often used to enhance the robustness of regression and classification methods to outliers in output space. Often, however, we are confronted with "outliers" in input space, which are isolated observations in sparsely populated regions. We show that heavy-tailed stochastic processes (which we construct from Gaussian processes via a copula), can be used to improve robustness of regression and classification estimators to such outliers by selectively shrinking them more strongly in sparse regions than in dense regions. We carry out a theoretical analysis to show that selective shrinkage occurs when the marginals of the heavy-tailed process have sufficiently heavy tails. The analysis is complemented by experiments on biological data which indicate significant improvements of estimates in sparse regions while producing competitive results in dense regions.

[1]  Robert B. Gramacy,et al.  Bayesian Treed Gaussian Process Models With an Application to Computer Modeling , 2007, 0710.4536.

[2]  David X. Li On Default Correlation: A Copula Function Approach , 1999 .

[3]  P. X. Song,et al.  Multivariate Dispersion Models Generated From Gaussian Copula , 2000 .

[4]  Wei Chu,et al.  Gaussian Processes for Ordinal Regression , 2005, J. Mach. Learn. Res..

[5]  Adrian Dobra,et al.  Copula Gaussian Graphical Models * , 2009 .

[6]  A. Dobra,et al.  Copula Gaussian graphical models and their application to modeling functional disability data , 2011, 1108.1680.

[7]  Larry A. Wasserman,et al.  The Nonparanormal: Semiparametric Estimation of High Dimensional Undirected Graphs , 2009, J. Mach. Learn. Res..

[8]  Robert B. Gramacy,et al.  Classification and Categorical Inputs with Treed Gaussian Process Models , 2009, J. Classif..

[9]  Michael I. Jordan,et al.  Regression with input-dependent noise: A Gaussian process treatment , 1998 .

[10]  Nello Cristianini,et al.  Kernel Methods for Pattern Analysis , 2006 .

[11]  Carl E. Rasmussen,et al.  Gaussian processes for machine learning , 2005, Adaptive computation and machine learning.

[12]  A. O'Hagan,et al.  Bayesian inference for non‐stationary spatial covariance structure via spatial deformations , 2003 .

[13]  Carl E. Rasmussen,et al.  Warped Gaussian Processes , 2003, NIPS.

[14]  M. Pitt,et al.  Efficient Bayesian inference for Gaussian copula regression models , 2006 .

[15]  P. Guttorp,et al.  Bayesian estimation of semi‐parametric non‐stationary spatial covariance structures , 2001 .

[16]  Sebastian Jaimungal,et al.  Kernel-Based Copula Processes , 2009, ECML/PKDD.

[17]  R. Nelsen An Introduction to Copulas , 1998 .

[18]  R. Nelsen An Introduction to Copulas (Springer Series in Statistics) , 2006 .