RobULA: Efficient Sampling for Robust Bayesian Inference

We study the problem of robustly estimating the posterior distribution for the setting where observed data can be contaminated with potentially adversarial outliers. We propose RobULA, a robust variant of the Unadjusted Langevin Algorithm (ULA), and provide a finite-sample analysis of its sampling distribution. The key to our approach is combining a gradient-based sampling approach with a robust mean gradient estimator. In particular, we show that after T = Õ(d/εacc) iterations, we can sample from pT such that dist(pT , p∗) ≤ εacc + Õ( ), where is the fraction of corruptions.

[1]  Pradeep Ravikumar,et al.  Adaptive Hard Thresholding for Near-optimal Consistent Robust Regression , 2019, COLT.

[2]  Jerry Li,et al.  Sever: A Robust Meta-Algorithm for Stochastic Optimization , 2018, ICML.

[3]  David B. Dunson,et al.  Robust Bayesian Inference via Coarsening , 2015, Journal of the American Statistical Association.

[4]  Pravesh Kothari,et al.  Efficient Algorithms for Outlier-Robust Regression , 2018, COLT.

[5]  Sivaraman Balakrishnan,et al.  Robust estimation via robust gradient estimation , 2018, Journal of the Royal Statistical Society: Series B (Statistical Methodology).

[6]  Martin J. Wainwright,et al.  Log-concave sampling: Metropolis-Hastings algorithms are fast! , 2018, COLT.

[7]  Peter L. Bartlett,et al.  Convergence of Langevin MCMC in KL-divergence , 2017, ALT.

[8]  Prateek Jain,et al.  Consistent Robust Regression , 2017, NIPS.

[9]  David B. Dunson,et al.  Robust and Scalable Bayes via a Median of Subset Posterior Measures , 2014, J. Mach. Learn. Res..

[10]  Santosh S. Vempala,et al.  Agnostic Estimation of Mean and Covariance , 2016, 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS).

[11]  Daniel M. Kane,et al.  Robust Estimators in High Dimensions without the Computational Intractability , 2016, 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS).

[12]  Prateek Jain,et al.  Robust Regression via Hard Thresholding , 2015, NIPS.

[13]  A. Dalalyan.  Theoretical guarantees for approximate sampling from smooth and log‐concave densities , 2014, 1412.7392.

[14]  T. L. McCluskey,et al.  An assessment of features related to phishing websites using an automated technique , 2012, 2012 International Conference for Internet Technology and Secured Transactions.

[15]  Yee Whye Teh,et al.  Bayesian Learning via Stochastic Gradient Langevin Dynamics , 2011, ICML.

[16]  Radford M. Neal.  MCMC Using Hamiltonian Dynamics , 2011, 1206.1901.

[17]  B. Ripley,et al.  Robust Statistics , 2018, Wiley Series in Probability and Statistics.

[18]  C. Villani,et al.  Generalization of an Inequality by Talagrand and Links with the Logarithmic Sobolev Inequality , 2000 .

[19]  R. Tweedie,et al.  Exponential convergence of Langevin distributions and their discrete approximations , 1996 .

[20]  R. Tweedie,et al.  Rates of convergence of the Hastings and Metropolis algorithms , 1996 .

[21]  James O. Berger,et al.  An overview of robust Bayesian analysis , 1994 .

[22]  Robert L. Smith,et al.  Hit-and-Run Algorithms for Generating Multivariate Distributions , 1993, Math. Oper. Res..

[23]  Miklós Simonovits,et al.  Random Walks in a Convex Body and an Improved Volume Algorithm , 1993, Random Struct. Algorithms.

[24]  Frederick R. Forst,et al.  On robust estimation of the location parameter , 1980 .

[25]  D. Ermak.  A computer simulation of charged particles in solution. I. Technique and equilibrium properties , 1975 .

[26]  P. J. Huber.  Robust Regression: Asymptotics, Conjectures and Monte Carlo , 1973 .

[27]  Bruno De Finetti,et al.  The Bayesian Approach to the Rejection of Outliers , 1961 .

[28]  J. Tukey.  A survey of sampling from contaminated distributions , 1960 .

[29]  G. Box.  NON-NORMALITY AND TESTS ON VARIANCES , 1953 .