Conditional mean field

Despite all the attention paid to variational methods based on sum-product message passing (loopy belief propagation, tree-reweighted sum-product), these methods are still bound to inference on a small set of probabilistic models. Mean field approximations have been applied to a broader set of problems, but the solutions are often poor. We propose a new class of conditionally-specified variational approximations based on mean field theory. While not usable on their own, combined with sequential Monte Carlo they produce guaranteed improvements over conventional mean field. Moreover, experiments on a well-studied problem— inferring the stable configurations of the Ising spin glass—show that the solutions can be significantly better than those obtained using sum-product-based methods.

[1]  Mark Jerrum,et al.  The Markov chain Monte Carlo method: an approach to approximate counting and integration , 1996 .

[2]  Christian P. Robert,et al.  Monte Carlo Statistical Methods (Springer Texts in Statistics) , 2005 .

[3]  Nando de Freitas,et al.  Variational MCMC , 2001, UAI.

[4]  Radford M. Neal.  Annealed importance sampling , 1998, Stat. Comput..

[5]  Michael I. Jordan,et al.  Exploiting Tractable Substructures in Intractable Networks , 1995, NIPS.

[6]  A. W. Rosenbluth,et al.  MONTE CARLO CALCULATION OF THE AVERAGE EXTENSION OF MOLECULAR CHAINS , 1955 .

[7]  Christian P. Robert,et al.  Monte Carlo Statistical Methods , 2005, Springer Texts in Statistics.

[8]  J.S. Sadowsky,et al.  On large deviations theory and asymptotically efficient Monte Carlo estimation , 1990, IEEE Trans. Inf. Theory.

[9]  J. Besag.  Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .

[10]  Nando de Freitas,et al.  Hot Coupling: A Particle Approach to Inference and Normalization on Pairwise Undirected Graphs , 2005, NIPS.

[11]  B. Arnold,et al.  Conditional specification of statistical models , 1999 .

[12]  Michael I. Jordan.  Graphical Models , 1998 .

[13]  Hilbert J. Kappen,et al.  Approximate Inference and Constrained Optimization , 2002, UAI.

[14]  Gerard T. Barkema,et al.  Monte Carlo Methods in Statistical Physics , 1999 .

[15]  S. Aji,et al.  The Generalized Distributive Law and Free Energy Minimization , 2001 .

[16]  Aleks Jakulin,et al.  Applying Discrete PCA in Data Analysis , 2004, UAI.

[17]  Michael I. Jordan,et al.  Mean Field Theory for Sigmoid Belief Networks , 1996, J. Artif. Intell. Res..

[18]  G. Kitagawa.  Monte Carlo Filter and Smoother for Non-Gaussian Nonlinear State Space Models , 1996 .

[19]  Zoubin Ghahramani,et al.  Variational Inference for Bayesian Mixtures of Factor Analysers , 1999, NIPS.

[20]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[21]  P. Moral,et al.  Sequential Monte Carlo samplers , 2002, cond-mat/0212648.

[22]  Nando de Freitas,et al.  A Blessing of Dimensionality: Measure Concentration and Probabilistic Inference , 2003, AISTATS.

[23]  Martin J. Wainwright,et al.  A new class of upper bounds on the log partition function , 2002, IEEE Transactions on Information Theory.

[24]  William T. Freeman,et al.  Nonparametric belief propagation , 2003, 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings..

[25]  M. Opper,et al.  Advanced mean field methods: theory and practice , 2001 .

[26]  C. Jarzynski.  Nonequilibrium Equality for Free Energy Differences , 1996, cond-mat/9610209.

[27]  Wim Wiegerinck,et al.  Variational Approximations between Mean Field Theory and the Junction Tree Algorithm , 2000, UAI.