Stochastic Reasoning, Free Energy, and Information Geometry

Belief propagation (BP) is a universal method of stochastic reasoning. It gives exact inference for stochastic models with tree interactions and works surprisingly well even if the models have loopy interactions. Its performance has been analyzed separately in many fields, such as AI, statistical physics, information theory, and information geometry. This article gives a unified framework for understanding BP and related methods and summarizes the results obtained in many fields. In particular, BP and its variants, including tree reparameterization and concave-convex procedure, are reformulated with information-geometrical terms, and their relations to the free energy function are elucidated from an information-geometrical viewpoint. We then propose a family of new algorithms. The stabilities of the algorithms are analyzed, and methods to accelerate them are investigated.

[1]  William H. Press,et al.  Numerical Recipes in C The Art of Scientific Computing , 1995 .

[2]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[3]  C. Itzykson,et al.  Statistical Field Theory , 1989 .

[4]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[5]  William H. Press,et al.  Numerical recipes in C++: the art of scientific computing, 2nd Edition (C++ ed., print. is corrected to software version 2.10) , 1994 .

[6]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[7]  D. Saad,et al.  Statistical mechanics of error-correcting codes , 1999 .

[8]  Toshiyuki Tanaka,et al.  Information Geometry of Mean-Field Approximation , 2000, Neural Computation.

[9]  Yair Weiss,et al.  Correctness of Local Probability Propagation in Graphical Models with Loops , 2000, Neural Computation.

[10]  Shun-ichi Amari,et al.  Methods of information geometry , 2000 .

[11]  Thomas J. Richardson,et al.  The geometry of turbo-decoding dynamics , 2000, IEEE Trans. Inf. Theory.

[12]  W. Freeman,et al.  Generalized Belief Propagation , 2000, NIPS.

[13]  Shun-ichi Amari,et al.  Information Geometrical Framework for Analyzing Belief Propagation Decoder , 2001, NIPS.

[14]  W. Freeman,et al.  Bethe free energy, Kikuchi approximations, and belief propagation algorithms , 2001 .

[15]  M. Opper,et al.  Advanced mean field methods: theory and practice , 2001 .

[16]  Martin J. Wainwright,et al.  Tree-based reparameterization for approximate inference on loopy graphs , 2001, NIPS.

[17]  M. Gribaudo,et al.  2002 , 2001, Cell and Tissue Research.

[18]  D. Saad,et al.  The TAP Approach to Intensive and Extensive Connectivity Systems , 2001 .

[19]  Alan L. Yuille,et al.  The Concave-Convex Procedure (CCCP) , 2001, NIPS.

[20]  Shun-ichi Amari,et al.  Information geometry on hierarchy of probability distributions , 2001, IEEE Trans. Inf. Theory.

[21]  Tom Heskes,et al.  Stable Fixed Points of Loopy Belief Propagation Are Local Minima of the Bethe Free Energy , 2002, NIPS.

[22]  Alan L. Yuille,et al.  CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation , 2002, Neural Computation.

[23]  T. Heskes.  Stable Fixed Points of Loopy Belief Propagation Are Minima of the Bethe Free Energy , 2002 .

[24]  William H. Press,et al.  Numerical recipes in C , 2002 .

[25]  Martin J. Wainwright,et al.  Tree-reweighted belief propagation algorithms and approximate ML estimation by pseudo-moment matching , 2003, AISTATS.

[26]  Alan L. Yuille,et al.  The Concave-Convex Procedure , 2003, Neural Computation.

[27]  Shun-ichi Amari,et al.  Information geometry of turbo and low-density parity-check codes , 2004, IEEE Transactions on Information Theory.

[28]  W. Wiegerinck,et al.  Approximate inference techniques with expectation constraints , 2005 .