Optimization by Max-Propagation Using Kikuchi Approximations

In this paper we address the problem of using region-based approximations to find the optimal points of a given function. Our approach combines the use of Kikuchi approximations with the application of generalized belief propagation (GBP) using maximization instead of marginalization. The relationship between the fixed points of maximum GBP and the free energy is elucidated. A straightforward connection between the function to be optimized and the Kikuchi approximation (which holds only for maximum GBP, not for marginal GBP) is proven. Later, we show that maximum GBP can be combined with a dynamic programming algorithm to find the most probable configurations of a f graphical model. We then analyze the dynamics of the procedure proposed and show how its different steps can be manipulated to influence the search for optimal solutions.

[1]  J. A. Lozano,et al.  Analyzing the PBIL Algorithm by Means of Discrete Dynamical Systems , 2000 .

[2]  Robin Hons,et al.  Estimation of Distribution Algorithms and Minimum Relative Entropy , 2005 .

[3]  Siddhartha Shakya,et al.  Using a Markov network model in a univariate EDA: an empirical cost-benefit analysis , 2005, GECCO '05.

[4]  Vladimir Kolmogorov,et al.  Comparison of Energy Minimization Algorithms for Highly Connected Graphs , 2006, ECCV.

[5]  Ivan Bratko,et al.  Testing the significance of attribute interactions , 2004, ICML.

[6]  Yair Weiss,et al.  Approximate Inference and Protein-Folding , 2002, NIPS.

[7]  Payam Pakzad,et al.  Estimation and Marginalization Using the Kikuchi Approximation Methods , 2005, Neural Computation.

[8]  Martin J. Wainwright,et al.  MAP estimation via agreement on trees: message-passing and linear programming , 2005, IEEE Transactions on Information Theory.

[9]  J. A. Lozano,et al.  Properties of Kikuchi approximations constructed from clique based decompositions , 2022 .

[10]  Frank Jensen,et al.  Optimal junction Trees , 1994, UAI.

[11]  H. Mühlenbein,et al.  From Recombination of Genes to the Estimation of Distributions I. Binary Parameters , 1996, PPSN.

[12]  Yee Whye Teh,et al.  On Improving the Efficiency of the Iterative Proportional Fitting Procedure , 2003, AISTATS.

[13]  Robert J. McEliece,et al.  Belief Propagation on Partially Ordered Sets , 2003, Mathematical Systems Theory in Biology, Communications, Computation, and Finance.

[14]  Yair Weiss,et al.  Correctness of Local Probability Propagation in Graphical Models with Loops , 2000, Neural Computation.

[15]  Tom Heskes,et al.  On the Uniqueness of Loopy Belief Propagation Fixed Points , 2004, Neural Computation.

[16]  Riccardo Zecchina,et al.  Survey propagation: An algorithm for satisfiability , 2002, Random Struct. Algorithms.

[17]  Martin J. Wainwright,et al.  Tree consistency and bounds on the performance of the max-product algorithm and its generalizations , 2004, Stat. Comput..

[18]  R. Kikuchi.  A Theory of Cooperative Phenomena , 1951 .

[19]  Yee Whye Teh,et al.  Bethe free energy and contrastive divergence approximations for undirected graphical models , 2003 .

[20]  Yair Weiss,et al.  Globally optimal solutions for energy minimization in stereo vision using reweighted belief propagation , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[21]  Max Welling,et al.  On the Choice of Regions for Generalized Belief Propagation , 2004, UAI.

[22]  D. Nilsson,et al.  An efficient algorithm for finding the M most probable configurationsin probabilistic expert systems , 1998, Stat. Comput..

[23]  J. A. Lozano,et al.  Towards a New Evolutionary Computation: Advances on Estimation of Distribution Algorithms (Studies in Fuzziness and Soft Computing) , 2006 .

[24]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[25]  A. Pelizzola.  Exactness of the cluster variation method and factorization of the equilibrium probability for the Wako-Saito-Munoz-Eaton model of protein folding , 2005 .

[26]  John N. Tsitsiklis,et al.  Introduction to linear optimization , 1997, Athena scientific optimization and computation series.

[27]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[28]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[29]  Robert J. McEliece,et al.  The generalized distributive law , 2000, IEEE Trans. Inf. Theory.

[30]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[31]  Hans L. Bodlaender,et al.  A Tourist Guide through Treewidth , 1993, Acta Cybern..

[32]  Vladimir Kolmogorov,et al.  Convergent Tree-Reweighted Message Passing for Energy Minimization , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Martin Pelikan,et al.  Scalable Optimization via Probabilistic Modeling: From Algorithms to Applications (Studies in Computational Intelligence) , 2006 .

[34]  Roberto Santana,et al.  Estimation of Distribution Algorithms with Kikuchi Approximations , 2005, Evolutionary Computation.

[35]  Max Henrion,et al.  Propagating uncertainty in bayesian networks by probabilistic logic sampling , 1986, UAI.

[36]  Qingfu Zhang,et al.  On stability of fixed points of limit models of univariate marginal distribution algorithm and factorized distribution algorithm , 2004, IEEE Transactions on Evolutionary Computation.

[37]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[38]  William T. Freeman,et al.  On the optimality of solutions of the max-product belief-propagation algorithm in arbitrary graphs , 2001, IEEE Trans. Inf. Theory.

[39]  Umberto Bertelè,et al.  Nonserial Dynamic Programming , 1972 .

[40]  Steffen L. Lauritzen,et al.  Graphical models in R , 1996 .

[41]  Jung-Fu Cheng,et al.  Turbo Decoding as an Instance of Pearl's "Belief Propagation" Algorithm , 1998, IEEE J. Sel. Areas Commun..

[42]  Y. Weiss,et al.  Finding the M Most Probable Configurations using Loopy Belief Propagation , 2003, NIPS 2003.

[43]  Qingfu Zhang,et al.  On the convergence of a class of estimation of distribution algorithms , 2004, IEEE Transactions on Evolutionary Computation.

[44]  Nanning Zheng,et al.  Stereo Matching Using Belief Propagation , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[45]  Heinz Mühlenbein,et al.  The Estimation of Distributions and the Minimum Relative Entropy Principle , 2005, Evol. Comput..

[46]  Riccardo Zecchina,et al.  Survey and Belief Propagation on Random K-SAT , 2003, SAT.