Approximate factorizations of distributions and the minimum relative entropy principle

Estimation of Distribution Algorithms (EDA) have been proposed as an extension of genetic algorithms. In this paper the major design issues of EDA's are discussed within a general interdisciplinary framework, the maximum entropy approximation. Our EDA algorithm FDA assumes that the function to be optimized is additively decomposed (ADF). The interaction graph GADF is used to create exact or approximate factorizations of the Boltzmann distribution. The relation between FDA factorizations and the MaxEnt solution is shown. We introduce a second algorithm, derived from the Bethe-Kikuchi approach developed in statistical physics. It tries to minimize the Kullback-Leibler divergence KLD(q\pβ) to the Boltzmann distribution pβ by solving a difficult constrained optimization problem. We present in detail the concave-convex minimization algorithm CCCP to solve the optimization problem. The two algorithms are compared using popular benchmark problems (2-d grid problems, 2-d Ising spin glasses, Kaufman's n — k function.) We use instances up to 900 variables.

[1]  Heinz Mühlenbein,et al.  Evolutionary Algorithms and the Boltzmann Distribution , 2002, FOGA.

[2]  Robert J. McEliece,et al.  Belief Propagation on Partially Ordered Sets , 2003, Mathematical Systems Theory in Biology, Communications, Computation, and Finance.

[3]  Umberto Bertelè,et al.  Nonserial Dynamic Programming , 1972 .

[4]  Russell G. Almond.  Graphical belief modeling , 1995 .

[5]  David E. Goldberg,et al.  Hierarchical BOA Solves Ising Spin Glasses and MAXSAT , 2003, GECCO.

[6]  Yang Xiang,et al.  A “Microscopic” Study of Minimum Entropy Search in Learning Decomposable Markov Networks , 2004, Machine Learning.

[7]  Heinz Mühlenbein,et al.  Evolutionary optimization and the estimation of search distributions with applications to graph bipartitioning , 2002, Int. J. Approx. Reason..

[8]  Y. Weiss,et al.  Finding the M Most Probable Configurations using Loopy Belief Propagation , 2003, NIPS 2003.

[9]  Roberto Santana,et al.  Estimation of Distribution Algorithms with Kikuchi Approximations , 2005, Evolutionary Computation.

[10]  Frank Jensen,et al.  Optimal junction Trees , 1994, UAI.

[11]  J. Davenport.  Editor , 1960 .

[12]  T. Mahnig,et al.  Mathematical Analysis of Evolutionary Algorithms , 2002 .

[13]  Heinz Mühlenbein,et al.  Schemata, Distributions and Graphical Models in Evolutionary Optimization , 1999, J. Heuristics.

[14]  H. Mühlenbein,et al.  From Recombination of Genes to the Estimation of Distributions I. Binary Parameters , 1996, PPSN.

[15]  E. T. Jaynes,et al.  Where do we Stand on Maximum Entropy , 1979 .

[16]  Robin Hons,et al.  Estimation of Distribution Algorithms and Minimum Relative Entropy , 2005 .

[17]  U. Montanari,et al.  Nonserial Dynamic Programming: On the Optimal Strategy of Variable Elimination for the Rectangular Lattice , 1972 .

[18]  Heinz Mühlenbein,et al.  The Estimation of Distributions and the Minimum Relative Entropy Principle , 2005, Evol. Comput..

[19]  Alan L. Yuille,et al.  CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation , 2002, Neural Computation.

[20]  S. Aji,et al.  The Generalized Distributive Law and Free Energy Minimization , 2001 .

[21]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[22]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[23]  Yee Whye Teh,et al.  On Improving the Efficiency of the Iterative Proportional Fitting Procedure , 2003, AISTATS.

[24]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[25]  Steffen L. Lauritzen,et al.  Graphical models in R , 1996 .

[26]  Yong Gao,et al.  Space Complexity of Estimation of Distribution Algorithms , 2005, Evolutionary Computation.

[27]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[28]  Heinz Mühlenbein,et al.  FDA -A Scalable Evolutionary Algorithm for the Optimization of Additively Decomposed Functions , 1999, Evolutionary Computation.