The Factorized Distribution Algorithm and the Minimum Relative Entropy Principle

We assume that the function to be optimized is additively decomposed (ADF). Then the interaction graph $G_{ADF}$ can be used to compute exact or approximate factorizations. For many practical problems only approximate factorizations lead to efficient optimization algorithms. The relation between the approximation used by the FDA algorithm and the minimum relative entropy principle is discussed. A new algorithm is presented, derived from the Bethe-Kikuchi approach in statistical physics. It minimizes the relative entropy to a Boltzmann distribution with fixed $eta$. We shortly compare different factorizations and algorithms within the FDA software. We use 2-d Ising spin glass problems and Kaufman's n-k function as examples.

[1]  Heinz Mühlenbein,et al.  The Estimation of Distributions and the Minimum Relative Entropy Principle , 2005, Evol. Comput..

[2]  I. Good,et al.  The Maximum Entropy Formalism. , 1979 .

[3]  Steffen L. Lauritzen,et al.  Graphical models in R , 1996 .

[4]  J. Darroch,et al.  Generalized Iterative Scaling for Log-Linear Models , 1972 .

[5]  Alan L. Yuille,et al.  CCCP Algorithms to Minimize the Bethe and Kikuchi Free Energies: Convergent Alternatives to Belief Propagation , 2002, Neural Computation.

[6]  Robin Höns,et al.  Estimation of distribution algorithms and minimum relative entropy , 2005 .

[7]  Robert J. McEliece,et al.  Belief Propagation on Partially Ordered Sets , 2003, Mathematical Systems Theory in Biology, Communications, Computation, and Finance.

[8]  Yee Whye Teh,et al.  On Improving the Efficiency of the Iterative Proportional Fitting Procedure , 2003, AISTATS.

[9]  S. Aji,et al.  The Generalized Distributive Law and Free Energy Minimization , 2001 .

[10]  William T. Freeman,et al.  Constructing free-energy approximations and generalized belief propagation algorithms , 2005, IEEE Transactions on Information Theory.

[11]  E. T. Jaynes,et al.  Where do we Stand on Maximum Entropy , 1979 .

[12]  Y. Weiss,et al.  Finding the M Most Probable Configurations using Loopy Belief Propagation , 2003, NIPS 2003.

[13]  Russell G. Almond.  Graphical belief modeling , 1995 .

[14]  David E. Goldberg,et al.  Hierarchical BOA Solves Ising Spin Glasses and MAXSAT , 2003, GECCO.

[15]  Heinz Mühlenbein,et al.  Schemata, Distributions and Graphical Models in Evolutionary Optimization , 1999, J. Heuristics.

[16]  Heinz Mühlenbein,et al.  Evolutionary optimization and the estimation of search distributions with applications to graph bipartitioning , 2002, Int. J. Approx. Reason..

[17]  Brian W. Kernighan,et al.  An efficient heuristic procedure for partitioning graphs , 1970, Bell Syst. Tech. J..

[18]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[19]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[20]  Heinz Mühlenbein,et al.  FDA -A Scalable Evolutionary Algorithm for the Optimization of Additively Decomposed Functions , 1999, Evolutionary Computation.

[21]  S. Griffis.  EDITOR , 1997, Journal of Navigation.

[22]  Robert J. McEliece,et al.  The generalized distributive law , 2000, IEEE Trans. Inf. Theory.

[23]  Yong Gao,et al.  Space Complexity of Estimation of Distribution Algorithms , 2005, Evolutionary Computation.

[24]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[25]  Heinz Mühlenbein,et al.  Evolutionary Algorithms and the Boltzmann Distribution , 2002, FOGA.

[26]  H. Mühlenbein,et al.  From Recombination of Genes to the Estimation of Distributions I. Binary Parameters , 1996, PPSN.

[27]  Roberto Santana,et al.  Estimation of Distribution Algorithms with Kikuchi Approximations , 2005, Evolutionary Computation.

[28]  Frank Jensen,et al.  Optimal junction Trees , 1994, UAI.

[29]  Brian Kernighan,et al.  An efficient heuristic for partitioning graphs , 1970 .

[30]  T. Mahnig,et al.  Mathematical Analysis of Evolutionary Algorithms , 2002 .