Continuous Estimation of Distribution Algorithms Based on Factorized Gaussian Markov Networks

Because of their intrinsic properties, the majority of the estimation of distribution algorithms proposed for continuous optimization problems are based on the Gaussian distribution assumption for the variables. This paper looks over the relation between the general multivariate Gaussian distribution and the popular undirected graphical model of Markov networks and discusses how they can be employed in estimation of distribution algorithms for continuous optimization. A number of learning and sampling techniques for thesemodels, including the promising regularized model learning, are also reviewed and their application for function optimization in the context of estimation of distribution algorithms is studied.

[1]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[2]  N. Meinshausen,et al.  High-dimensional graphs and variable selection with the Lasso , 2006, math/0608017.

[3]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[4]  Chi-Keong Goh,et al.  Computational Intelligence in Expensive Optimization Problems , 2010 .

[5]  Hsiao-Ping Hsu,et al.  Structure optimization in an off-lattice protein model. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[6]  Robin Hons,et al.  Estimation of Distribution Algorithms and Minimum Relative Entropy , 2005 .

[7]  Alberto Ochoa,et al.  Opportunities for Expensive Optimization with Estimation of Distribution Algorithms , 2010 .

[8]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[9]  Xin Yao,et al.  Parallel Problem Solving from Nature PPSN VI , 2000, Lecture Notes in Computer Science.

[10]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[11]  Max Henrion,et al.  Propagating uncertainty in bayesian networks by probabilistic logic sampling , 1986, UAI.

[12]  Siddhartha Shakya,et al.  A Markovianity based optimisation algorithm , 2012, Genetic Programming and Evolvable Machines.

[13]  Hans-Paul Schwefel,et al.  Parallel Problem Solving from Nature — PPSN IV , 1996, Lecture Notes in Computer Science.

[14]  T. Hesterberg,et al.  Least angle and ℓ1 penalized regression: A review , 2008, 0802.0964.

[15]  David E. Goldberg,et al.  Real-Coded Extended Compact Genetic Algorithm Based on Mixtures of Models , 2008, Linkage in Evolutionary Computation.

[16]  Peter Tiño,et al.  Scaling Up Estimation of Distribution Algorithms for Continuous Optimization , 2011, IEEE Transactions on Evolutionary Computation.

[17]  Dirk Thierens,et al.  Numerical Optimization with Real-Valued Estimation-of-Distribution Algorithms , 2006, Scalable Optimization via Probabilistic Modeling.

[18]  Susan A. Murphy,et al.  Monographs on statistics and applied probability , 1990 .

[19]  J. A. Hartigan,et al.  A k-means clustering algorithm , 1979 .

[20]  Pankaj K. Agarwal,et al.  Exact and Approximation Algortihms for Clustering , 1997 .

[21]  Pedro Larrañaga,et al.  Learning Factorizations in Estimation of Distribution Algorithms Using Affinity Propagation , 2010, Evolutionary Computation.

[22]  Pedro Larrañaga,et al.  Estimation of Distribution Algorithms , 2002, Genetic Algorithms and Evolutionary Computation.

[23]  H. Mühlenbein,et al.  From Recombination of Genes to the Estimation of Distributions I. Binary Parameters , 1996, PPSN.

[24]  David E. Goldberg,et al.  Getting the best of both worlds: Discrete and continuous genetic and evolutionary algorithms in concert , 2003, Inf. Sci..

[25]  Bin Yu,et al.  Model Selection in Gaussian Graphical Models: High-Dimensional Consistency of boldmathell_1-regularized MLE , 2008, NIPS 2008.

[26]  Leonhard Held,et al.  Gaussian Markov Random Fields: Theory and Applications , 2005 .

[27]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[28]  T. Speed,et al.  Gaussian Markov Distributions over Finite Graphs , 1986 .

[29]  Dirk Thierens,et al.  Expanding from Discrete to Continuous Estimation of Distribution Algorithms: The IDEA , 2000, PPSN.

[30]  Nir Friedman,et al.  Probabilistic Graphical Models: Principles and Techniques - Adaptive Computation and Machine Learning , 2009 .

[31]  Maurice K. Wong,et al.  Algorithm AS136: A k-means clustering algorithm. , 1979 .

[32]  R. Tibshirani,et al.  Covariance‐regularized regression and classification for high dimensional problems , 2009, Journal of the Royal Statistical Society. Series B, Statistical methodology.

[33]  David E. Goldberg,et al.  A Simple Real-Coded Extended Compact Genetic Algorithm , 2007, 2007 IEEE Congress on Evolutionary Computation.

[34]  Hendrik Blockeel,et al.  Machine Learning: ECML 2003 , 2003, Lecture Notes in Computer Science.

[35]  Jinglu Hu,et al.  A novel clustering based niching EDA for protein folding , 2009, 2009 World Congress on Nature & Biologically Inspired Computing (NaBIC).

[36]  K. Dill.  Theory for the folding and stability of globular proteins. , 1985, Biochemistry.

[37]  Pedro Larrañaga,et al.  Optimization in Continuous Domains by Learning and Simulation of Gaussian Networks , 2000 .

[38]  Matteo Matteucci,et al.  Introducing ℓ1-regularized logistic regression in Markov Networks based EDAs , 2011, 2011 IEEE Congress of Evolutionary Computation (CEC).

[39]  Martin Pelikan,et al.  Scalable Optimization via Probabilistic Modeling: From Algorithms to Applications (Studies in Computational Intelligence) , 2006 .

[40]  A. E. Hoerl,et al.  Ridge Regression: Applications to Nonorthogonal Problems , 1970 .

[41]  K. Strimmer,et al.  Statistical Applications in Genetics and Molecular Biology A Shrinkage Approach to Large-Scale Covariance Matrix Estimation and Implications for Functional Genomics , 2011 .

[42]  David E. Goldberg,et al.  Real-coded ECGA for solving decomposable real-valued optimization problems , 2007, 2007 IEEE Congress on Evolutionary Computation.

[43]  Siddhartha Shakya,et al.  Optimization by estimation of distribution with DEUM framework based on Markov random fields , 2007, Int. J. Autom. Comput..

[44]  M. Yuan,et al.  Model selection and estimation in the Gaussian graphical model , 2007 .

[45]  Hsiao-Ping Hsu,et al.  Growth-based optimization algorithm for lattice heteropolymers. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[46]  Marcus Gallagher,et al.  Real-valued Evolutionary Optimization using a Flexible Probability Density Estimator , 1999, GECCO.

[47]  Riccardo Poli,et al.  Genetic and Evolutionary Computation – GECCO 2004 , 2004, Lecture Notes in Computer Science.

[48]  R. Tibshirani,et al.  Sparse inverse covariance estimation with the graphical lasso. , 2008, Biostatistics.

[49]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  David E. Goldberg,et al.  Real-Coded Bayesian Optimization Algorithm: Bringing the Strength of BOA into the Continuous World , 2004, GECCO.

[51]  Roberto Santana.  A Markov Network Based Factorized Distribution Algorithm for Optimization , 2003, ECML.

[52]  Pedro Larrañaga,et al.  Experimental Results in Function Optimization with EDAs in Continuous Domain , 2002, Estimation of Distribution Algorithms.

[53]  Head-Gordon,et al.  Toy model for protein folding. , 1993, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[54]  Thomas G. Dietterich.  Adaptive computation and machine learning , 1998 .

[55]  Petros Koumoutsakos,et al.  Learning probability distributions in continuous evolutionary algorithms – a comparative review , 2004, Natural Computing.

[56]  Delbert Dueck,et al.  Clustering by Passing Messages Between Data Points , 2007, Science.

[57]  Concha Bielza,et al.  Regularized k-order markov models in EDAs , 2011, GECCO '11.