Globally Multimodal Problem Optimization Via an Estimation of Distribution Algorithm Based on Unsupervised Learning of Bayesian Networks

Many optimization problems are what can be called globally multimodal, i.e., they present several global optima. Unfortunately, this is a major source of difficulties for most estimation of distribution algorithms, making their effectiveness and efficiency degrade, due to genetic drift. With the aim of overcoming these drawbacks for discrete globally multimodal problem optimization, this paper introduces and evaluates a new estimation of distribution algorithm based on unsupervised learning of Bayesian networks. We report the satisfactory results of our experiments with symmetrical binary optimization problems.

[1]  Max Henrion,et al.  Propagating uncertainty in bayesian networks by probabilistic logic sampling , 1986, UAI.

[2]  Pedro Larrañaga,et al.  Dimensionality Reduction in Unsupervised Learning of Conditional Gaussian Networks , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Eric Bauer,et al.  Update Rules for Parameter Estimation in Bayesian Networks , 1997, UAI.

[4]  José M. Peña,et al.  On Local Optima in Learning Bayesian Networks , 2003, UAI.

[5]  K. Dejong,et al.  An analysis of the behavior of a class of genetic adaptive systems , 1975 .

[6]  Rajkumar Roy,et al.  Advances in Soft Computing: Engineering Design and Manufacturing , 1998 .

[7]  Kristian Kersting,et al.  Scaled CGEM: A Fast Accelerated EM , 2003, ECML.

[8]  D. E. Goldberg,et al.  Genetic Algorithms in Search , 1989 .

[9]  Nir Friedman,et al.  Being Bayesian About Network Structure. A Bayesian Approach to Structure Discovery in Bayesian Networks , 2004, Machine Learning.

[10]  D. Goldberg,et al.  BOA: the Bayesian optimization algorithm , 1999 .

[11]  Finn V. Jensen,et al.  Bayesian Networks and Decision Graphs , 2001, Statistics for Engineering and Information Science.

[12]  Pedro Larrañaga,et al.  Performance evaluation of compromise conditional Gaussian networks for data clustering , 2001, Int. J. Approx. Reason..

[13]  David E. Goldberg,et al.  Finite Markov Chain Analysis of Genetic Algorithms , 1987, ICGA.

[14]  Pedro Larrañaga,et al.  Learning Recursive Bayesian Multinets for Data Clustering by Means of Constructive Induction , 2002, Machine Learning.

[15]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[16]  Eamonn J. Keogh,et al.  Learning augmented Bayesian classifiers: A comparison of distribution-based and classification-based approaches , 1999, AISTATS.

[17]  Jan Naudts,et al.  The Effect of Spin-Flip Symmetry on the Performance of the Simple GA , 1998, PPSN.

[18]  D. Goldberg,et al.  Compressed introns in a linkage learning genetic algorithm , 1998 .

[19]  David Maxwell Chickering,et al.  Learning Equivalence Classes of Bayesian Network Structures , 1996, UAI.

[20]  Pedro Larrañaga,et al.  Geographical clustering of cancer incidence by means of Bayesian networks and conditional Gaussian networks , 2001, AISTATS.

[21]  C. Van Hoyweghen,et al.  Symmetry in the search space , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[22]  Shumeet Baluja,et al.  A Method for Integrating Genetic Search Based Function Optimization and Competitive Learning , 1994 .

[23]  Gregory F. Cooper,et al.  A Bayesian method for the induction of probabilistic networks from data , 1992, Machine Learning.

[24]  José Manuel Gutiérrez,et al.  Expert Systems and Probabiistic Network Models , 1996 .

[25]  David J. Spiegelhalter,et al.  Probabilistic Networks and Expert Systems , 1999, Information Science and Statistics.

[26]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[27]  José Manuel Peña Palomar On unsupervised learning of bayesian networks and conditional gaussian networks , 2001 .

[28]  Enrique F. Castillo,et al.  Expert Systems and Probabilistic Network Models , 1996, Monographs in Computer Science.

[29]  David E. Goldberg,et al.  Genetic Algorithms, Clustering, and the Breaking of Symmetry , 2000, PPSN.

[30]  David Maxwell Chickering,et al.  Learning Bayesian Networks: The Combination of Knowledge and Statistical Data , 1994, Machine Learning.

[31]  M. Pelikán,et al.  The Bivariate Marginal Distribution Algorithm , 1999 .

[32]  Kalyanmoy Deb,et al.  RapidAccurate Optimization of Difficult Problems Using Fast Messy Genetic Algorithms , 1993, ICGA.

[33]  E. Thorndike On the Organization of Intellect. , 1921 .

[34]  Michael I. Jordan Graphical Models , 1998 .

[35]  Pedro Larrañaga,et al.  An improved Bayesian structural EM algorithm for learning Bayesian networks for clustering , 2000, Pattern Recognit. Lett..

[36]  Michael R. Anderberg,et al.  Cluster Analysis for Applications , 1973 .

[37]  Shumeet Baluja,et al.  Using Optimal Dependency-Trees for Combinational Optimization , 1997, ICML.

[38]  Bo Thiesson,et al.  Learning Mixtures of DAG Models , 1998, UAI.

[39]  H. Mühlenbein,et al.  From Recombination of Genes to the Estimation of Distributions I. Binary Parameters , 1996, PPSN.

[40]  Pedro Larrañaga,et al.  Learning Bayesian network structures by searching for the best ordering with genetic algorithms , 1996, IEEE Trans. Syst. Man Cybern. Part A.

[41]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[42]  Jeffrey Horn,et al.  Finite Markov Chain Analysis of Genetic Algorithms with Niching , 1993, ICGA.

[43]  G. Schwarz Estimating the Dimension of a Model , 1978 .

[44]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[45]  Arthur C. Sanderson,et al.  Multimodal Function Optimization Using Minimal Representation Size Clustering and Its Application to Planning Multipaths , 1997, Evolutionary Computation.

[46]  C. V. Hoyweghen Detecting spin-flip symmetry in optimization problems , 2001 .

[47]  Pedro Larrañaga,et al.  Learning Bayesian networks for clustering by means of constructive induction , 1999, Pattern Recognit. Lett..

[48]  Heinz Mühlenbein,et al.  Schemata, Distributions and Graphical Models in Evolutionary Optimization , 1999, J. Heuristics.

[49]  Michael I. Jordan,et al.  Learning with Mixtures of Trees , 2001, J. Mach. Learn. Res..

[50]  G. Harik Linkage Learning via Probabilistic Modeling in the ECGA , 1999 .

[51]  Nir Friedman,et al.  The Bayesian Structural EM Algorithm , 1998, UAI.

[52]  Vincent Kanade,et al.  Clustering Algorithms , 2021, Wireless RF Energy Transfer in the Massive IoT Era.

[53]  David E. Goldberg,et al.  Bayesian optimization algorithm: from single level to hierarchy , 2002 .

[54]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems , 1988 .

[55]  S. Baluja,et al.  Using Optimal Dependency-Trees for Combinatorial Optimization: Learning the Structure of the Search Space , 1997 .

[56]  Shumeet Baluja,et al.  Fast Probabilistic Modeling for Combinatorial Optimization , 1998, AAAI/IAAI.

[57]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[58]  Ingo Rechenberg,et al.  Evolutionsstrategie : Optimierung technischer Systeme nach Prinzipien der biologischen Evolution , 1973 .

[59]  Bo Thiesson,et al.  Learning Mixtures of Bayesian Networks , 1997, UAI 1997.

[60]  Pedro Larrañaga,et al.  Combinatonal Optimization by Learning and Simulation of Bayesian Networks , 2000, UAI.

[61]  Morgan B Kaufmann,et al.  Finite Markov Chain Analysis of Genetic Algorithms with Niching , 1993 .

[62]  Jiri Ocenasek,et al.  EXPERIMENTAL STUDY: HYPERGRAPH PARTITIONING BASED ON THE SIMPLE AND ADVANCED GENETIC ALGORITHM BMDA AND BOA , 2002 .

[63]  David E. Goldberg,et al.  Bayesian optimization algorithm, decision graphs, and Occam's razor , 2001 .

[64]  David E. Goldberg,et al.  A Survey of Optimization by Building and Using Probabilistic Models , 2002, Comput. Optim. Appl..

[65]  Heinz Mühlenbein,et al.  The Equation for Response to Selection and Its Use for Prediction , 1997, Evolutionary Computation.

[66]  Martin Pelikan,et al.  Hill Climbing with Learning (An Abstraction of Genetic Algorithm) , 1995 .

[67]  David Maxwell Chickering,et al.  Learning Bayesian Networks is NP-Complete , 2016, AISTATS.

[68]  Marcus Gallagher,et al.  Real-valued Evolutionary Optimization using a Flexible Probability Density Estimator , 1999, GECCO.

[69]  Steffen L. Lauritzen,et al.  Bayesian updating in causal probabilistic networks by local computations , 1990 .

[70]  Paul A. Viola,et al.  MIMIC: Finding Optima by Estimating Probability Densities , 1996, NIPS.

[71]  David E. Goldberg,et al.  Genetic Algorithms with Sharing for Multimodalfunction Optimization , 1987, ICGA.

[72]  J. A. Lozano,et al.  Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation , 2001 .

[73]  Nir Friedman,et al.  Bayesian Network Classifiers , 1997, Machine Learning.

[74]  Stuart J. Russell,et al.  Adaptive Probabilistic Networks with Hidden Variables , 1997, Machine Learning.

[75]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[76]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[77]  Geoffrey J. McLachlan,et al.  Mixture models : inference and applications to clustering , 1989 .

[78]  G. McLachlan,et al.  The EM algorithm and extensions , 1996 .