SEARCH, Blackbox Optimization, And Sample Complexity

The SEARCH (Search Envisioned As Relation and Class Hierarchizing) framework developed elsewhere (Kargupta, 1995; Kargupta and Goldberg, 1995) offered an alternate perspective toward blackbox optimization -- optimization in presence of little domain knowledge. The SEARCH framework investigates the conditions essential for transcending the limits of random enumerative search using a framework developed in terms of relations, classes and partial ordering. This paper presents a summary of some of the main results of that work. A closed form bound on the sample complexity in terms of the cardinality of the relation space, class space, desired quality of the solution and the reliability is presented. This also leads to the identification of the class of order-k delineable problems that can be solved in polynomial sample complexity. These results are applicable to any blackbox search algorithms, including evolutionary optimization techniques.

[1]  George B. Dantzig,et al.  Linear programming and extensions , 1965 .

[2]  David E. Goldberg,et al.  A Genetic Algorithm for Parallel Simulated Annealing , 1992, PPSN.

[3]  S. Gupta.  Selecting and Ordering Populations: A New Statistical Methodology (Jean Dickinson Gibbons, Ingram Olkin and Milton Sobel) , 1982 .

[4]  K. Dejong,et al.  An analysis of the behavior of a class of genetic adaptive systems , 1975 .

[5]  Kalyanmoy Deb,et al.  Messy Genetic Algorithms: Motivation, Analysis, and First Results , 1989, Complex Syst..

[6]  Lawrence Davis,et al.  Genetic Algorithms and Simulated Annealing , 1987 .

[7]  H. Kargupta,et al.  Constrained blackbox optimization: The SEARCH perspective , 1996 .

[8]  Fabio Schoen,et al.  Stochastic techniques for global optimization: A survey of recent advances , 1991, J. Glob. Optim..

[9]  Kalyanmoy Deb,et al.  RapidAccurate Optimization of Difficult Problems Using Fast Messy Genetic Algorithms , 1993, ICGA.

[10]  B. Natarajan.  Machine Learning: A Theoretical Approach , 1992 .

[11]  Ryszard S. Michalski,et al.  A theory and methodology of inductive learning , 1993 .

[12]  Helmut Ratschek,et al.  What can interval analysis do for global optimization? , 1991, Journal of Global Optimization.

[13]  Afonso Ferreira,et al.  BOUNDING THE PROBABILITY OF SUCCESS OF STOCHASTIC METHODS FOR GLOBAL OPTIMIZATION , 1993 .

[14]  David Haussler,et al.  Occam's Razor , 1987, Inf. Process. Lett..

[15]  Francesco Archetti,et al.  A survey on the global optimization problem: General theory and computational approaches , 1984, Ann. Oper. Res..

[16]  K. Deb.  Binary and floating-point function optimization using messy genetic algorithms , 1991 .

[17]  David Haussler,et al.  Quantifying Inductive Bias: AI Learning Algorithms and Valiant's Learning Framework , 1988, Artif. Intell..

[18]  Lashon B. Booker,et al.  Proceedings of the fourth international conference on Genetic algorithms , 1991 .

[19]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[20]  S. Vavasis.  Nonlinear optimization: complexity issues , 1991 .

[21]  Terry Jones,et al.  Fitness Distance Correlation as a Measure of Problem Difficulty for Genetic Algorithms , 1995, ICGA.

[22]  Perttunen,et al.  The rank transformation applied to a multi-univariate method of global optimization , 1989 .

[23]  M. R. Rao,et al.  Combinatorial Optimization , 1992, NATO ASI Series.

[24]  D. Wolpert,et al.  No Free Lunch Theorems for Search , 1995 .

[25]  David Haussler,et al.  Learnability and the Vapnik-Chervonenkis dimension , 1989, JACM.

[26]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[27]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, CACM.

[28]  W. Hart.  Adaptive global optimization with local search , 1994 .

[29]  David J. Sirag,et al.  Toward a Unified Thermoaynamic Genetic Operator , 1987, International Conference on Genetic Algorithms.

[30]  Averill M. Law,et al.  The art and theory of dynamic programming , 1977 .

[31]  Patrick D. Surry,et al.  Fundamental Limitations on Search Algorithms: Evolutionary Computing in Perspective , 1995, Computer Science Today.

[32]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[33]  David E. Goldberg,et al.  A Note on Boltzmann Tournament Selection for Genetic Algorithms and Population-Oriented Simulated Annealing , 1990, Complex Syst..

[34]  Satosi Watanabe,et al.  Knowing and guessing , 1969 .

[35]  Melanie Mitchell,et al.  Relative Building-Block Fitness and the Building Block Hypothesis , 1992, FOGA.

[36]  Lawrence J. Fogel,et al.  Artificial Intelligence through Simulated Evolution , 1966 .

[37]  Alexander H. G. Rinnooy Kan,et al.  Stochastic methods for global optimization , 1984 .

[38]  Dirk Thierens,et al.  Mixing in Genetic Algorithms , 1993, ICGA.

[39]  H. Kargupta.  Search, polynomial complexity, and the fast messy genetic algorithm , 1996 .

[40]  Günter Rudolph,et al.  Massively Parallel Simulated Annealing and Its Relation to Evolutionary Algorithms , 1993, Evolutionary Computation.

[41]  Gerhard W. Dueck,et al.  Threshold accepting: a general purpose optimization algorithm appearing superior to simulated anneal , 1990 .

[42]  Tim Jones.  Evolutionary Algorithms, Fitness Landscapes and Search , 1995 .

[43]  David E. Goldberg,et al.  Alleles, loci and the traveling salesman problem , 1985 .

[44]  J. Mockus,et al.  The Bayesian approach to global optimization , 1989 .

[45]  Gilbert Syswerda,et al.  Uniform Crossover in Genetic Algorithms , 1989, ICGA.