A Probabilistic Reformulation of No Free Lunch: Continuous Lunches Are Not Free

No Free Lunch (NFL) theorems have been developed in many settings over the last two decades. Whereas NFL is known to be possible in any domain based on set-theoretic concepts, probabilistic versions of NFL are presently believed to be impossible in continuous domains. This article develops a new formalization of probabilistic NFL that is sufficiently expressive to prove the existence of NFL in large search domains, such as continuous spaces or function spaces. This formulation is arguably more complicated than its set-theoretic variants, mostly as a result of the numerous technical complications within probability theory itself. However, a probabilistic conceptualization of NFL is important because stochastic optimization methods inherently need to be evaluated probabilistically. Thus the present study fills an important gap in the study of performance of stochastic optimizers.

[1]  Alan J. Lockett.  Insights From Adversarial Fitness Functions , 2015, FOGA.

[2]  Kim C. Border,et al.  Infinite Dimensional Analysis: A Hitchhiker’s Guide , 1994 .

[3]  Michael Koller.  An Introduction to Stochastic Integration , 2011 .

[4]  Michael D. Vose,et al.  No Free Lunch and Benchmarks , 2013, Evolutionary Computation.

[5]  Jan Mařík.  The Baire and Borel measure , 1957 .

[6]  Marc Toussaint,et al.  A No-Free-Lunch Theorem for Non-Uniform Distributions of Target Functions , 2004 .

[7]  Risto Miikkulainen,et al.  A measure-theoretic analysis of stochastic optimization , 2013, FOGA XII '13.

[8]  Gábor Lugosi,et al.  Introduction to Statistical Learning Theory , 2004, Advanced Lectures on Machine Learning.

[9]  F. W. Roush,et al.  Introduction to stochastic integration: K.L. Chung and R.J. Williams, Boston: Birkhauser, 1983, 191 pages, $19.95 , 1984 .

[10]  J. Norris.  Appendix: probability and measure , 1997 .

[11]  D. Wolpert,et al.  No Free Lunch Theorems for Search , 1995 .

[12]  Michael D. Vose,et al.  Black box search: framework and methods , 2000 .

[13]  Thomas M. English,et al.  Evaluation of Evolutionary and Genetic Optimizers: No Free Lunch , 1996, Evolutionary Programming.

[14]  Patrick D. Surry,et al.  Fundamental Limitations on Search Algorithms: Evolutionary Computing in Perspective , 1995, Computer Science Today.

[15]  Michael D. Vose,et al.  Random Heuristic Search , 1999, Theor. Comput. Sci..

[16]  T. M. English.  Optimization is easy and learning is hard in the typical function , 2000, Proceedings of the 2000 Congress on Evolutionary Computation. CEC00 (Cat. No.00TH8512).

[17]  Alan S. Lockett.  Measure-theoretic analysis of performance in evolutionary algorithms , 2013, 2013 IEEE Congress on Evolutionary Computation.

[18]  Ioannis Karatzas,et al.  Brownian Motion and Stochastic Calculus , 1987 .

[19]  Warren B. Powell,et al.  Optimal Learning , 2022, Encyclopedia of Machine Learning and Data Mining.

[20]  Thomas Jansen,et al.  Optimization with randomized search heuristics - the (A)NFL theorem, realistic scenarios, and difficult functions , 2002, Theor. Comput. Sci..

[21]  L. D. Whitley,et al.  The No Free Lunch and problem description length , 2001 .

[22]  Ying-ping Chen,et al.  Free lunches on the discrete Lipschitz class , 2011, Theor. Comput. Sci..

[23]  J A F MAUfK.  THE BAIRE AND BOREL MEASURE , 2016 .

[25]  Alan J. Lockett.  Model-optimal optimization by solving bellman equations , 2014, GECCO.

[26]  Olivier Teytaud,et al.  Continuous lunches are free! , 2007, GECCO '07.

[27]  P. Spreij.  Probability and Measure , 1996 .

[28]  Christian Igel,et al.  No Free Lunch Theorems: Limitations and Perspectives of Metaheuristics , 2014, Theory and Principled Methods for the Design of Metaheuristics.

[29]  Joseph C. Culberson,et al.  On the Futility of Blind Search: An Algorithmic View of No Free Lunch , 1998, Evolutionary Computation.

[30]  Warren B. Powell,et al.  Optimal Learning: Powell/Optimal , 2012 .

[31]  David H. Wolpert,et al.  No free lunch theorems for optimization , 1997, IEEE Trans. Evol. Comput..

[32]  A. Kolmogoroff.  Grundbegriffe der Wahrscheinlichkeitsrechnung , 1933 .

[33]  Olivier Teytaud,et al.  Continuous Lunches Are Free Plus the Design of Optimal Optimization Algorithms , 2010, Algorithmica.