An Investigation of Reinforcement Learning for Reactive Search Optimization

Reactive Search Optimization advocates the adoption of learning mechanisms as an integral part of a heuristic optimization scheme. This work studies reinforcement learning methods for the online tuning of parameters in stochastic local search algorithms. In particular, the reactive tuning is obtained by learning a (near-)optimal policy in a Markov decision process where the states summarize relevant information about the recent history of the search. The learning process is performed by the Least Squares Policy Iteration (LSPI) method. The proposed framework is applied for tuning the prohibition value in the Reactive Tabu Search, the noise parameter in the Adaptive Walksat, and the smoothing probability in the Reactive Scaling and Probabilistic Smoothing (RSAPS) algorithm. The novel approach is experimentally compared with the original ad hoc. reactive schemes.

[1]  Alan Bundy,et al.  Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence - IJCAI-95 , 1995 .

[2]  Bart Selman,et al.  Noise Strategies for Improving Local Search , 1994, AAAI.

[3]  Holger H. Hoos,et al.  Scaling and Probabilistic Smoothing: Efficient Dynamic Local Search for SAT , 2002, CP.

[4]  Andrew W. Moore,et al.  Learning evaluation functions for global optimization , 1998 .

[5]  Teofilo F. Gonzalez,et al.  Reactive Search: Machine Learning for Memory-Based Heuristics , 2007 .

[6]  Anton Schwartz,et al.  A Reinforcement Learning Method for Maximizing Undiscounted Rewards , 1993, ICML.

[7]  S.D. Muller,et al.  Step size adaptation in evolution strategies using reinforcement learning , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[8]  Thomas Stützle,et al.  Stochastic Local Search: Foundations & Applications , 2004 .

[10]  Thomas G. Dietterich,et al.  High-Performance Job-Shop Scheduling With A Time-Delay TD(λ) Network , 1995, NIPS 1995.

[11]  Steven David Prestwich,et al.  Tuning Local Search by Average-Reward Reinforcement Learning , 2008, LION.

[12]  Roberto Battiti,et al.  The Reactive Tabu Search , 1994, INFORMS J. Comput..

[13]  Michail G. Lagoudakis,et al.  Learning to Select Branching Rules in the DPLL Procedure for Satisfiability , 2001, Electron. Notes Discret. Math..

[14]  Mauro Brunato,et al.  Reactive Search and Intelligent Optimization , 2008 .

[15]  Susan L. Epstein,et al.  LEARNING TO SUPPORT CONSTRAINT PROGRAMMERS , 2005, Comput. Intell..

[16]  John N. Tsitsiklis,et al.  Neuro-Dynamic Programming , 1996, Encyclopedia of Machine Learning.

[17]  Roberto Battiti,et al.  Reinforcement Learning and Reactive Search: an adaptive MAX-SAT solver , 2008, ECAI.

[18]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[19]  Thomas Stützle,et al.  Automatic Algorithm Configuration Based on Local Search , 2007, AAAI.

[20]  Holger H. Hoos,et al.  An adaptive noise mechanism for walkSAT , 2002, AAAI/IAAI.

[21]  Hector J. Levesque,et al.  Hard and Easy Distributions of SAT Problems , 1992, AAAI.

[22]  Michail G. Lagoudakis,et al.  Least-Squares Policy Iteration , 2003, J. Mach. Learn. Res..

[23]  R. Battiti,et al.  A Memory-Based RASH Optimizer , 2006 .

[24]  Frédéric Saubion,et al.  What Is Autonomous Search , 2011 .

[25]  Henry Kautz,et al.  Noise Strategies for Local Search , 1994, AAAI 1994.

[26]  Roberto Battiti,et al.  Reactive search, a history-sensitive heuristic for MAX-SAT , 1997, JEAL.

[27]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[28]  T. Kohonen,et al.  Bibliography of Self-Organizing Map SOM) Papers: 1998-2001 Addendum , 2003 .

[29]  Roberto Battiti,et al.  Machine learning methods for parameter tuning in heuristics , 2006 .

[30]  Philip W. L. Fong.  A Quantitative Study of Hypothesis Selection , 1995, ICML.

[31]  Kee-Eung Kim,et al.  Statistical Machine Learning for Large-Scale Optimization , 2000 .

[32]  Nicolas Barnier,et al.  Solving the Kirkman's schoolgirl problem in a few seconds , 2002 .

[33]  David H. Stern,et al.  Learning Adaptation to Solve Constraint Satisfaction Problems , 2009 .

[34]  Martijn C. Schut,et al.  Reinforcement Learning for Online Control of Evolutionary Algorithms , 2006, ESOA.

[35]  Michail G. Lagoudakis,et al.  Reinforcement Learning for Algorithm Selection , 2000, AAAI/IAAI.

[36]  Kristin P. Bennett,et al.  The Interplay of Optimization and Machine Learning Research , 2006, J. Mach. Learn. Res..

[37]  Wei Zhang,et al.  A Reinforcement Learning Approach to job-shop Scheduling , 1995, IJCAI.