Evolutionary Bilevel Optimization for Complex Control Tasks

Most optimization algorithms must undergo time consuming parameter adaptation in order to optimally solve complex, real-world control tasks. Parameter adaptation is inherently a bilevel optimization problem where the lower level objective function is the performance of the control parameters discovered by an optimization algorithm and the upper level objective function is the performance of the algorithm given its parametrization. In this paper, a novel method called MetaEvolutionary Algorithm (MEA) is presented and shown to be capable of efficiently discovering optimal parameters for neuroevolution to solve control problems. In two challenging examples, double pole balancing and helicopter hovering, MEA discovers optimized parameters that result in better performance than hand tuning and other automatic methods. Bilevel optimization in general and MEA in particular, is thus a promising approach for solving difficult control tasks.

[1]  Marko Robnik,et al.  Improving Random Forests , 2004 .

[2]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[3]  Petros Faloutsos,et al.  Evolved Controllers for Simulated Locomotion , 2009, MIG.

[4]  A. E. Eiben,et al.  Efficient relevance estimation and value calibration of evolutionary algorithm parameters , 2007, 2007 IEEE Congress on Evolutionary Computation.

[5]  W. Spears,et al.  On the Virtues of Parameterized Uniform Crossover , 1995 .

[6]  Dario Floreano,et al.  Neuroevolution: from architectures to learning , 2008, Evol. Intell..

[7]  Thomas Stützle,et al.  F-Race and Iterated F-Race: An Overview , 2010, Experimental Methods for the Analysis of Optimization Algorithms.

[8]  Risto Miikkulainen,et al.  Robust non-linear control through neuroevolution , 2003 .

[9]  Risto Miikkulainen,et al.  Accelerated Neural Evolution through Cooperatively Coevolved Synapses , 2008, J. Mach. Learn. Res..

[10]  Julian Togelius,et al.  Evolving controllers for simulated car racing , 2005, 2005 IEEE Congress on Evolutionary Computation.

[11]  Manuel Laguna,et al.  Fine-Tuning of Algorithms Using Fractional Experimental Designs and Local Search , 2006, Oper. Res..

[12]  Gilbert Syswerda,et al.  Uniform Crossover in Genetic Algorithms , 1989, ICGA.

[13]  Adrião Duarte Dória Neto,et al.  Logistic regression for parameter tuning on an evolutionary algorithm , 2005, 2005 IEEE Congress on Evolutionary Computation.

[14]  Aravind Srinivasan,et al.  A Population-Based, Parent Centric Procedure for Constrained Real-Parameter Optimization , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[15]  Kalyanmoy Deb,et al.  A bilevel optimization approach to automated parameter tuning , 2014, GECCO.

[16]  X. Yao Evolving Artificial Neural Networks , 1999 .

[17]  John J. Grefenstette,et al.  Optimization of Control Parameters for Genetic Algorithms , 1986, IEEE Transactions on Systems, Man, and Cybernetics.

[18]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[19]  Alain Ratle,et al.  Accelerating the Convergence of Evolutionary Algorithms by Fitness Landscape Approximation , 1998, PPSN.

[20]  Nostrand Reinhold,et al.  the utility of using the genetic algorithm approach on the problem of Davis, L. (1991), Handbook of Genetic Algorithms. Van Nostrand Reinhold, New York. , 1991 .

[21]  Oliver Kramer,et al.  Evolutionary self-adaptation: a survey of operators and strategy parameters , 2010, Evol. Intell..

[22]  Thomas Bartz-Beielstein,et al.  Experimental Methods for the Analysis of Optimization Algorithms , 2010 .

[23]  Yaochu Jin,et al.  A comprehensive survey of fitness approximation in evolutionary computation , 2005, Soft Comput..

[24]  Jeff G. Schneider,et al.  Autonomous helicopter control using reinforcement learning policy search methods , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[25]  Hamid R. Berenji,et al.  Learning and tuning fuzzy logic controllers through reinforcements , 1992, IEEE Trans. Neural Networks.

[26]  John J. Grefenstette,et al.  Evolutionary Algorithms for Reinforcement Learning , 1999, J. Artif. Intell. Res..

[27]  Risto Miikkulainen,et al.  Active Guidance for a Finless Rocket Using Neuroevolution , 2003, GECCO.

[28]  Mihai Oltean,et al.  Evolving evolutionary algorithms using evolutionary algorithms , 2007, GECCO '07.

[29]  A. E. Eiben,et al.  Parameter tuning for configuring and analyzing evolutionary algorithms , 2011, Swarm Evol. Comput..

[30]  Lawrence. Davis,et al.  Handbook Of Genetic Algorithms , 1990 .

[31]  Andrew W. Moore,et al.  Gradient Descent for General Reinforcement Learning , 1998, NIPS.

[32]  Marko Robnik-Sikonja,et al.  Improving Random Forests , 2004, ECML.

[33]  Peter Stone,et al.  Policy gradient reinforcement learning for fast quadrupedal locomotion , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[34]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[35]  A. P. Wieland,et al.  Evolving neural network controllers for unstable systems , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[36]  Shimon Whiteson,et al.  Neuroevolutionary reinforcement learning for generalized control of simulated helicopters , 2011, Evol. Intell..

[37]  Carlos Ansótegui,et al.  A Gender-Based Genetic Algorithm for the Automatic Configuration of Algorithms , 2009, CP.

[38]  Risto Miikkulainen,et al.  Efficient Non-linear Control Through Neuroevolution , 2006, ECML.

[39]  Edwin R. Hancock,et al.  Empirical Modelling of Genetic Algorithms , 2001, Evolutionary Computation.