Evolving large-scale neural networks for vision-based reinforcement learning

The idea of using evolutionary computation to train artificial neural networks, or neuroevolution (NE), for reinforcement learning (RL) tasks has now been around for over 20 years. However, as RL tasks become more challenging, the networks required become larger, as do their genomes. But, scaling NE to large nets (i.e. tens of thousands of weights) is infeasible using direct encodings that map genes one-to-one to network components. In this paper, we scale-up our compressed network encoding where network weight matrices are represented indirectly as a set of Fourier-type coefficients, to tasks that require very-large networks due to the high-dimensionality of their input space. The approach is demonstrated successfully on two reinforcement learning tasks in which the control networks receive visual input: (1) a vision-based version of the octopus control task requiring networks with over 3 thousand weights, and (2) a version of the TORCS driving game where networks with over 1 million weights are evolved to drive a car around a track using video images from the driver's perspective.

[1]  Lawrence G. Roberts,et al.  Machine Perception of Three-Dimensional Solids , 1963, Outstanding Dissertations in the Computer Sciences.

[2]  B. Widrow,et al.  An adaptive 'broom balancer' with visual inputs , 1988, IEEE 1988 International Conference on Neural Networks.

[3]  Hiroaki Kitano,et al.  Designing Neural Networks Using Genetic Algorithms with Graph Generation System , 1990, Complex Syst..

[4]  Corso Elvezia,et al.  Discovering Neural Nets with Low Kolmogorov Complexity and High Generalization Capability , 1997 .

[5]  Tamar Flash,et al.  Dynamic model of the octopus arm. I. Biomechanics of the octopus reaching movement. , 2005, Journal of neurophysiology.

[6]  Y. Engel,et al.  , Ranit Aharonov , Yaakov Engel , Binyamin of the Octopus Reaching Movement Dynamic Model of the Octopus Arm , 2005 .

[7]  Kenneth O. Stanley,et al.  Generating large-scale neural networks through discovering geometric regularities , 2007, GECCO '07.

[8]  Kenneth O. Stanley,et al.  A novel generative encoding for exploiting neural network sensor and output geometry , 2007, GECCO '07.

[9]  Risto Miikkulainen,et al.  Accelerated Neural Evolution through Cooperatively Coevolved Synapses , 2008, J. Mach. Learn. Res..

[10]  Jürgen Schmidhuber,et al.  Searching for Minimal Neural Networks in Fourier Space , 2010, AGI 2010.

[11]  Jürgen Schmidhuber,et al.  Evolving neural networks in compressed weight space , 2010, GECCO '10.

[12]  Julian Togelius,et al.  The 2009 Simulated Car Racing Championship , 2010, IEEE Transactions on Computational Intelligence and AI in Games.

[13]  Faustino J. Gomez,et al.  Intrinsically Motivated Evolutionary Search for Vision-Based Reinforcement Learning , 2011 .

[14]  W. Marsden I and J , 2012 .

[15]  Jürgen Schmidhuber,et al.  Compressed Network Complexity Search , 2012, PPSN.

[16]  Jürgen Schmidhuber,et al.  Generalized Compressed Network Search , 2012, PPSN.

[17]  Jürgen Schmidhuber,et al.  A Frequency-Domain Encoding for Neuroevolution , 2012, ArXiv.