Optimal Neuron Selection: NK Echo State Networks for Reinforcement Learning

This paper introduces the NK Echo State Network. The problem of learning in the NK Echo State Network is reduced to the problem of optimizing a special form of a Spin Glass Problem known as an NK Landscape. No weight adjustment is used; all learning is accomplished by spinning up (turning on) or spinning down (turning off) neurons in order to find a combination of neurons that work together to achieve the desired computation. For special types of NK Landscapes, an exact global solution can be obtained in polynomial time using dynamic programming. The NK Echo State Network is applied to a reinforcement learning problem requiring a recurrent network: balancing two poles on a cart given no velocity information. Empirical results shows that the NK Echo State Network learns very rapidly and yields very good generalization.

[1]  Sébastien Vérel,et al.  Complex-network analysis of combinatorial spaces: The NK landscape case , 2008, Physical review. E, Statistical, nonlinear, and soft matter physics.

[2]  L. Darrell Whitley,et al.  Genetic Reinforcement Learning for Neurocontrol Problems , 2004, Machine Learning.

[3]  A. P. Wieland,et al.  Evolving neural network controllers for unstable systems , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[4]  Risto Miikkulainen,et al.  Efficient Reinforcement Learning Through Evolving Neural Network Topologies , 2002, GECCO.

[5]  S Kirkpatrick,et al.  Critical Behavior in the Satisfiability of Random Boolean Expressions , 1994, Science.

[6]  Stuart A. Kauffman,et al.  The origins of order , 1993 .

[7]  José Carlos Príncipe,et al.  Analysis and Design of Echo State Networks , 2007, Neural Computation.

[8]  Jochen J. Steil,et al.  Analyzing the weight dynamics of recurrent learning algorithms , 2005, Neurocomputing.

[9]  F. Guerra.  Spin Glasses , 2005, cond-mat/0507581.

[10]  Ashwin Ram,et al.  Experiments with Reinforcement Learning in Problems with Continuous State and Action Spaces , 1997, Adapt. Behav..

[11]  Risto Miikkulainen,et al.  Accelerated Neural Evolution through Cooperatively Coevolved Synapses , 2008, J. Mach. Learn. Res..

[12]  Alden H. Wright,et al.  The computational complexity of N-K fitness functions , 2000, IEEE Trans. Evol. Comput..

[13]  Hector J. Levesque,et al.  Hard and Easy Distributions of SAT Problems , 1992, AAAI.

[14]  Larry D. Pyeatt,et al.  A comparison between cellular encoding and direct encoding for genetic neural networks , 1996 .

[15]  Helmut Hauser,et al.  Echo state networks with filter neurons and a delay&sum readout , 2010, Neural Networks.

[16]  Andrew M. Sutton,et al.  Efficient identification of improving moves in a ball for pseudo-boolean problems , 2014, GECCO.

[17]  Dario Floreano,et al.  Neuroevolution with Analog Genetic Encoding , 2006, PPSN.

[18]  Risto Miikkulainen,et al.  Solving Non-Markovian Control Tasks with Neuro-Evolution , 1999, IJCAI.

[19]  Mantas Lukosevicius,et al.  A Practical Guide to Applying Echo State Networks , 2012, Neural Networks: Tricks of the Trade.

[20]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[21]  L. Darrell Whitley,et al.  Constant time steepest descent local search with lookahead for NK-landscapes and MAX-kSAT , 2012, GECCO '12.

[22]  Marc Schoenauer,et al.  Supervised and Evolutionary Learning of Echo State Networks , 2008, PPSN.