Evolving neural networks for fractured domains

Evolution of neural networks, or neuroevolution, bas been successful on many low-level control problems such as pole balancing, vehicle control, and collision warning. However, high-level strategy problems that require the integration of multiple sub-behaviors have remained difficult for neuroevolution to solve. This paper proposes the hypothesis that such problems are difficult because they are fractured: the correct action varies discontinuously as the agent moves from state to state. This hypothesis is evaluated on several examples of fractured high-level reinforcement learning domains. Standard neuroevolution methods such as NEAT indeed have difficulty solving them. However, a modification of NEAT that uses radial basis function (RBF) nodes to make precise local mutations to network output is able to do much better. These results provide a better understanding of the different types of reinforcement learning problems and the limitations of current neuroevolution methods. Thus, they lay the groundwork for creating the next generation of neuroevolution algorithms that can learn strategic high-level behavior in fractured domains.

[1]  Jorma Rissanen,et al.  The Minimum Description Length Principle in Coding and Modeling , 1998, IEEE Trans. Inf. Theory.

[2]  Héctor Pomares,et al.  Improving the Performance of Multi-objective Genetic Algorithm for Function Approximation Through Parallel Islands Specialisation , 2006, Australian Conference on Artificial Intelligence.

[3]  Daniele Loiacono,et al.  Classifier prediction based on tile coding , 2006, GECCO '06.

[4]  X. Yao Evolving Artificial Neural Networks , 1999 .

[5]  Daniele Loiacono,et al.  XCS with computed prediction for the learning of Boolean functions , 2005, 2005 IEEE Congress on Evolutionary Computation.

[6]  Didier Guériot,et al.  RBF neural network, basis functions and genetic algorithm , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[7]  Lin Guo,et al.  Combining genetic optimisation with hybrid learning algorithm for radial basis function neural networks , 2003 .

[8]  Shimon Whiteson,et al.  Comparing evolutionary and temporal difference methods in a reinforcement learning domain , 2006, GECCO.

[9]  Risto Miikkulainen,et al.  Coevolving Strategies for General Game Playing , 2007, 2007 IEEE Symposium on Computational Intelligence and Games.

[10]  Matthew Taylor and Shimon Whiteson and Peter Stone,et al.  Comparing Evolutionary and Temporal Difference Methods for Reinforcement Learning , 2006 .

[11]  Héctor Pomares,et al.  Multiobjective evolutionary optimization of the size, shape, and position parameters of radial basis function networks for function approximation , 2003, IEEE Trans. Neural Networks.

[12]  Hans-Martin Gutmann,et al.  A Radial Basis Function Method for Global Optimization , 2001, J. Glob. Optim..

[13]  Risto Miikkulainen,et al.  Efficient Non-linear Control Through Neuroevolution , 2006, ECML.

[14]  Martin V. Butz Kernel-based, ellipsoidal conditions in the real-valued XCS classifier system , 2005, GECCO '05.

[15]  David E. Goldberg,et al.  Genetic Algorithms with Sharing for Multimodalfunction Optimization , 1987, ICGA.

[16]  P. Angeline Evolving basis functions with dynamic receptive fields , 1997, 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation.

[17]  Risto Miikkulainen,et al.  Neuroevolution of an automobile crash warning system , 2005, GECCO '05.

[18]  Héctor Pomares,et al.  Parallel Multi-objective Memetic RBFNNs Design and Feature Selection for Function Approximation Problems , 2007, IWANN.

[19]  Ali M. S. Zalzala,et al.  Evolving hybrid RBF-MLP networks using combined genetic/unsupervised/supervised learning , 1998 .

[20]  Julian Togelius,et al.  The 2007 IEEE CEC simulated car racing competition , 2008, Genetic Programming and Evolvable Machines.

[21]  Jun Li,et al.  Q-RAN: A Constructive Reinforcement Learning Approach for Robot Behavior Learning , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[22]  Stephen A. Billings,et al.  Radial basis function network configuration using genetic algorithms , 1995, Neural Networks.

[23]  Héctor Pomares,et al.  Parallel multiobjective memetic RBFNNs design and feature selection for function approximation problems , 2009, Neurocomputing.

[24]  Peter Stone,et al.  Keepaway Soccer: From Machine Learning Testbed to Benchmark , 2005, RoboCup.

[25]  Risto Miikkulainen,et al.  Evolving a real-world vehicle warning system , 2006, GECCO.

[26]  Larry Bull,et al.  Accuracy-based Neuro And Neuro-fuzzy Classifier Systems , 2002, GECCO.

[27]  Nicholas J. Radcliffe,et al.  Genetic set recombination and its application to neural network topology optimisation , 1993, Neural Computing & Applications.

[28]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[29]  Haralambos Sarimveis,et al.  A new algorithm for developing dynamic radial basis function neural network models based on genetic algorithms , 2004, Comput. Chem. Eng..

[30]  Stewart W. Wilson Classifier Conditions Using Gene Expression Programming , 2008, IWLCS.

[31]  Stewart W. Wilson Classifiers that approximate functions , 2002, Natural Computing.

[32]  Julian Togelius,et al.  Point-to-Point Car Racing: an Initial Study of Evolution Versus Temporal Difference Learning , 2007, 2007 IEEE Symposium on Computational Intelligence and Games.

[33]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[34]  Risto Miikkulainen,et al.  Evolving a Roving Eye for Go , 2004, GECCO.

[35]  Charles W. Anderson,et al.  Comparison of CMACs and radial basis functions for local function approximators in reinforcement learning , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[36]  Li Jun,et al.  Q-Learning with a growing RBF network for behavior learning in mobile robotics , 2005 .

[37]  Risto Miikkulainen,et al.  Real-time neuroevolution in the NERO video game , 2005, IEEE Transactions on Evolutionary Computation.

[38]  Risto Miikkulainen,et al.  Competitive Coevolution through Evolutionary Complexification , 2011, J. Artif. Intell. Res..

[39]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[40]  Larry D. Pyeatt,et al.  A comparison between cellular encoding and direct encoding for genetic neural networks , 1996 .

[41]  Bruce A. Whitehead,et al.  Cooperative-competitive genetic evolution of radial basis function centers and widths for time series prediction , 1996, IEEE Trans. Neural Networks.

[42]  Vladimir Vapnik,et al.  Chervonenkis: On the uniform convergence of relative frequencies of events to their probabilities , 1971 .