Evolving neural networks

Neuroevolution, i.e. evolution of artificial neural networks, has recently emerged as a powerful technique for solving challenging reinforcement learning problems. Compared to traditional(e.g. value-function based) methods, neuroevolution is especially strong in domains where the state of the world is not fully known: the state can be disambiguated through recurrency, and novel situations handled through pattern matching. In this tutorial, we will review (1)neuroevolution methods that evolve fixed-topology networks, network topologies, and network construction processes, (2) ways of combining traditional neural network learning algorithms with evolutionary methods, and (3) applications of neuroevolution to game playing, robot control, resource optimization, and cognitive science.

[1]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[2]  L. D. Whitley,et al.  Genetic Reinforcement Learning for Neurocontrol Problems , 2004, Machine Learning.

[3]  David J. Chalmers,et al.  The Evolution of Learning: An Experiment in Genetic Connectionism , 1991 .

[4]  G. M. Werner Evolution of Communication in Artificial Organisms, Artifial Life II , 1991 .

[5]  Risto Miikkulainen,et al.  A neuro-evolution method for dynamic resource allocation on a chip multiprocessor , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[6]  Donald E. Waagen,et al.  Evolving recurrent perceptrons for time-series modeling , 1994, IEEE Trans. Neural Networks.

[7]  Dario Floreano,et al.  Analog Genetic Encoding for the Evolution of Circuits and Networks , 2007, IEEE Transactions on Evolutionary Computation.

[8]  Risto Miikkulainen,et al.  Efficient evolution of neural networks through complexification , 2004 .

[9]  Risto Miikkulainen,et al.  Transfer of Neuroevolved Controllers in Unstable Domains , 2004, GECCO.

[10]  E. Ruppin Evolutionary autonomous agents: A neuroscience perspective , 2002, Nature Reviews Neuroscience.

[11]  Kenneth O. Stanley,et al.  Exploiting Open-Endedness to Solve Problems Through the Search for Novelty , 2008, ALIFE.

[12]  Risto Miikkulainen,et al.  Constructing good learners using evolved pattern generators , 2005, GECCO '05.

[13]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[14]  Chris Aldrich,et al.  Intelligent process control utilising symbiotic memetic neuro-evolution , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[15]  Jimmy Secretan,et al.  Picbreeder: evolving pictures collaboratively online , 2008, CHI.

[16]  Kenneth O. Stanley,et al.  Generative encoding for multiagent learning , 2008, GECCO '08.

[17]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[18]  Simon M. Lucas,et al.  A comparison of matrix rewriting versus direct encoding for evolving neural networks , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[19]  Risto Miikkulainen,et al.  Evolving Keepaway Soccer Players through Task Decomposition , 2003, GECCO.

[20]  Kenneth O. Stanley,et al.  A novel generative encoding for exploiting neural network sensor and output geometry , 2007, GECCO '07.

[21]  Risto Miikkulainen,et al.  Evolving a real-world vehicle warning system , 2006, GECCO.

[22]  Risto Miikkulainen,et al.  Coevolution of Role-Based Cooperation in Multiagent Systems , 2009, IEEE Transactions on Autonomous Mental Development.

[23]  Risto Miikkulainen,et al.  Efficient credit assignment through evaluation function decomposition , 2005, GECCO '05.

[24]  David H. Sharp,et al.  Scaling, machine learning, and genetic neural nets , 1989 .

[25]  Kenneth O. Stanley,et al.  NEAT Particles: Design, Representation, and Animation of Particle System Effects , 2007, 2007 IEEE Symposium on Computational Intelligence and Games.

[26]  Risto Miikkulainen,et al.  Evolving a Roving Eye for Go , 2004, GECCO.

[27]  Risto Miikkulainen,et al.  Evolving neural networks for strategic decision-making problems , 2009, Neural Networks.

[28]  Julian Togelius,et al.  Evolving robust and specialized car racing skills , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[29]  Risto Miikkulainen,et al.  Creating melodies with evolving recurrent neural networks , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[30]  J. Baldwin A New Factor in Evolution , 1896, The American Naturalist.

[31]  Shimon Whiteson,et al.  Stochastic Optimization for Collision Selection in High Energy Physics , 2006, AAAI.

[32]  Eytan Ruppin,et al.  Emergence of Memory-Driven Command Neurons in Evolved Artificial Agents , 2001, Neural Computation.

[33]  Risto Miikkulainen,et al.  Active Guidance for a Finless Rocket Using Neuroevolution , 2003, GECCO.

[34]  Kenneth O. Stanley,et al.  Compositional Pattern Producing Networks : A Novel Abstraction of Development , 2007 .

[35]  Risto Miikkulainen,et al.  Incorporating Advice into Neuroevolution of Adaptive Agents , 2006, AIIDE.

[36]  John J. Grefenstette,et al.  Evolutionary Algorithms for Reinforcement Learning , 1999, J. Artif. Intell. Res..

[37]  Richard K. Belew,et al.  Evolution, Learning, and Culture: Computational Metaphors for Adaptive Algorithms , 1990, Complex Syst..

[38]  L. Darrell Whitley,et al.  Adding Learning to the Cellular Development of Neural Networks: Evolution and the Baldwin Effect , 1993, Evolutionary Computation.

[39]  Risto Miikkulainen,et al.  Evolving neural networks for fractured domains , 2008, GECCO '08.

[40]  David B. Fogel,et al.  Further Evolution of a Self-Learning Chess Program , 2005, CIG.

[41]  Risto Miikkulainen,et al.  Cultural enhancement of neuroevolution , 2002 .

[42]  Risto Miikkulainen,et al.  Utilizing Domain Knowledge in Neuroevolution , 2003, ICML.

[43]  Risto Miikkulainen,et al.  Neuroevolution for adaptive teams , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[44]  Dario Floreano,et al.  Neuroevolution: from architectures to learning , 2008, Evol. Intell..

[45]  Risto Miikkulainen,et al.  Accelerated Neural Evolution through Cooperatively Coevolved Synapses , 2008, J. Mach. Learn. Res..

[46]  Pattie Maes,et al.  Evolving Obstacle Avoidance Behavior in a Robot Arm , 1996 .

[47]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[48]  Risto Miikkulainen,et al.  Automatic feature selection in neuroevolution , 2005, GECCO '05.

[49]  David B. Fogel,et al.  Evolution, neural networks, games, and intelligence , 1999, Proc. IEEE.

[50]  Isaac Meilijson,et al.  Axiomatic Scalable Neurocontroller Analysis via the Shapley Value , 2006, Artificial Life.

[51]  Risto Miikkulainen,et al.  COOPERATIVE COEVOLUTION OF MULTI-AGENT SYSTEMS , 2001 .

[52]  Richard K. Belew,et al.  New Methods for Competitive Coevolution , 1997, Evolutionary Computation.

[53]  Risto Miikkulainen,et al.  Adaptive Control Utilising Neural Swarming , 2002, GECCO.

[54]  Geoffrey E. Hinton,et al.  How Learning Can Guide Evolution , 1996, Complex Syst..

[55]  Inman Harvey,et al.  Explorations in Evolutionary Robotics , 1993, Adapt. Behav..

[56]  Risto Miikkulainen,et al.  Numerical optimization with neuroevolution , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[57]  Risto Miikkulainen,et al.  Robust non-linear control through neuroevolution , 2003 .

[58]  David B. Fogel,et al.  Blondie24: Playing at the Edge of AI , 2001 .

[59]  Risto Miikkulainen,et al.  Incremental Evolution of Complex General Behavior , 1997, Adapt. Behav..

[60]  Risto Miikkulainen,et al.  Efficient Non-linear Control Through Neuroevolution , 2006, ECML.

[61]  Risto Miikkulainen,et al.  Modular neuroevolution for multilegged locomotion , 2008, GECCO '08.

[62]  Nicholas J. Radcliffe,et al.  Genetic set recombination and its application to neural network topology optimisation , 1993, Neural Computing & Applications.

[63]  Larry D. Pyeatt,et al.  A comparison between cellular encoding and direct encoding for genetic neural networks , 1996 .

[64]  Risto Miikkulainen,et al.  Learning basic navigation for personal satellite assistant using neuroevolution , 2005, GECCO '05.

[65]  Risto Miikkulainen,et al.  Evolving adaptive neural networks with and without adaptive synapses , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[66]  Magnus Thor Jonsson,et al.  Evolution and design of distributed learning rules , 2000, 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No.00.

[67]  Marco Colombetti,et al.  Incremental Robot Shaping , 1998, Connect. Sci..

[68]  Brad Fullmer and Risto Miikkulainen Using Marker-Based Genetic Encoding Of Neural Networks To Evolve Finite-State Behaviour , 1991 .

[69]  Risto Miikkulainen,et al.  Evolving Soccer Keepaway Players Through Task Decomposition , 2005, Machine Learning.

[70]  Byoung-Tak Zhang,et al.  Evolving Optimal Neural Networks Using Genetic Algorithms with Occam's Razor , 1993, Complex Syst..

[71]  Christian Igel,et al.  Neuroevolution for reinforcement learning using evolution strategies , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[72]  S. Vijayakumar,et al.  Evolving Walking: The Anatomy of an Evolutionary Search , 2004 .

[73]  Jeffrey L. Elman,et al.  Learning and Evolution in Neural Networks , 1994, Adapt. Behav..

[74]  Risto Miikkulainen,et al.  Discovering Complex Othello Strategies through Evolutionary Neural Networks , 1995, Connect. Sci..

[75]  Kenneth O. Stanley,et al.  A Case Study on the Critical Role of Geometric Regularity in Machine Learning , 2008, AAAI.

[76]  Kenneth A. De Jong,et al.  Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents , 2000, Evolutionary Computation.

[77]  Risto Miikkulainen,et al.  Evolving explicit opponent models in game playing , 2007, GECCO '07.

[78]  Risto Miikkulainen,et al.  Forming Neural Networks Through Efficient and Adaptive Coevolution , 1997, Evolutionary Computation.

[79]  Stefano Nolfi,et al.  Good teaching inputs do not correspond to desired responses in ecological neural networks , 1994, Neural Processing Letters.

[80]  Risto Miikkulainen,et al.  Evolving Obstacle Avoidance Behavior in a Robot Arm , 1996 .

[81]  Risto Miikkulainen,et al.  Neuro-evolution and natural deduction , 2000, 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No.00.

[82]  Masahiro Fujita,et al.  Evolution of Controllers from a High-Level Simulator to a High DOF Robot , 2000, ICES.

[83]  Chern Han Yong and Kenneth O. Stanley and Risto Miikkulainen Incorporating Advice into Evolution of Neural Networks , 2005 .

[84]  Chad W. Seys,et al.  Evolving Walking : The Anatomy of an Evolutionary Search , 2004 .

[85]  Pavel Brazdil,et al.  Proceedings of the European Conference on Machine Learning , 1993 .

[86]  Alexis P. Wieland,et al.  Evolving Controls for Unstable Systems , 1991 .

[87]  Riccardo Poli,et al.  Evolving the Topology and the Weights of Neural Networks Using a Dual Representation , 2004, Applied Intelligence.

[88]  Risto Miikkulainen,et al.  Acquiring evolvability through adaptive representations , 2007, GECCO '07.

[89]  Karl Sims,et al.  Evolving 3d morphology and behavior by competition , 1994 .

[90]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[91]  Lakhmi C. Jain,et al.  Computational Intelligence in Games , 2005, IEEE Transactions on Neural Networks.

[92]  V. Ramakrishnan,et al.  Measurement of the top-quark mass with dilepton events selected using neuroevolution at CDF. , 2008, Physical review letters.

[93]  Michael G. Dyer,et al.  Evolution of herding behavior in artificial animals , 1993 .

[94]  Risto Miikkulainen,et al.  Coevolution of neural networks using a layered pareto archive , 2006, GECCO.

[95]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[96]  Risto Miikkulainen,et al.  Real-time neuroevolution in the NERO video game , 2005, IEEE Transactions on Evolutionary Computation.

[97]  Shimon Whiteson,et al.  Comparing evolutionary and temporal difference methods in a reinforcement learning domain , 2006, GECCO.

[98]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[99]  Risto Miikkulainen,et al.  A Taxonomy for Artificial Embryogeny , 2003, Artificial Life.

[100]  Shimon Whiteson,et al.  Evolutionary Function Approximation for Reinforcement Learning , 2006, J. Mach. Learn. Res..

[101]  Risto Miikkulainen,et al.  Acquiring Visibly Intelligent Behavior with Example-Guided Neuroevolution , 2007, AAAI.

[102]  Risto Miikkulainen,et al.  Evolving neural network ensembles for control problems , 2005, GECCO '05.

[103]  Jude W. Shavlik,et al.  Creating advice-taking reinforcement learners , 1998 .

[104]  Risto Miikkulainen,et al.  Evolving Reusable Neural Modules , 2004, GECCO.

[105]  Risto Miikkulainen,et al.  Competitive Coevolution through Evolutionary Complexification , 2011, J. Artif. Intell. Res..

[106]  Francesco Mondada,et al.  Evolutionary neurocontrollers for autonomous mobile robots , 1998, Neural Networks.

[107]  Kenneth O. Stanley,et al.  Scaffolding for Interactively Evolving Novel Drum Tracks for Existing Songs , 2008, EvoWorkshops.

[108]  Jordi Madrenas,et al.  Evolvable Systems: From Biology to Hardware , 1996, Lecture Notes in Computer Science.

[109]  Dario Floreano,et al.  Evolutionary robots with on-line self-organization and behavioral fitness , 2000, Neural Networks.

[110]  David E. Moriarty,et al.  Symbiotic Evolution of Neural Networks in Sequential Decision Tasks , 1997 .

[111]  Jordan B. Pollack,et al.  Creating High-Level Components with a Generative Representation for Body-Brain Evolution , 2002, Artificial Life.