Evolving neural networks

Neuroevolution, i.e. evolution of artificial neural networks, has recently emerged as a powerful technique for solving challenging reinforcement learning problems. Compared to traditional (e.g. value-function based) methods, neuroevolution is especially strong in domains where the state of the world is not fully known: the state can be disambiguated through recurrency, and novel situations handled through pattern matching. In this tutorial, we will review (1) neuroevolution methods that evolve fixed-topology networks, network topologies, and network construction processes, (2) ways of combining traditional neural network learning algorithms with evolutionary methods, and (3) applications of neuroevolution to control, robotics, artificial life, and games.

[1]  Risto Miikkulainen,et al.  A Taxonomy for Artificial Embryogeny , 2003, Artificial Life.

[2]  Risto Miikkulainen,et al.  Neuroevolution for adaptive teams , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[3]  Stefano Nolfi,et al.  Evolution of Communication and Language in Embodied Agents , 2009 .

[4]  Risto Miikkulainen,et al.  Robust non-linear control through neuroevolution , 2003 .

[5]  Risto Miikkulainen,et al.  Evolving explicit opponent models in game playing , 2007, GECCO '07.

[6]  Risto Miikkulainen,et al.  Forming Neural Networks Through Efficient and Adaptive Coevolution , 1997, Evolutionary Computation.

[7]  Risto Miikkulainen,et al.  Accelerating evolution via egalitarian social learning , 2012, GECCO '12.

[8]  Risto Miikkulainen,et al.  Acquiring evolvability through adaptive representations , 2007, GECCO '07.

[9]  Risto Miikkulainen,et al.  Evolution of a Communication Code in Cooperative Tasks , 2012, ALIFE.

[10]  Risto Miikkulainen,et al.  Neuro-evolution and natural deduction , 2000, 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No.00.

[11]  Risto Miikkulainen,et al.  Open-ended behavioral complexity for evolved virtual creatures , 2013, GECCO '13.

[12]  Risto Miikkulainen,et al.  Evolving agent behavior in multiobjective domains using fitness-based shaping , 2010, GECCO '10.

[13]  Masahiro Fujita,et al.  Evolution of Controllers from a High-Level Simulator to a High DOF Robot , 2000, ICES.

[14]  Risto Miikkulainen,et al.  Constructing competitive and cooperative agent behavior using coevolution , 2010, CIG.

[15]  L. Darrell Whitley,et al.  Adding Learning to the Cellular Development of Neural Networks: Evolution and the Baldwin Effect , 1993, Evolutionary Computation.

[16]  Kenneth O. Stanley,et al.  A novel generative encoding for exploiting neural network sensor and output geometry , 2007, GECCO '07.

[17]  Risto Miikkulainen,et al.  Incremental Evolution of Complex General Behavior , 1997, Adapt. Behav..

[18]  Eytan Ruppin,et al.  Emergence of Memory-Driven Command Neurons in Evolved Artificial Agents , 2001, Neural Computation.

[19]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[20]  Risto Miikkulainen,et al.  Real-time neuroevolution in the NERO video game , 2005, IEEE Transactions on Evolutionary Computation.

[21]  Risto Miikkulainen,et al.  Evolving adaptive neural networks with and without adaptive synapses , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[22]  Magnus Thor Jonsson,et al.  Evolution and design of distributed learning rules , 2000, 2000 IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks. Proceedings of the First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks (Cat. No.00.

[23]  Shimon Whiteson,et al.  Evolutionary Function Approximation for Reinforcement Learning , 2006, J. Mach. Learn. Res..

[24]  Elio Tuci,et al.  An investigation of the evolutionary origin of reciprocal communication using simulated autonomous agents , 2009, Biological Cybernetics.

[25]  Risto Miikkulainen,et al.  Acquiring Visibly Intelligent Behavior with Example-Guided Neuroevolution , 2007, AAAI.

[26]  Risto Miikkulainen,et al.  Effective diversity maintenance in deceptive domains , 2013, GECCO '13.

[27]  Geoffrey E. Hinton,et al.  How Learning Can Guide Evolution , 1996, Complex Syst..

[28]  Risto Miikkulainen,et al.  Incorporating Advice into Neuroevolution of Adaptive Agents , 2006, AIIDE.

[29]  Kenneth O. Stanley,et al.  Abandoning Objectives: Evolution Through the Search for Novelty Alone , 2011, Evolutionary Computation.

[30]  Inman Harvey,et al.  Explorations in Evolutionary Robotics , 1993, Adapt. Behav..

[31]  John J. Grefenstette,et al.  Evolutionary Algorithms for Reinforcement Learning , 1999, J. Artif. Intell. Res..

[32]  Risto Miikkulainen,et al.  Numerical optimization with neuroevolution , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[33]  Michael G. Dyer,et al.  Evolution of herding behavior in artificial animals , 1993 .

[34]  Risto Miikkulainen,et al.  Task decomposition with neuroevolution in extended predator-prey domain , 2012, ALIFE.

[35]  Daniele Loiacono,et al.  Computational intelligence in games , 2012, GECCO '12.

[36]  Risto Miikkulainen,et al.  Cultural enhancement of neuroevolution , 2002 .

[37]  Marco Colombetti,et al.  Incremental Robot Shaping , 1998, Connect. Sci..

[38]  Brad Fullmer and Risto Miikkulainen Using Marker-Based Genetic Encoding Of Neural Networks To Evolve Finite-State Behaviour , 1991 .

[39]  Risto Miikkulainen,et al.  Evolving neural network ensembles for control problems , 2005, GECCO '05.

[40]  Risto Miikkulainen,et al.  Evolving symmetric and modular neural networks for distributed control , 2009, GECCO.

[41]  X. Yao Evolving Artificial Neural Networks , 1999 .

[42]  David J. Chalmers,et al.  The Evolution of Learning: An Experiment in Genetic Connectionism , 1991 .

[43]  G. M. Werner Evolution of Communication in Artificial Organisms, Artifial Life II , 1991 .

[44]  Christian Igel,et al.  Neuroevolution for reinforcement learning using evolution strategies , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[45]  Stefano Nolfi,et al.  Good teaching inputs do not correspond to desired responses in ecological neural networks , 1994, Neural Processing Letters.

[46]  Risto Miikkulainen,et al.  Evolving Obstacle Avoidance Behavior in a Robot Arm , 1996 .

[47]  Shimon Whiteson,et al.  Comparing evolutionary and temporal difference methods in a reinforcement learning domain , 2006, GECCO.

[48]  Dario Floreano,et al.  Neuroevolution: from architectures to learning , 2008, Evol. Intell..

[49]  Xin Yao,et al.  Evolutionary ensembles with negative correlation learning , 2000, IEEE Trans. Evol. Comput..

[50]  Kenneth O. Stanley,et al.  Automatic Content Generation in the Galactic Arms Race Video Game , 2009, IEEE Transactions on Computational Intelligence and AI in Games.

[51]  Risto Miikkulainen,et al.  Accelerated Neural Evolution through Cooperatively Coevolved Synapses , 2008, J. Mach. Learn. Res..

[52]  Kenneth O. Stanley,et al.  Picbreeder: A Case Study in Collaborative Evolutionary Exploration of Design Space , 2011, Evolutionary Computation.

[53]  Risto Miikkulainen,et al.  Evolving a Roving Eye for Go , 2004, GECCO.

[54]  Risto Miikkulainen,et al.  Evolving neural networks for strategic decision-making problems , 2009, Neural Networks.

[55]  Risto Miikkulainen,et al.  Utilizing Domain Knowledge in Neuroevolution , 2003, ICML.

[56]  Richard K. Belew,et al.  Evolution, Learning, and Culture: Computational Metaphors for Adaptive Algorithms , 1990, Complex Syst..

[57]  Phil Husbands,et al.  Evolutionary robotics , 2014, Evolutionary Intelligence.

[58]  Risto Miikkulainen,et al.  Learning basic navigation for personal satellite assistant using neuroevolution , 2005, GECCO '05.

[59]  Pattie Maes,et al.  Evolving Obstacle Avoidance Behavior in a Robot Arm , 1996 .

[60]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[61]  David B. Fogel,et al.  Evolution, neural networks, games, and intelligence , 1999, Proc. IEEE.

[62]  Isaac Meilijson,et al.  Axiomatic Scalable Neurocontroller Analysis via the Shapley Value , 2006, Artificial Life.

[63]  Risto Miikkulainen,et al.  Efficient credit assignment through evaluation function decomposition , 2005, GECCO '05.

[64]  Julian Togelius,et al.  Evolving robust and specialized car racing skills , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[65]  Risto Miikkulainen,et al.  Efficient evolution of neural networks through complexification , 2004 .

[66]  F. Varela,et al.  Toward a Practice of Autonomous Systems: Proceedings of the First European Conference on Artificial Life , 1992 .

[67]  Peter J. Angeline,et al.  An evolutionary algorithm that constructs recurrent neural networks , 1994, IEEE Trans. Neural Networks.

[68]  Chris Aldrich,et al.  Intelligent process control utilising symbiotic memetic neuro-evolution , 2002, Proceedings of the 2002 Congress on Evolutionary Computation. CEC'02 (Cat. No.02TH8600).

[69]  Jimmy Secretan,et al.  Picbreeder: evolving pictures collaboratively online , 2008, CHI.

[70]  Kenneth O. Stanley,et al.  Generative encoding for multiagent learning , 2008, GECCO '08.

[71]  Jeffrey L. Elman,et al.  Learning and Evolution in Neural Networks , 1994, Adapt. Behav..

[72]  Kenneth O. Stanley,et al.  A Case Study on the Critical Role of Geometric Regularity in Machine Learning , 2008, AAAI.

[73]  Kenneth A. De Jong,et al.  Cooperative Coevolution: An Architecture for Evolving Coadapted Subcomponents , 2000, Evolutionary Computation.

[74]  Kenneth O. Stanley,et al.  Learning to Dance through Interactive Evolution , 2010, EvoApplications.

[75]  Chad W. Seys,et al.  Evolving Walking : The Anatomy of an Evolutionary Search , 2004 .

[76]  Risto Miikkulainen,et al.  Modular neuroevolution for multilegged locomotion , 2008, GECCO '08.

[77]  Alexis P. Wieland,et al.  Evolving Controls for Unstable Systems , 1991 .

[78]  Risto Miikkulainen,et al.  The role of reward structure, coordination mechanism and net return in the evolution of cooperation , 2011, CIG.

[79]  Risto Miikkulainen,et al.  Transfer of Neuroevolved Controllers in Unstable Domains , 2004, GECCO.

[80]  E. Ruppin Evolutionary autonomous agents: A neuroscience perspective , 2002, Nature Reviews Neuroscience.

[81]  Risto Miikkulainen,et al.  Constructing good learners using evolved pattern generators , 2005, GECCO '05.

[82]  Risto Miikkulainen,et al.  Constructing controllers for physical multilegged robots using the ENSO neuroevolution approach , 2012, Evol. Intell..

[83]  Simon M. Lucas,et al.  A comparison of matrix rewriting versus direct encoding for evolving neural networks , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[84]  Risto Miikkulainen,et al.  Evolving Keepaway Soccer Players through Task Decomposition , 2003, GECCO.

[85]  Risto Miikkulainen,et al.  Evolving multi-modal behavior in NPCs , 2009, CIG.

[86]  Risto Miikkulainen,et al.  Evolving a real-world vehicle warning system , 2006, GECCO.

[87]  Risto Miikkulainen,et al.  Coevolution of Role-Based Cooperation in Multiagent Systems , 2009, IEEE Transactions on Autonomous Mental Development.

[88]  David H. Sharp,et al.  Scaling, machine learning, and genetic neural nets , 1989 .

[89]  Kenneth O. Stanley,et al.  Efficiently evolving programs through the search for novelty , 2010, GECCO '10.

[90]  Risto Miikkulainen,et al.  COOPERATIVE COEVOLUTION OF MULTI-AGENT SYSTEMS , 2001 .

[91]  Richard K. Belew,et al.  New Methods for Competitive Coevolution , 1997, Evolutionary Computation.

[92]  Risto Miikkulainen,et al.  Adaptive Control Utilising Neural Swarming , 2002, GECCO.

[93]  Kenneth O. Stanley,et al.  Revising the evolutionary computation abstraction: minimal criteria novelty search , 2010, GECCO '10.

[94]  David B. Fogel,et al.  Further Evolution of a Self-Learning Chess Program , 2005, CIG.

[95]  Karl Sims,et al.  Evolving 3D Morphology and Behavior by Competition , 1994, Artificial Life.

[96]  Jordi Madrenas,et al.  Evolvable Systems: From Biology to Hardware , 1996, Lecture Notes in Computer Science.

[97]  Dario Floreano,et al.  Evolutionary robots with on-line self-organization and behavioral fitness , 2000, Neural Networks.

[98]  Risto Miikkulainen,et al.  Evolving Symmetry for Modular System Design , 2011, IEEE Trans. Evol. Comput..

[99]  David E. Moriarty,et al.  Symbiotic Evolution of Neural Networks in Sequential Decision Tasks , 1997 .

[100]  John Hallam,et al.  IEEE International Joint Conference on Neural Networks , 2005 .

[101]  Risto Miikkulainen,et al.  Evolving Reusable Neural Modules , 2004, GECCO.

[102]  Risto Miikkulainen,et al.  Competitive Coevolution through Evolutionary Complexification , 2011, J. Artif. Intell. Res..

[103]  Francesco Mondada,et al.  Evolutionary neurocontrollers for autonomous mobile robots , 1998, Neural Networks.

[104]  Kenneth O. Stanley,et al.  Scaffolding for Interactively Evolving Novel Drum Tracks for Existing Songs , 2008, EvoWorkshops.

[105]  Stéphane Doncieux,et al.  Overcoming the bootstrap problem in evolutionary robotics using behavioral diversity , 2009, 2009 IEEE Congress on Evolutionary Computation.

[106]  Risto Miikkulainen,et al.  Creating melodies with evolving recurrent neural networks , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[107]  J. Baldwin A New Factor in Evolution , 1896, The American Naturalist.

[108]  Shimon Whiteson,et al.  Stochastic Optimization for Collision Selection in High Energy Physics , 2006, AAAI.

[109]  Risto Miikkulainen,et al.  Active Guidance for a Finless Rocket Using Neuroevolution , 2003, GECCO.

[110]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[111]  L. D. Whitley,et al.  Genetic Reinforcement Learning for Neurocontrol Problems , 2004, Machine Learning.

[112]  Risto Miikkulainen,et al.  A neuro-evolution method for dynamic resource allocation on a chip multiprocessor , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[113]  Donald E. Waagen,et al.  Evolving recurrent perceptrons for time-series modeling , 1994, IEEE Trans. Neural Networks.