Reuse of Neural Modules for General Video Game Playing

A general approach to knowledge transfer is introduced in which an agent controlled by a neural network adapts how it reuses existing networks as it learns in a new domain. Networks trained for a new domain can improve their performance by routing activation selectively through previously learned neural structure, regardless of how or for what it was learned. A neuroevolution implementation of this approach is presented with application to high-dimensional sequential decision-making domains. This approach is more general than previous approaches to neural transfer for reinforcement learning. It is domain-agnostic and requires no prior assumptions about the nature of task relatedness or mappings. The method is analyzed in a stochastic version of the Arcade Learning Environment, demonstrating that it improves performance in some of the more complex Atari 2600 games, and that the success of transfer can be predicted based on a high-level characterization of game dynamics.

[1]  Tom Schaul,et al.  A video game description language for model-based or interactive learning , 2013, 2013 IEEE Conference on Computational Inteligence in Games (CIG).

[2]  Jürgen Schmidhuber,et al.  Training Recurrent Networks by Evolino , 2007, Neural Computation.

[3]  Kenneth O. Stanley,et al.  Evolving Static Representations for Task Transfer , 2010, J. Mach. Learn. Res..

[4]  Thomas R. Shultz,et al.  Knowledge-based cascade-correlation: Using knowledge to speed learning , 2001, Connect. Sci..

[5]  Peter Stone,et al.  The Impact of Determinism on Learning Atari 2600 Games , 2015, AAAI Workshop: Learning for General Competency in Video Games.

[6]  S. Shen-Orr,et al.  Network motifs: simple building blocks of complex networks. , 2002, Science.

[7]  Shimon Whiteson,et al.  Transfer via inter-task mappings in policy search reinforcement learning , 2007, AAMAS '07.

[8]  Julian Togelius,et al.  Ieee Transactions on Computational Intelligence and Ai in Games the 2014 General Video Game Playing Competition , 2022 .

[9]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[10]  Herbert Jaeger,et al.  Reservoir computing approaches to recurrent neural network training , 2009, Comput. Sci. Rev..

[11]  Elliot Meyerson,et al.  Frame Skip Is a Powerful Parameter for Learning to Play Atari , 2015, AAAI Workshop: Learning for General Competency in Video Games.

[12]  Risto Miikkulainen,et al.  Evolving Reusable Neural Modules , 2004, GECCO.

[13]  Peter Stone,et al.  Autonomous transfer for reinforcement learning , 2008, AAMAS.

[14]  Jürgen Schmidhuber,et al.  Evolving deep unsupervised convolutional networks for vision-based reinforcement learning , 2014, GECCO.

[15]  Matthew E. Taylor,et al.  Policy Transfer using Reward Shaping , 2015, AAMAS.

[16]  Marc G. Bellemare,et al.  The Arcade Learning Environment: An Evaluation Platform for General Agents (Extended Abstract) , 2012, IJCAI.

[17]  Risto Miikkulainen,et al.  Incremental Evolution of Complex General Behavior , 1997, Adapt. Behav..

[18]  Peter Stone,et al.  Transfer Learning for Reinforcement Learning Domains: A Survey , 2009, J. Mach. Learn. Res..

[19]  Erik Talvitie,et al.  An Experts Algorithm for Transfer Learning , 2007, IJCAI.

[20]  Eric Eaton,et al.  Unsupervised Cross-Domain Transfer in Policy Gradient Reinforcement Learning via Manifold Alignment , 2015, AAAI.

[21]  Jürgen Schmidhuber,et al.  Co-evolving recurrent neurons learn deep memory POMDPs , 2005, GECCO '05.

[22]  Xin Yao,et al.  Co-evolutionary modular neural networks for automatic problem decomposition , 2005, 2005 IEEE Congress on Evolutionary Computation.

[23]  Andrew G. Barto,et al.  Transfer in Reinforcement Learning via Shared Features , 2012, J. Mach. Learn. Res..

[24]  Eric Eaton,et al.  Autonomous Cross-Domain Knowledge Transfer in Lifelong Policy Gradient Reinforcement Learning , 2015, IJCAI.

[25]  Elliot Meyerson,et al.  On the Cross-Domain Reusability of Neural Modules for General Video Game Playing , 2015, CGW/GIGA@IJCAI.

[26]  Risto Miikkulainen,et al.  A Neuroevolution Approach to General Atari Game Playing , 2014, IEEE Transactions on Computational Intelligence and AI in Games.

[27]  Risto Miikkulainen,et al.  Solving Non-Markovian Control Tasks with Neuro-Evolution , 1999, IJCAI.

[28]  Risto Miikkulainen,et al.  Robust non-linear control through neuroevolution , 2003 .

[29]  Samarth Swarup,et al.  Cross-Domain Knowledge Transfer Using Structured Representations , 2006, AAAI.

[30]  Michael L. Anderson.  Neural reuse: A fundamental organizational principle of the brain , 2010, Behavioral and Brain Sciences.

[31]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.