Incorporating Advice into Neuroevolution of Adaptive Agents

Neuroevolution is a promising learning method in tasks with extremely large state and action spaces and hidden states. Recent advances allow neuroevolution to take place in real time, making it possible to e.g. construct video games with adaptive agents. Often some of the desired behaviors for such agents are known, and it would make sense to prescribe them, rather than requiring evolution to discover them. This paper presents a technique for incorporating human-generated advice in real time into neuroevolution. The advice is given in a formal language and converted to a neural network structure through KBANN. The NEAT neuroevolution method then incorporates the structure into existing networks through evolution of network weights and topology. The method is evaluated in the NERO video game, where it makes learning faster even when the tasks change and novel ways of making use of the advice are required. Such ability to incorporate human knowledge into neuroevolution in real time may prove useful in several interactive adaptive domains in the future.

[1]  Jude W. Shavlik,et al.  Creating Advice-Taking Reinforcement Learners , 1998, Machine Learning.

[2]  Risto Miikkulainen,et al.  Active Guidance for a Finless Rocket Using Neuroevolution , 2003, GECCO.

[3]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[4]  David E. Goldberg,et al.  Genetic Algorithms in Search Optimization and Machine Learning , 1988 .

[5]  Christian Igel,et al.  Neuroevolution for reinforcement learning using evolution strategies , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[6]  John McCarthy,et al.  Programs with common sense , 1960 .

[7]  Devika Subramanian,et al.  A Multistrategy Learning Scheme for Agent Knowledge Acquisition , 1993, Informatica.

[8]  Risto Miikkulainen,et al.  Robust non-linear control through neuroevolution , 2003 .

[9]  Philip Klahr,et al.  Advice-Taking and Knowledge Refinement: An Iterative View of Skill Acquisition , 1980 .

[10]  Risto Miikkulainen,et al.  Evolving Keepaway Soccer Players through Task Decomposition , 2003, GECCO.

[11]  David E. Goldberg,et al.  Genetic Algorithms with Sharing for Multimodalfunction Optimization , 1987, ICGA.

[12]  David B. Fogel,et al.  Evolution, neural networks, games, and intelligence , 1999, Proc. IEEE.

[13]  David C. Noelle.  A Connectionist Model of Instructed Learning , 1996, AAAI/IAAI, Vol. 2.

[14]  Dario Floreano,et al.  Evolutionary robots with on-line self-organization and behavioral fitness , 2000, Neural Networks.

[15]  X. Yao.  Evolving Artificial Neural Networks , 1999 .

[16]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[17]  Risto Miikkulainen,et al.  Competitive Coevolution through Evolutionary Complexification , 2011, J. Artif. Intell. Res..

[18]  Risto Miikkulainen,et al.  Real-time neuroevolution in the NERO video game , 2005, IEEE Transactions on Evolutionary Computation.