Evolving visibly intelligent behavior for embedded game agents

Machine learning has proven useful for producing solutions to various problems, including the creation of controllers for autonomous intelligent agents. However, the control requirements for an intelligent agent sometimes go beyond the simple ability to complete a task, or even to complete it efficiently: An agent must sometimes complete a task in style. For example, if an autonomous intelligent agent is embedded in a game where it is visible to human observers, and plays a role that evokes human intuitions about how that role should be fulfilled, then the agent must fulfill that role in a manner that does not dispel the illusion of intelligence for the observers. Such visibly intelligent behavior is a subset of general intelligent behavior: a subset that we must be able to provide if our methods are to be adopted by the developers of games and simulators. This dissertation continues the tradition of using neuroevolution to train artificial neural networks as controllers for agents embedded in strategy games or simulators, expanding that work to address selected issues of visibly intelligent behavior. A test environment is created and used to demonstrate that modified methods can create desirable behavioral traits such as flexibility, consistency, and adherence to a doctrine, and suppress undesirable traits such as seemingly erratic behavior and excessive predictability. These methods are designed to expand a program of work leading toward adoption of neuroevolution by the commercial gaming industry, increasing player satisfaction with their products, and perhaps helping to set AI forward as The Next Big Thing in that industry. As the capabilities of research-grade machine learning converge with the needs of the commercial gaming industry, work of this sort can be expected to expand into a broad and productive area of research into the nature of intelligence and the behavior of autonomous agents.

[1]  J. Baldwin.  A New Factor in Evolution , 1896, The American Naturalist.

[2]  Claude E. Shannon,et al.  Programming a computer for playing chess , 1950 .

[3]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[4]  Arthur L. Samuel,et al.  Some Studies in Machine Learning Using the Game of Checkers , 1967, IBM J. Res. Dev..

[5]  Marvin Minsky,et al.  Perceptrons: An Introduction to Computational Geometry , 1969 .

[6]  Nils J. Nilsson,et al.  Artificial Intelligence , 1974, IFIP Congress.

[7]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[8]  R. Pagano.  Understanding Statistics in the Behavioral Sciences , 1981 .

[9]  Andrew W. Appel,et al.  Rog-O-Matic : a belligerent expert system , 1983 .

[10]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[11]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[12]  Terrence J. Sejnowski,et al.  A 'Neural' Network that Learns to Play Backgammon , 1987, NIPS.

[13]  Geoffrey E. Hinton,et al.  How Learning Can Guide Evolution , 1996, Complex Syst..

[14]  Lawrence Davis,et al.  Training Feedforward Neural Networks Using Genetic Algorithms , 1989, IJCAI.

[15]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[16]  Paul J. Werbos,et al.  Backpropagation Through Time: What It Does and How to Do It , 1990, Proc. IEEE.

[17]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[18]  John J. Grefenstette,et al.  Improving tactical plans with genetic algorithms , 1990, [1990] Proceedings of the 2nd International IEEE Conference on Tools for Artificial Intelligence.

[19]  Kenneth J. Hintz,et al.  Evolving a neural network , 1990, Proceedings. 5th IEEE International Symposium on Intelligent Control 1990.

[20]  Michael I. Jordan,et al.  Hierarchies of Adaptive Experts , 1991, NIPS.

[21]  R. A. Brooks,et al.  Intelligence without Representation , 1991, Artif. Intell..

[22]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[23]  John J. Grefenstette,et al.  Lamarckian Learning in Multi-Agent Environments , 1991, ICGA.

[24]  John J. Grefenstette,et al.  Using a Genetic Algorithm to Learn Behaviors for Autonomous Vehicles , 1992 .

[25]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[26]  Marco Colombetti,et al.  Robot shaping: developing situated agents through learning , 1992 .

[27]  J. D. Schaffer,et al.  Combinations of genetic algorithms and neural networks: a survey of the state of the art , 1992, [Proceedings] COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks.

[28]  John H. Holland,et al.  Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence , 1992 .

[29]  John J. Grefenstette,et al.  Case-Based Initialization of Genetic Algorithms , 1993, ICGA.

[30]  Thomas Ellman,et al.  Abstraction via Approximate Symmetry , 1993, IJCAI.

[31]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[32]  John A. Biles,et al.  GenJam: A Genetic Algorithm for Generating Jazz Solos , 1994, ICMC.

[33]  Gerald Tesauro,et al.  TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play , 1994, Neural Computation.

[34]  Jude W. Shavlik,et al.  Knowledge-Based Artificial Neural Networks , 1994, Artif. Intell..

[35]  Hava T. Siegelmann,et al.  Analog computation via neural networks , 1993, [1993] The 2nd Israel Symposium on Theory and Computing Systems.

[36]  Hervé Bourlard,et al.  Connectionist probability estimators in HMM speech recognition , 1994, IEEE Trans. Speech Audio Process..

[37]  Joseph Bates,et al.  The role of emotion in believable agents , 1994, CACM.

[38]  Edmund H. Durfee,et al.  Blissful Ignorance: Knowing Just Enough to Coordinate Well , 1995, ICMAS.

[39]  Claude Sammut,et al.  A Framework for Behavioural Cloning , 1995, Machine Intelligence 15.

[40]  Maja J. Mataric,et al.  Designing and Understanding Adaptive Group Behavior , 1995, Adapt. Behav..

[41]  David B. Fogel,et al.  Evolutionary Computation: Towards a New Philosophy of Machine Intelligence , 1995 .

[42]  Hava T. Siegelmann,et al.  On the Computational Power of Neural Nets , 1995, J. Comput. Syst. Sci..

[43]  Risto Miikkulainen,et al.  Discovering Complex Othello Strategies through Evolutionary Neural Networks , 1995, Connect. Sci..

[44]  Risto Miikkulainen,et al.  Learning Sequential Decision Tasks , 1995 .

[45]  Yiannis Aloimonos,et al.  Artificial intelligence - theory and practice , 1995 .

[46]  Risto Miikkulainen,et al.  Evolving Obstacle Avoidance Behavior in a Robot Arm , 1996 .

[47]  Risto Miikkulainen,et al.  Subsymbolic Case-Role Analysis of Sentences With Embedded Clauses , 1993, Cogn. Sci..

[48]  Jordan B. Pollack,et al.  Coevolution of a Backgammon Player , 1996 .

[49]  Gillian M. Hayes,et al.  Robot Shaping --- Principles, Methods and Architectures , 1996 .

[50]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[51]  Ah Chung Tsoi,et al.  On the distribution of performance from multiple neural-network trials , 1997, IEEE Trans. Neural Networks.

[52]  Hiroaki Kitano,et al.  RoboCup: The Robot World Cup Initiative , 1997, AGENTS '97.

[53]  David E. Moriarty,et al.  Symbiotic Evolution of Neural Networks in Sequential Decision Tasks , 1997 .

[54]  Risto Miikkulainen,et al.  Incremental Evolution of Complex General Behavior , 1997, Adapt. Behav..

[55]  Tucker Balch,et al.  Learning Roles: Behavioral Diversity in Robot Teams , 1997 .

[56]  C. Lee Giles,et al.  Presenting and Analyzing the Results of AI Experiments: Data Averaging and Data Snooping , 1997, AAAI/IAAI.

[57]  Thomas Bäck,et al.  Evolutionary computation: Toward a new philosophy of machine intelligence , 1997, Complex..

[58]  Scott Hamilton,et al.  Deep Blue's Hardware-Software Synergy , 1997, Computer.

[59]  Sandip Sen,et al.  Co-adaptation in a Team , 1997 .

[60]  Maja J. Mataric,et al.  Interference as a Tool for Designing and Evaluating Multi-Robot Controllers , 1997, AAAI/IAAI.

[61]  Takuji Nishimura,et al.  Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator , 1998, TOMC.

[62]  Brian Scassellati,et al.  Alternative Essences of Intelligence , 1998, AAAI/IAAI.

[63]  Risto Miikkulainen,et al.  2-D Pole Balancing with Recurrent Evolutionary Networks , 1998 .

[64]  R. Arkin,et al.  Behavioral diversity in learning robot teams , 1998 .

[65]  S. Pattinson,et al.  Learning to fly. , 1998 .

[66]  B. Habibi,et al.  Pengi : An Implementation of A Theory of Activity , 1998 .

[67]  Maria Fox,et al.  The Detection and Exploitation of Symmetry in Planning Problems , 1999, IJCAI.

[68]  X. Yao.  Evolving Artificial Neural Networks , 1999 .

[69]  Risto Miikkulainen,et al.  Solving Non-Markovian Control Tasks with Neuro-Evolution , 1999, IJCAI.

[70]  Barbara Hayes-Roth,et al.  Staffing the web with interactive characters , 1999, CACM.

[71]  Gerhard Weiss,et al.  Multiagent Systems , 1999 .

[72]  Manuela M. Veloso,et al.  Multiagent Systems: A Survey from a Machine Learning Perspective , 2000, Auton. Robots.

[73]  Wan-Chi Siu,et al.  A study of the Lamarckian evolution of recurrent neural networks , 2000, IEEE Trans. Evol. Comput..

[74]  Andrew Y. Ng,et al.  Pharmacokinetics of a novel formulation of ivermectin after administration to goats , 2000, ICML.

[75]  Manuela M. Veloso,et al.  Layered Learning , 2000, ECML.

[76]  Jette Randløv,et al.  Shaping in Reinforcement Learning by Changing the Physics of the Problem , 2000, ICML.

[77]  Graham Kendall,et al.  An evolutionary approach for the tuning of a chess evaluation function using population dynamics , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[78]  Tucker R. Balch,et al.  Symmetry in Markov Decision Processes and its Implications for Single Agent and Multiagent Learning , 2001, ICML.

[79]  Johannes Fürnkranz,et al.  Machine learning in games: a survey , 2001 .

[80]  T. M. English.  Learning to focus selectively on possible lines of play in checkers , 2001, Proceedings of the 2001 Congress on Evolutionary Computation (IEEE Cat. No.01TH8546).

[81]  James L. Carroll,et al.  Towards automatic shaping in robot navigation , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[82]  Risto Miikkulainen,et al.  COOPERATIVE COEVOLUTION OF MULTI-AGENT SYSTEMS , 2001 .

[83]  John E. Laird,et al.  Human-Level AI's Killer Application: Interactive Computer Games , 2000, AI Mag..

[84]  Alex Lubberts and Risto Miikkulainen.  Co-Evolving a Go-Playing Neural network , 2001 .

[85]  John E. Laird,et al.  Learning procedural knowledge through observation , 2001, K-CAP '01.

[86]  Shrikanth Narayanan,et al.  Limited domain synthesis of expressive military speech for animated characters , 2002, Proceedings of 2002 IEEE Workshop on Speech Synthesis, 2002..

[87]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[88]  Christian Igel,et al.  Neuroevolution for reinforcement learning using evolution strategies , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[89]  Risto Miikkulainen,et al.  Neuroevolution for adaptive teams , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[90]  Risto Miikkulainen,et al.  Active Guidance for a Finless Rocket Using Neuroevolution , 2003, GECCO.

[91]  Risto Miikkulainen,et al.  Evolving adaptive neural networks with and without adaptive synapses , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[92]  C. Boutilier,et al.  Accelerating Reinforcement Learning through Implicit Imitation , 2003, J. Artif. Intell. Res..

[93]  Siang Yew Chong,et al.  Evolved neural networks learning Othello strategies , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[94]  Maja J. Matarić,et al.  A framework for learning from demonstration, generalization and practice in human-robot domains , 2003 .

[95]  Risto Miikkulainen,et al.  Robust non-linear control through neuroevolution , 2003 .

[96]  Michael C. Fu,et al.  Guest editorial , 2003, TOMC.

[97]  Marcus Gallagher,et al.  Learning to play Pac-Man: an evolutionary, rule-based approach , 2003, The 2003 Congress on Evolutionary Computation, 2003. CEC '03..

[98]  L. D. Whitley,et al.  Genetic Reinforcement Learning for Neurocontrol Problems , 2004, Machine Learning.

[99]  Simon M. Lucas,et al.  Cellz: a simple dynamic game for testing evolutionary algorithms , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[100]  Risto Miikkulainen,et al.  Evolving Neural Networks to Play Go , 2004, Applied Intelligence.

[101]  Graham Kendall,et al.  An investigation of an evolutionary approach to the opening of Go , 2004, Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753).

[102]  Risto Miikkulainen,et al.  Online Interactive Neuro-evolution , 2000, Neural Processing Letters.

[103]  Risto Miikkulainen,et al.  Competitive Coevolution through Evolutionary Complexification , 2011, J. Artif. Intell. Res..

[104]  Stacy Marsella,et al.  A domain-independent framework for modeling emotion , 2004, Cognitive Systems Research.

[105]  Leo Breiman,et al.  Bagging Predictors , 1996, Machine Learning.

[106]  Gregory Kuhlmann and Peter Stone and Raymond J. Mooney and Shavlik.  Guiding a Reinforcement Learner with Natural Language Advice: Initial Results in RoboCup Soccer , 2004, AAAI 2004.

[107]  Risto Miikkulainen,et al.  Evolving a Roving Eye for Go , 2004, GECCO.

[108]  Chern Han Yong and Kenneth O. Stanley and Risto Miikkulainen.  Incorporating Advice into Evolution of Neural Networks , 2005 .

[109]  Aliza Gold.  Academic AI and Videogames: A Case Study of Incorporating Innovative Academic Research Into a Videogame Prototype , 2005, CIG.

[110]  Risto Miikkulainen,et al.  Real-time neuroevolution in the NERO video game , 2005, IEEE Transactions on Evolutionary Computation.

[111]  Julian Togelius,et al.  Forcing Neurocontrollers to Exploit Sensory Symmetry Through Hard-wired Modularity in the Game of Cellz , 2005, CIG.

[112]  Raymond J. Mooney,et al.  Creating diversity in ensembles using artificial data , 2005, Inf. Fusion.

[113]  Simon M. Lucas,et al.  Evolving a Neural Network Location Evaluator to Play Ms. Pac-Man , 2005, CIG.

[114]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[115]  Stacy Marsella,et al.  Evaluating a Computational Model of Emotion , 2005, Autonomous Agents and Multi-Agent Systems.

[116]  Risto Miikkulainen,et al.  Retaining Learned Behavior During Real-Time Neuroevolution , 2005, AIIDE.

[117]  David B. Fogel,et al.  Further Evolution of a Self-Learning Chess Program , 2005, CIG.

[118]  Sushil J. Louis,et al.  Playing to learn: case-injected genetic algorithms for learning to play computer games , 2005, IEEE Transactions on Evolutionary Computation.

[119]  Jude W. Shavlik,et al.  Creating Advice-Taking Reinforcement Learners , 1998, Machine Learning.

[120]  Kenneth O. Stanley and Bobby D. Bryant and Risto Miikkulainen,et al.  Evolving Neural Network Agents in the NERO Video Game , 2005 .

[121]  Shimon Whiteson,et al.  Evolutionary Function Approximation for Reinforcement Learning , 2006, J. Mach. Learn. Res..

[122]  Mitchell A. Potter,et al.  EVOLVING NEURAL NETWORKS WITH COLLABORATIVE SPECIES , 2006 .

[123]  Risto Miikkulainen,et al.  Exploiting Sensor Symmetries in Example-based Training for Intelligent Agents , 2006, 2006 IEEE Symposium on Computational Intelligence and Games.

[124]  Risto Miikkulainen,et al.  Evolving Stochastic Controller Networks for Intelligent Game Agents , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[125]  Agostino Poggi,et al.  Multiagent Systems , 2006, Intelligenza Artificiale.