Incremental Evolution of Complex General Behavior

Several researchers have demonstrated how complex action sequences can be learned through neuroevolution (i.e., evolving neural networks with genetic algorithms). However, complex general behavior such as evading predators or avoiding obstacles, which is not tied to specific environments, turns out to be very difficult to evolve. Often the system discovers mechanical strategies, such as moving back and forth, that help the agent cope but are not very effective, do not appear believable, and do not generalize to new environments. The problem is that a general strategy is too difficult for the evolution system to discover directly. This article proposes an approach wherein such complex general behavior is learned incrementally, by starting with simpler behavior and gradually making the task more challenging and general. The task transitions are implemented through successive stages of Delta coding (i.e., evolving modifications), which allows even converged populations to adapt to the new task. The method is tested in the stochastic, dynamic task of prey capture and is compared with direct evolution. The incremental approach evolves more effective and more general behavior and should also scale up to harder tasks.

[1]  Richard S. Sutton,et al.  Neuronlike adaptive elements that can solve difficult learning control problems , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[2]  Rodney A. Brooks,et al.  A Robust Layered Control Syste For A Mobile Robot , 2022 .

[3]  Darrell Whitley,et al.  Genitor: a different genetic algorithm , 1988 .

[4]  C.W. Anderson,et al.  Learning to control an inverted pendulum using neural networks , 1989, IEEE Control Systems Magazine.

[5]  Stewart W. Wilson.  The animat path to AI , 1991 .

[6]  Alexis P. Wieland,et al.  Evolving Controls for Unstable Systems , 1991 .

[7]  L. Darrell Whitley,et al.  Delta Coding: An Iterative Search Strategy for Genetic Algorithms , 1991, ICGA.

[8]  A. P. Wieland,et al.  Evolving neural network controllers for unstable systems , 1991, IJCNN-91-Seattle International Joint Conference on Neural Networks.

[9]  Marco Colombetti,et al.  Robot shaping: developing situated agents through learning , 1992 .

[10]  Satinder Singh.  Transfer of Learning by Composing Solutions of Elemental Sequential Tasks , 1992, Mach. Learn..

[11]  L.-J. Lin,et al.  Hierarchical learning of robot skills by reinforcement , 1993, IEEE International Conference on Neural Networks.

[12]  Marco Colombetti,et al.  Learning to control an autonomous robot by distributed genetic algorithms , 1993 .

[13]  Lorien Y. Pratt,et al.  Transferring previously learned back-propagation neural networks to new learning tasks , 1993 .

[14]  Inman Harvey,et al.  Incremental evolution of neural network architectures for adaptive behavior , 1993, ESANN.

[15]  Randall D. Beer,et al.  Integrating reactive, sequential, and learning behavior using dynamical neural networks , 1994 .

[16]  Mark D. Pendrith.  On Reinforcement Learning of Control Actions in Noisy and Non-Markovian Domains , 1994 .

[17]  Lorien Y. Pratt,et al.  Experiments on the transfer of knowledge between neural networks , 1994, COLT 1994.

[18]  Risto Miikkulainen,et al.  Grounding Robotic Control with Genetic Neural Networks , 1994 .

[19]  Jeffrey L. Elman,et al.  Learning and Evolution in Neural Networks , 1994, Adapt. Behav..

[20]  Mark B. Ring.  Continual learning in reinforcement environments , 1995, GMD-Bericht.

[21]  David B. Fogel,et al.  Evolving Neural Control Systems , 1995, IEEE Expert.

[22]  Risto Miikkulainen,et al.  Efficient Learning from Delayed Rewards through Symbiotic Evolution , 1995, ICML.

[23]  Sebastian Thrun,et al.  Explanation-based neural network learning a lifelong learning approach , 1995 .

[24]  Stefano Nolfi,et al.  Learning to Adapt to Changing Environments in Evolving Neural Networks , 1996, Adapt. Behav..

[25]  Risto Miikkulainen,et al.  Evolving Obstacle Avoidance Behavior in a Robot Arm , 1996 .

[26]  Gillian M. Hayes,et al.  Robot Shaping --- Principles, Methods and Architectures , 1996 .

[27]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[28]  Dave Cliff,et al.  Challenges in evolving controllers for physical robots , 1996, Robotics Auton. Syst..

[29]  David E. Moriarty,et al.  Symbiotic Evolution of Neural Networks in Sequential Decision Tasks , 1997 .

[30]  Risto Miikkulainen,et al.  Hierarchical evolution of neural networks , 1998, 1998 IEEE International Conference on Evolutionary Computation Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98TH8360).

[31]  Stefano Nolfi,et al.  Learning and Evolution , 1999, Auton. Robots.