Efficient credit assignment through evaluation function decomposition

Evolutionary methods are powerful tools in discovering solutions for difficult continuous tasks. When such a solution is encoded over multiple genes, a genetic algorithm faces the difficult credit assignment problem of evaluating how a single gene in a chromosome contributes to the full solution. Typically a single evaluation function is used for the entire chromosome, implicitly giving each gene in the chromosome the same evaluation. This method is inefficient because a gene will get credit for the contribution of all the other genes as well. Accurately measuring the fitness of individual genes in such a large search space requires many trials. This paper instead proposes turning this single complex search problem into a multi-agent search problem, where each agent has the simpler task of discovering a suitable gene. Gene-specific evaluation functions can then be created that have better theoretical properties than a single evaluation function over all genes. This method is tested in the difficult double-pole balancing problem, showing that agents using gene-specific evaluation functions can create a successful control policy in 20% fewer trials than the best existing genetic algorithms. The method is extended to more distributed problems, achieving 95% performance gains over tradition methods in the multi-rover domain.

[1]  Kagan Tumer,et al.  Using Collective Intelligence to Route Internet Traffic , 1998, NIPS.

[2]  Francesco Mondada,et al.  Automatic creation of an autonomous agent: genetic evolution of a neural-network driven robot , 1994 .

[3]  Steven Dubowsky,et al.  A Genetic Algorithm Based Navigation and Planning Methodology for Planetary Robotic Exploration , 1997 .

[4]  Kagan Tumer,et al.  Collectives and Design Complex Systems , 2004 .

[5]  Stefano Nolfi,et al.  Evolving Mobile Robots Able to Display Collective Behaviors , 2003, Artificial Life.

[6]  Kagan Tumer,et al.  Efficient Evaluation Functions for Multi-rover Systems , 2004, GECCO.

[7]  Kagan Tumer,et al.  A Survey of Collectives , 2004 .

[8]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[9]  Hans-Paul Schwefel,et al.  Evolution strategies – A comprehensive introduction , 2002, Natural Computing.

[10]  S. Hyakin,et al.  Neural Networks: A Comprehensive Foundation , 1994 .

[11]  Risto Miikkulainen,et al.  Efficient Reinforcement Learning Through Evolving Neural Network Topologies , 2002, GECCO.

[12]  Risto Miikkulainen,et al.  Active Guidance for a Finless Rocket Using Neuroevolution , 2003, GECCO.

[13]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[14]  Edwin D. de Jong,et al.  Evolutionary Multi-agent Systems , 2004, PPSN.

[15]  Risto Miikkulainen,et al.  Forming Neural Networks Through Efficient and Adaptive Coevolution , 1997, Evolutionary Computation.

[16]  J. Nash.  Equilibrium Points in N-Person Games. , 1950, Proceedings of the National Academy of Sciences of the United States of America.