A Neuroevolution Approach to General

This paper addresses the challenge of learning to play many different video games with little domain-specific knowledge. Specifically, it introduces a neuroevolution approach to general Atari 2600 game playing. Four neuroevolution algo- rithms were paired with three different state representations and evaluated on a set of 61 Atari games. The neuroevolution agents represent different points along the spectrum of algorithmic sophistication—including weight evolution on topologically fixed neural networks (conventional neuroevolution), covariance ma- trix adaptation evolution strategy (CMA-ES), neuroevolution of augmenting topologies (NEAT), and indirect network encoding (HyperNEAT). State representations include an object represen- tation of the game screen, the raw pixels of the game screen, and seeded noise (a comparative baseline). Results indicate that di- rect-encoding methods work best on compact state representations while indirect-encoding methods (i.e., HyperNEAT) allow scaling to higher dimensional representations (i.e., the raw game screen). Previous approaches based on temporal-difference (TD) learning had trouble dealing with the large state spaces and sparse reward gradients often found in Atari games. Neuroevolution ameliorates these problems and evolved policies achieve state-of-the-art re- sults, even surpassing human high scores on three games. These results suggest that neuroevolution is a promising approach to general video game playing (GVGP).