Learning to overtake in TORCS using simple reinforcement learning

In modern racing games programming non-player characters with believable and sophisticated behaviors is getting increasingly challenging. Recently, several works in the literature suggested that computational intelligence can provide effective solutions to support the development process of NPCs. In this paper, we applied the Behavior Analysis and Training (BAT) methodology to define a behavior-based architecture for the NPCs in The Open Racing Car Simulator (TORCS), a well-known open source racing game. Then, we focused on two major overtaking behaviors: (i) the overtaking of a fast opponent either on a straight stretch or on a large bend; (ii) the overtaking on a tight bend, which typically requires a rather advanced braking policy. We applied simple reinforcement learning, namely Q-learning, to learn both these overtaking behaviors. We tested our approach in several overtaking situations and compared the learned behaviors against one of the best NPC provided with TORCS. Our results suggest that, exploiting the proposed behavior-based architecture, Q-learning can effectively learn sophisticated behaviors and outperform programmed NPCs. In addition, we also show that the same approach can be successfully applied to adapt a previously learned behavior to a dynamically changing game situation.

[1]  Julian Togelius,et al.  The 2007 IEEE CEC simulated car racing competition , 2008, Genetic Programming and Evolvable Machines.

[2]  Julian Togelius,et al.  Arms Races and Car Races , 2006, PPSN.

[3]  Larry D. Pyeatt,et al.  Learning to Race: Experiments with a Simulated Race Car , 1998, FLAIRS.

[4]  Araceli Sanchis,et al.  Controller for TORCS created by imitation , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[5]  Julian Togelius,et al.  Evolving robust and specialized car racing skills , 2006, 2006 IEEE International Conference on Evolutionary Computation.

[6]  Jonathan M. Garibaldi,et al.  A fuzzy approach for the 2007 CIG simulated car racing competition , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[7]  Marco Colombetti,et al.  Robot Shaping: An Experiment in Behavior Engineering , 1997 .

[8]  Risto Miikkulainen,et al.  Neuroevolution of an automobile crash warning system , 2005, GECCO '05.

[9]  Martin V. Butz,et al.  Optimized sensory-motor couplings plus strategy extensions for the TORCS car racing challenge , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[10]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[11]  Marc Ebner,et al.  Evolving driving controllers using Genetic Programming , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[12]  Daniele Loiacono,et al.  On-line neuroevolution applied to The Open Racing Car Simulator , 2009, 2009 IEEE Congress on Evolutionary Computation.

[13]  Daniele Loiacono,et al.  On-line and Off-line Learning of Driving Tasks for The Open Racing Car Simulator (TORCS) Using Neuroevolution , 2008 .

[14]  Julian Togelius,et al.  Competitions @ WCCI-2008: simulated car racing competition , 2007, SEVO.

[15]  Pedro Isasi Viñuela,et al.  Evolving a fuzzy controller for a Car Racing Competition , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[16]  Peter J. Bentley,et al.  Optimising the Performance of a Formula One Car Using a Genetic Algorithm , 2004, PPSN.

[17]  Vicente Milanés Montero,et al.  A modular parametric architecture for the TORCS racing engine , 2009, 2009 IEEE Symposium on Computational Intelligence and Games.

[18]  Julian Togelius,et al.  Evolving controllers for simulated car racing , 2005, 2005 IEEE Congress on Evolutionary Computation.

[19]  Colin Fyfe,et al.  Improving Artificial Intelligence In a Motocross Game , 2006, 2006 IEEE Symposium on Computational Intelligence and Games.

[20]  Julian Togelius,et al.  The WCCI 2008 simulated car racing competition , 2008, 2008 IEEE Symposium On Computational Intelligence and Games.

[21]  Daniele Loiacono,et al.  Evolving competitive car controllers for racing games with neuroevolution , 2009, GECCO '09.