On Reinforcement Learning of Control Actions in Noisy and Non-Markovian Domains

If reinforcement learning RL techniques are to be used for real world dynamic system control the problems of noise and plant disturbance will have to be addressed This study investigates the e ects of noise disturbance on ve di erent RL algorithms Watkins Q Learning QL Barto Sutton and Ander son s Adaptive Heuristic Critic AHC Sammut and Law s modern variant of Michie and Chamber s BOXES algorithm and two new algorithms developed during the course of this study Both these new algorithms are conceptually re lated to QL both algorithms called P Trace and Q Trace respectively provide for substantially faster learning than straight QL overall and for dramatically faster learning by up to a factor of in the special case of learning in a noisy environment for the dynamic system studied here a pole and cart simulation As well as speeding learning both the P Trace and Q Trace algorithms have been designed to preserve the convergence with probability formal properties of standard QL i e that they be provably correct algorithms for Markovian domains for the same conditions that QL is guaranteed to be correct We present both arguments and experimental evidence that trace methods may prove to be both faster and more powerful in general than TD Temporal Di erence methods The potential performance improvements using trace over pure TD methods may turn out to be particularly important when learning is to occur in noisy or stochastic environments and in the case where the domain is not well modelled by Markovian processes A surprising result to emerge from this study is evidence for hitherto un suspected chaotic behaviour with respect to learning rates exhibited by the well studied AHC algorithm The e ect becomes more pronounced as noise increases