Reinforcement learning (RL) algorithms attempt to assign the credit for rewards to the actions that contributed to the reward. Thus far, credit assignment has been done in one of two ways: uniformly, or using a discounting model that assigns exponentially more credit to recent actions. This paper demonstrates an alternative approach to temporal credit assignment, taking advantage of exact or approximate prior information about correct credit assignment. Infinite impulse response (IIR) filters are used to model credit assignment information. IIR filters generalise exponentially discounting eligibility traces to arbitrary credit assignment models. This approach can be applied to any RL algorithm that employs an eligibility trace. The use of IIR credit assignment filters is explored using both the GPOMDP policy-gradient algorithm and the Sarsa(λ) temporal-difference algorithm. A drop in bias and variance of value or gradient estimates is demonstrated, resulting in faster convergence to better policies.
[1]
Ben J. A. Kröse,et al.
Learning from delayed rewards
,
1995,
Robotics Auton. Syst..
[2]
Richard S. Sutton,et al.
Reinforcement Learning with Replacing Eligibility Traces
,
2005,
Machine Learning.
[3]
Richard S. Sutton,et al.
Introduction to Reinforcement Learning
,
1998
.
[4]
Stephen J. Elliott,et al.
Signal Processing for Active Control
,
2000
.
[5]
Peter L. Bartlett,et al.
Infinite-Horizon Policy-Gradient Estimation
,
2001,
J. Artif. Intell. Res..
[6]
M. Boroujerdi.
Pharmacokinetics : Principles and Applications
,
2001
.
[7]
Douglas Aberdeen,et al.
Policy-Gradient Algorithms for Partially Observable Markov Decision Processes
,
2003
.
[8]
Ronald J. Williams,et al.
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
,
2004,
Machine Learning.
[9]
Richard S. Sutton,et al.
Reinforcement Learning: An Introduction
,
1998,
IEEE Trans. Neural Networks.