Regret Minimization in Nonstationary Markov Decision Processes

We consider decision-making problems in Markov decision processes where both the rewards and the transition probabilities vary in an arbitrary ( e.g., nonstationary) fashion to some extent. We propose online learning algorithms and provide guarantees on their performance evaluated in retrospect against stationary policies. Unlike previous works, the gu arantees depend critically on the variability of the uncertainty in the transition probabilities, but hol d regardless of arbitrary changes in rewards and transition probabilities. First, we use an approach bas ed on robust dynamic programming and extend it to the case where reward observation is limited to the actual state-action trajectory. Next, we present a computationally efficient simulation-based Q-learning style algorithm that requires neither prior knowledge nor estimation of the transition probabilities. We show both probabilistic performance guarantees and deterministic guarantees on the expected performance 1 .