Guided Reinforcement Learning Under Partial Observability

Due to recent breakthroughs in artificial intelligence, reinforcement learning (RL) gained considerable attention in the last years. Especially combining RL with deep neural networks, which could already achieve great success in other research fields, has contributed to these breakthroughs. In many real-world scenarios it is not always possible to perceive the true and complete state of the environment. These scenarios are known as learning under uncertainty or under partial observability. Formulating such problems as partially observable Markov decision processes (POMDPs) allows us to solve decision making processes under uncertainty. Guided RL approaches address this problem by supporting the RL algorithms with additional state information during the learning process to increase their performance for solving POMDPs. However, these guided approaches are relatively rare in literature and most existing approaches are modelbased, meaning that they require to learn an appropriate model of the environment first. For this reason, we will propose a novel model-free guided RL approach, called guided reinforcement learning (GRL). The guidance is mainly based on mixing samples containing full or partial state information while we gradually decrease the amount of full state information during training which ends up with a policy compatible with partial observations. The general formulation of our simple GRL approach allows to combine this approach with a variety of existing model-free RL algorithms as well as a variety of settings where these algorithms can be applied to. We demonstrate that our GRL approach can outperform other baseline algorithms that are trained directly on partial observations on nine different tasks. These tasks include two instances of the well-known discrete action space problem RockSample and the continuous action space problem LunarLander-POMDP which is a partially observable modification of the LunarLanderContinuous-v2 environment. Further, the benchmark tasks also include six partially observable tasks that we have constructed based on continuous control problems, simulated in the MuJoCo physics simulator.