Data-Efficient Learning of Robotic Grasps From Human Preferences

The ability to grasp various types of objects from the environment is an important manipulation skill for autonomous robots. It poses a prerequisite for solving many real-world tasks, ranging from object handover to household tasks such as clearing the dish washer. However, the huge variety of objects in our environment presents a major obstacle for robots attempting to acquire grasping skills that match the abilities of humans. Although the field of robotic grasping has received great attention during the last decades, grasping arbitrary objects therefore remains a challenging task. In contrast to earlier work that was often restricted to analytic approaches in simulation, reinforcement learning techniques allow to learn from grasp experience by trial and error. While the use of reinforcement learning approaches has led to promising results, defining a reward function is still an open problem. Prior work learned a reward model from human feedback for a single grasp type. However, to capture the whole variety of objects in the real world, it is necessary to learn grasping motions across different grasp types. The goal of this thesis is to devise a reinforcement learning approach that enables the robot to learn how to grasp objects across different grasp types without prior knowledge of the reward function. Because robot operation is both time-consuming and costly, the algorithm should additionally be data-efficient. To achieve this goal, we leverage human feedback to learn multiple grasp policies using a hierarchical reinforcement learning approach. On the upper level, the robot chooses a grasp type and location based on the predicted reward of the resulting trajectory. Subsequently, a grasping motion is generated by the lower-level policy of the selected grasp type. In order to evaluate the outcome of a grasp, we introduce an additional reward model learned entirely from preference feedback. Bayesian optimization and active learning techniques are employed to reduce the number of feedback requests and achieve data-efficient learning. We examined the usefulness of our approach in various experiments on a ball throwing toy task and a simulated grasping task. We showed that learning an outcome reward model from preferences can improve the performance of the system. Moreover, we were able to reduce the amount of required feedback significantly by introducing an active learning criterion. When applied to robotic grasping, our approach was able to learn how to grasp few known objects using different grasp types. However, it failed when more objects were added to the task. Future work could improve the modeling of the reward function or the generation of grasp locations. Furthermore, additional experiments on a real robotic system are required to evaluate the performance of the proposed approach.