Meta-learning for Predictive Knowledge Architectures: A Case Study Using TIDBD on a Sensor-rich Robotic Arm

Predictive approaches to modelling the environment have seen recent successes in robotics and other long-lived applications. These predictive knowledge architectures are learned incrementally and online, through interaction with the environment. One challenge for applications of predictive knowledge is the necessity of tuning feature representations and parameter values: no single step size will be appropriate for every prediction. Furthermore, as sensor signals might be subject to change in a non-stationary world, predefined step sizes cannot be sufficient for an autonomous agent. In this paper, we explore Temporal-Difference Incremental Delta-Bar-Delta (TIDBD)-a meta-learning method for temporal-difference (TD) learning which adapts a vector of many step sizes, allowing for simultaneous step size tuning and representation learning. We demonstrate that, for a predictive knowledge application, TIDBD is a viable alternative to tuning step-size parameters, by showing that the performance of TIDBD is comparable to that of TD with an exhaustive parameter search. Performance here is measured in terms of root mean squared difference from the true value, calculated offline. Moreover, TIDBD can perform representation learning, potentially supporting robust learning in the face of failing sensors. The ability for an autonomous agent to adapt its own learning and adjust its representation based on interactions with its environment is a key capability. With its potential to fulfill these desiderata, meta-learning is a promising component for future systems.