As the field of reinforcement learning has advanced, interest in solving realistic control problems has increased. However, Markov Decision Process (MDP) models have not proven sufficient to the task. This has led to increased use of Semi-Markov Decision Process models and the development of Hierarchical Reinforcement Learning (HRL). This chapter is an overview of HRL beginning with a discussion of the problems with the standard MDP models, then presenting the theory behind HRL, and finishing with some actual HRL algorithms that have been proposed. To see some examples of how hierarchical methods perform, see Chapter 11.