Online learning and optimization of Markov jump linear models

The problem of online learning and optimization of unknown Markov jump linear models is considered. A new online learning algorithm, referred to as Markovian simultaneous perturbations stochastic approximation (MSPSA), is proposed. It is shown that ν/ MSPSA achieves the minimax regret order of Θ(√T). Using the Van Trees inequality (stochastic Cramér-Rao bound), it is shown ν/ that Θ(√T) is the lowest regret order achievable. Simulation results show scenarios that MSPSA offers significant gain over the greedy certainty equivalent approaches.