Hierarchical Decision Making In Electricity Grid Management

The power grid is a complex and vital system that necessitates careful reliability management. Managing the grid is a difficult problem with multiple time scales of decision making and stochastic behavior due to renewable energy generations, variable demand and unplanned outages. Solving this problem in the face of uncertainty requires a new methodology with tractable algorithms. In this work, we introduce a new model for hierarchical decision making in complex systems. We apply reinforcement learning (RL) methods to learn a proxy, i.e., a level of abstraction, for real-time power grid reliability. We devise an algorithm that alternates between slow time-scale policy improvement, and fast time-scale value function approximation. We compare our results to prevailing heuristics, and show the strength of our method.

[1]  Ning Lu,et al.  A comparison of forecast error generators for modeling wind and load uncertainty , 2013, 2013 IEEE Power & Energy Society General Meeting.

[2]  Gary W. Chang,et al.  Power System Analysis , 1994 .

[3]  Xu Andy Sun,et al.  Adaptive Robust Optimization for the Security Constrained Unit Commitment Problem , 2013, IEEE Transactions on Power Systems.

[4]  Warren B. Powell,et al.  Approximate Dynamic Programming: Solving the Curses of Dimensionality (Wiley Series in Probability and Statistics) , 2007 .

[5]  B. Ripley,et al.  Pattern Recognition , 1968, Nature.

[6]  Sridhar Mahadevan,et al.  Recent Advances in Hierarchical Reinforcement Learning , 2003, Discret. Event Dyn. Syst..

[7]  Thomas G. Dietterich The MAXQ Method for Hierarchical Reinforcement Learning , 1998, ICML.

[8]  Daniel S. Kirschen,et al.  Near-Optimal Method for Siting and Sizing of Distributed Storage in a Transmission Network , 2015, IEEE Transactions on Power Systems.

[9]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.

[10]  Vincenzo Marano,et al.  A stochastic dynamic programming model for co-optimization of distributed energy storage , 2013, Energy Systems.

[11]  Allen J. Wood,et al.  Power Generation, Operation, and Control , 1984 .

[12]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1971 .

[13]  Shie Mannor,et al.  Reinforcement learning for the unit commitment problem , 2015, 2015 IEEE Eindhoven PowerTech.

[14]  Mohammad Shahidehpour,et al.  The IEEE Reliability Test System-1996. A report prepared by the Reliability Test System Task Force of the Application of Probability Methods Subcommittee , 1999 .

[15]  Michael Chertkov,et al.  Chance-Constrained Optimal Power Flow: Risk-Aware Network Control under Uncertainty , 2012, SIAM Rev..

[16]  Masood Parvania,et al.  A Two-Stage Framework for Power Transformer Asset Maintenance Management—Part I: Models and Formulations , 2013, IEEE Transactions on Power Systems.

[17]  D. Ernst,et al.  The cross-entropy method for power system combinatorial optimization problems , 2007, 2007 IEEE Lausanne Power Tech.

[18]  Can Anil,et al.  Benchmarking of Data Mining Techniques as Applied to Power System Analysis , 2013 .

[19]  Leandros Tassiulas,et al.  Optimal Control Policies for Power Demand Scheduling in the Smart Grid , 2012, IEEE Journal on Selected Areas in Communications.

[20]  R. Buizza,et al.  Neural Network Load Forecasting with Weather Ensemble Predictions , 2002, IEEE Power Engineering Review.

[21]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[22]  Daniel Urieli,et al.  TacTex'13: A Champion Adaptive Power Trading Agent , 2014, AAAI.

[23]  Mohammad Shahidehpour,et al.  Security-Constrained Generation and Transmission Outage Scheduling With Uncertainties , 2010, IEEE Transactions on Power Systems.

[24]  Nicola Secomandi,et al.  An Approximate Dynamic Programming Approach to Benchmark Practice-Based Heuristics for Natural Gas Storage Valuation , 2010, Oper. Res..

[25]  Daniel Bienstock,et al.  Optimal control of cascading power grid failures , 2011, IEEE Conference on Decision and Control and European Control Conference.

[26]  Warren B. Powell,et al.  Handbook of Learning and Approximate Dynamic Programming , 2006, IEEE Transactions on Automatic Control.

[27]  Warren B. Powell,et al.  “Approximate dynamic programming: Solving the curses of dimensionality” by Warren B. Powell , 2007, Wiley Series in Probability and Statistics.

[28]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[29]  M. B. Cain,et al.  History of Optimal Power Flow and Formulations , 2012 .

[30]  Masood Parvania,et al.  A two-stage framework for power transformer asset maintenance management—Part I: Models and formulations , 2013, 2013 IEEE Power & Energy Society General Meeting.

[31]  Patrick Panciatici,et al.  iTesla: Innovative tools for electrical system security within large areas , 2014, 2014 IEEE PES General Meeting | Conference & Exposition.

[32]  Doina Precup,et al.  Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning , 1999, Artif. Intell..

[33]  András Lörincz,et al.  Learning Tetris Using the Noisy Cross-Entropy Method , 2006, Neural Computation.

[34]  Bruno Scherrer,et al.  Classification-based Policy Iteration with a Critic , 2011, ICML.

[35]  Richard P. O'Neill,et al.  History of Optimal Power Flow and Formulations Optimal Power Flow Paper 1 , 2012 .

[36]  Anthony Papavasiliou,et al.  Multiarea Stochastic Unit Commitment for High Wind Penetration in a Transmission Constrained Network , 2013, Oper. Res..

[37]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[38]  N.P. Padhy,et al.  Unit commitment-a bibliographical survey , 2004, IEEE Transactions on Power Systems.

[39]  Warren B. Powell,et al.  Optimal Hour-Ahead Bidding in the Real-Time Electricity Market with Battery Storage Using Approximate Dynamic Programming , 2014, INFORMS J. Comput..

[40]  Warren B. Powell,et al.  A comparison of approximate dynamic programming techniques on benchmark energy storage problems: Does anything work? , 2014, 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL).

[41]  M. Fotuhi-Firuzabad,et al.  An Efficient Mixed-Integer Linear Formulation for Long-Term Overhead Lines Maintenance Scheduling in Power Distribution Systems , 2009, IEEE Transactions on Power Delivery.

[42]  Roy Billinton,et al.  Reliability evaluation of power systems , 1984 .

[43]  Warren B. Powell,et al.  Tutorial on Stochastic Optimization in Energy—Part I: Modeling and Policies , 2016, IEEE Transactions on Power Systems.

[44]  Shie Mannor,et al.  A Tutorial on the Cross-Entropy Method , 2005, Ann. Oper. Res..

[45]  G. Sheblé,et al.  Power generation operation and control — 2nd edition , 1996 .

[46]  Stuart J. Russell,et al.  Reinforcement Learning with Hierarchies of Machines , 1997, NIPS.