Using reinforcement learning to optimize occupant comfort and energy usage in HVAC systems

The present paper suggests a procedure to enhance the operation of the heating, ventilation and air conditioning system, following the idea that a multi-objective optimal supervisory control for such a system should consider the cost of energy, activity schedules, occupancy patterns and the individual comfort preferences of each tenant. Considering that tenants tend to forget to adjust systems appropriately and that, in many spaces, the conditioning requirements are not adjusted to the occupancy of those spaces, the result is unnecessary energy waste. This paper studies the application of a discrete and a continuous reinforcement-learning-based supervisory control approach, which actively learns how to appropriately schedule thermostat temperature setpoints. The result is a learning controller that learns the statistical regularities in the tenant's behavior, allowing him/her to meet comfort requirements and optimize energy costs. Results are presented for a simulated thermal zone and tenant.

[1]  Mglc Marcel Loomans,et al.  Assessment of thermal comfort in relation to applied low exergy systems - The design of a climate chamber and the use of a thermophysiological model , 2010 .

[2]  Rod Janssen,et al.  Towards Energy Efficient Buildings in Europe , 2004 .

[3]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.

[4]  Edward Henry Mathews,et al.  Developing cost efficient control strategies to ensure optimal energy use and sufficient indoor comfort , 2000 .

[5]  Johnny Wong,et al.  Evaluating the system intelligence of the intelligent building systems: Part 2: Construction and validation of analytical models , 2008 .

[6]  Yoseba K. Penya,et al.  Short-term load forecasting in air-conditioned non-residential Buildings , 2011, 2011 IEEE International Symposium on Industrial Electronics.

[7]  Servet Soyguder,et al.  An expert system for the humidity and temperature control in HVAC systems using ANFIS and optimization with Fuzzy Modeling Approach , 2009 .

[8]  P. O. Fanger,et al.  Thermal comfort: analysis and applications in environmental engineering, , 1972 .

[9]  Alexander Zelinsky,et al.  Q-Learning in Continuous State and Action Spaces , 1999, Australian Joint Conference on Artificial Intelligence.

[10]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[11]  Kent Larson,et al.  Adding GPS-Control to Traditional Thermostats: An Exploration of Potential Energy Savings and Design Challenges , 2009, Pervasive.

[12]  Alan Meier,et al.  Thermostat Interface and Usability: A Survey , 2011 .

[13]  Rafael Alcalá,et al.  Fuzzy Control of HVAC Systems Optimized by Genetic Algorithms , 2003, Applied Intelligence.

[14]  Ueli Rutishauser,et al.  Control and learning of ambience by an intelligent building , 2005, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[15]  A. Omer.  Energy, environment and sustainable development , 2008 .

[16]  Juan Carlos Augusto,et al.  Learning patterns in ambient intelligence environments: a survey , 2010, Artificial Intelligence Review.

[17]  Anastasios I. Dounis,et al.  Advanced control systems engineering for energy and comfort management in a building environment--A review , 2009 .

[18]  Richard S. Sutton,et al.  Introduction to Reinforcement Learning , 1998 .

[19]  van J Joost Hoof,et al.  Forty years of Fanger’s model of thermal comfort: comfort for all? , 2008 .

[20]  Paola Britos,et al.  Optimizing Building's Environments Performance Using Intelligent Systems , 2008, IEA/AIE.

[21]  Richard S. Sutton,et al.  Learning to predict by the methods of temporal differences , 1988, Machine Learning.

[22]  M.A. Wiering,et al.  Reinforcement Learning in Continuous Action Spaces , 2007, 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning.

[23]  Andreas Krause,et al.  Intelligent light control using sensor networks , 2005, SenSys '05.

[24]  Johnny Wong,et al.  Evaluating the system intelligence of the intelligent building systems : Part 1 : Development of key intelligent indicators and conceptual analytical framework , 2008 .

[25]  Cesar A. Hernandez,et al.  Optimization of the Use of Residential Lighting with Neural Network , 2010, 2010 International Conference on Computational Intelligence and Software Engineering.

[26]  David H. Ackley.  Associative Learning via Inhibitory Search , 1988, NIPS.

[27]  Plamen Angelov.  A fuzzy approach to building thermal systems optimization. , 1999 .

[28]  H.-B. Kuntze,et al.  A new fuzzy-based supervisory control concept for the demand-responsive optimization of HVAC control systems , 1998, Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No.98CH36171).

[29]  Konstantinos Dalamagkidis,et al.  Reinforcement Learning for Building Environmental Control , 2008 .

[30]  V. I. Hanby,et al.  System optimization for HVAC energy management using the robust evolutionary algorithm. , 2009 .

[31]  Peter Boait,et al.  A method for fully automatic operation of domestic heating. , 2010 .

[32]  Shengwei Wang,et al.  Intelligent building research: a review , 2005 .

[33]  Jelena Godjevac,et al.  Comparative Study of Fuzzy Control, Neural Network Control and Neuro-Fuzzy Control , 1995 .

[34]  M. Kintner-Meyer,et al.  Optimal control of an HVAC system using cold storage and building thermal capacitance , 1995 .

[35]  A. Harry Klopf,et al.  Reinforcement Learning: An Alternative Approach to Machine Intelligence , 1996 .

[36]  P. Anandan,et al.  Pattern-recognizing stochastic learning automata , 1985, IEEE Transactions on Systems, Man, and Cybernetics.

[37]  Francisco Herrera,et al.  A genetic rule weighting and selection process for fuzzy control of heating, ventilating and air conditioning systems , 2005, Eng. Appl. Artif. Intell..

[38]  Shengwei Wang,et al.  Optimal and robust control of outdoor ventilation airflow rate for improving energy efficiency and IAQ , 2004 .

[39]  D Velimir Congradac,et al.  CONTROL OF THE LIGHTING SYSTEM USING A GENETIC ALGORITHM , 2012 .

[40]  Leemon C Baird,et al.  Reinforcement Learning With High-Dimensional, Continuous Actions , 1993 .

[41]  H. R. Berenji,et al.  Fuzzy Logic Controllers , 1992 .

[42]  D. Subbaram Naidu,et al.  Advanced control strategies for HVAC&R systems—An overview: Part II: Soft and fusion control , 2011 .

[43]  Therese Peffer,et al.  How people use thermostats in homes: A review , 2011, Building and Environment.

[44]  Robert B. Allen.  Developing agent models with a neural reinforcement technique , 1989, Conference Proceedings., IEEE International Conference on Systems, Man and Cybernetics.

[45]  Luis Onieva,et al.  New model for the search for comfort through surveys , 2012 .

[46]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[47]  Mo-Yuen Chow,et al.  Application of functional link neural network to HVAC thermal dynamic system identification , 1998, IEEE Trans. Ind. Electron..

[48]  R. Garcia-Martinez,et al.  Fuzzy Control For Improving Energy Management Within Indoor Building Environments , 2007, Electronics, Robotics and Automotive Mechanics Conference (CERMA 2007).

[49]  Michael A. Humphreys,et al.  Thermal comfort temperatures and the habits of Hobbits , 2015 .