Intelligent Residential Energy Management System using Deep
Reinforcement Learning
Alwyn Mathew Abhijit Roy Jimson Mathew
Indian Institute of Technology Patna{alwyn.pcs16, abhijit.cs15, jimson}@iitp.ac.in
Abstract
The rising demand for electricity and its essential na-ture in today’s world calls for intelligent home energymanagement (HEM) systems that can reduce energyusage. This involves scheduling of loads from peakhours of the day when energy consumption is at itshighest to leaner off-peak periods of the day when en-ergy consumption is relatively lower thereby reducingthe system’s peak load demand, which would conse-quently result in lesser energy bills, and improved loaddemand profile. This work introduces a novel way todevelop a learning system that can learn from expe-rience to shift loads from one time instance to an-other and achieve the goal of minimizing the aggregatepeak load. This paper proposes a Deep ReinforcementLearning (DRL) model for demand response where thevirtual agent learns the task like humans do. The agentgets feedback for every action it takes in the environ-ment; these feedbacks will drive the agent to learnabout the environment and take much smarter stepslater in its learning stages. Our method outperformedthe state of the art mixed integer linear programming(MILP) for load peak reduction. The authors havealso designed an agent to learn to minimize both con-sumers’ electricity bills and utilities’ system peak loaddemand simultaneously. The proposed model was an-alyzed with loads from five different residential con-sumers; the proposed method increases the monthlysavings of each consumer by reducing their electricitybill drastically along with minimizing the peak load onthe system when time shiftable loads are handled bythe proposed method.
Keywords: Home Energy Management, Reinforce-ment Learning.
Energy generated from power grids fuels the modernlifestyle. Per-user consumption of energy is ever in-creasing. People, nowadays, use a lot of modern ap-pliances for their day to day chores. With technologi-cal advances, the invention of new appliances, and theever-increasing interest of the new generations in thegadget market, investment in the household appliancemarket has increased manifold. With most of theseappliances running on electricity, the rising electricityconsumption also increases the load on power grids.People nowadays are willing to pay more for electric-ity rather than live without it. Household appliancesin the US are responsible for approximately 42% ofthe total energy consumption [1]. Given the high de-mand for electricity, efforts are being made continu-ously to improve smart grids with advanced researchin power system and computer science. Rising energyrequirement increases the load on power grids. Also,energy consumption follow specific trends that lead todisparity in demands from grids based on the time ofthe day, i.e, energy demand during particular periodscan be higher than usual, whereas during other peri-ods of the day energy requirements can be pretty low.During peak hours, the load on power grids increasesdrastically. To avert this problem, Demand Side Man-agement (DSM) strategies are used. DSM strategiesinvolve Demand Response (DR), energy efficiency, andconservation schemes.
DR [2, 3] focuses on modifying consumers’ demandon the benefit of reducing peak load on the grid andin turn, giving some incentives back to customers whoparticipate in it. DR encourages consumers to use lesspower during peak hours of the day or shift their timeof energy usage to off-peak hours of the day. Examplesof DR may include storing energy during the off-peakperiods from the grid and using the stored energy dur-ing the peak period. One could also make use of renew-able sources of energy by storing these energies suchas solar, wind, geothermal and biogas, which could beused during the peak periods of the day. The bene-fits of DSM are discussed in [4]. Home Energy Man-agement (HEM) [5] is a part of DSM. HEM systemsmanage the usage of electricity in smart homes. DR isa crucial component of HEM systems. DR [6] is oneof the techniques used in DSM. It involves scheduling
1
arX
iv:2
005.
1425
9v1
[ee
ss.S
P] 2
8 M
ay 2
020
Figure 1: Proposed architecture of smart gird with RL-DSM at consumer end.
loads on the timescale by moving high wattage loadsto a different time to reduce the maximum load on thegrid without changing the net energy consumed. It fo-cuses on changing the “when consumed” rather thanchanging the “how much consumed”.
Electricity generation involves several generatingstations employing different generation technologiesworking in conjunction, thus making the process dy-namic in its behavior. This translates to varying costsof electricity generation at any given point in time.This is where load shifting comes into play. Schedulingthe load on the timescale that would reduce the overallpeak demand on the grid and also saves electricity billsfor the consumers. Most of the DR based optimiza-tion models are based on two broad categories: price-based and incentive-based. Price based optimization isdiscussed in the study conducted in this paper. Theprice-based models consider Time of Use (ToU) pric-ing, peak load pricing, critical peak pricing, and real-time pricing [7, 8], which take into account the peakand off-peak tariffs. The varying tariffs of energy con-sumption during the entire duration of a day based onthe aggregate load on the electrical systems act as amotivation for consumers to adjust their appliance us-age to take advantage of the lower prices during specificperiods. Incentive-Based DR programs are two types:Classical programs and Market-based programs. Clas-sical programs include Direct Load Control and Inter-ruptible programs, Market-based are Emergency DR,Demand Bidding, Capacity Market, and Ancillary ser-vices market [9]. In [10], the authors proposed a pric-ing scheme for consumers with incentives to achievea lower aggregate load profile. They also studied theload demand minimization possible with the amountof information that consumers share. In [11] linearand nonlinear modeling for incentive-based DR for realpower markets was proposed. System-level dispatchof demand response resources with a novel incentive-based demand response model was proposed by [12].In [13], the authors proposes an Interruptible program,including penalties for customers in case they do notrespond to load reduction. A real-time implementationof incentive-based DR programs with hardware for res-idential buildings is shown in [10]. In [14] a novel DRprogram targeting small to medium size commercial,industrial, and residential customers is proposed.
Reinforcement Learning (RL) [15] is an area of ma-chine learning in computer science where a learning
agent interacts with an environment and receives re-wards as feedback of the interaction with the ulti-mate goal of maximizing the cumulative reward. Toachieve this, RL agents attempt to come up with apolicy mapping states to the best action at any givenstate, which would result in maximum expected futurerewards. Well designed RL agents have displayed im-peccable decision-making capabilities, such as Google’sAlphaGo and OpenAI Five, in complex environmentswithout the requirement of any prior domain knowl-edge. Deep Reinforcement Learning (DRL) [16] is amerger of deep learning and reinforcement learningwhere deep learning architectures such as neural net-works are used with reinforcement learning algorithmslike Q-learning, actor-critic, etc. [17] discusses build-ing DRL agents for playing Atari games and how DQN(Deep Q-Network) shows exceptional performance inplaying the games.
The proposed work aim at applying deep reinforce-ment learning techniques to the scenario of load shift-ing and comparing the results obtained with that ofthe MILP [18–20] based methods. We also propose asmart grid architecture, as shown in Figure 1 where RLbased DSM controller can be placed at the consumerend. The grid end DSM with RL is an extension of thiswork, where raw loads come from residential microgridsinstead of individual homes. An automated RL agentperforming the task of load shifting would go a longway into optimizing load consumption by distributingloads from peak to off-peak periods, thus reducing thetotal load on the power grid during peak periods andreducing the energy bills of consumers. The main con-tributions of the proposed work are:
1. Introduced Deep reinforcement learning in De-mand Side Management (RL-DSM) for DR.
2. Analyzed that the impact of a well-calculated re-ward system is crucial for Demand Side Manage-ment Reinforcement Learning models.
3. The proposed reinforcement learning model sur-passed traditional methods with a single objectiveby saving 6.04% of the monthly utility bill.
4. The proposed reinforcement learning model withmulti-objective saved 11.66% of the monthly util-ity bill, which shows the superior ability of thelearning model over traditional methods for De-mand Side Management.
2
2 Related Work
Demand response optimization is extensively exploredin literature. [21] gives an overview of DSM with var-ious types and the latest demonstration projects inDSM. [22] discusses a greedy iterative algorithm thatenables users to schedule appliances. [23] presents lin-ear programming based load scheduling algorithms.Mixed-integer linear programming model for optimalscheduling of appliances has been discussed in [18–20].Heuristic-based scheduling algorithms that aim at costminimization, user comfort maximization, and peak toaverage ratio minimization have been discussed in de-tail in [24]. A constrained multi-objective schedulingmodel for the purpose of optimizing utility and min-imizing cost is proposed in [25]. A dynamic pricingmodel for energy consumption cost minimization andcomfort maximization in the context of smart homeshas been explored in [26]. A study of various controlalgorithms and architectures applied to DR was car-ried out in [27]. [28] proposes a demand response costminimization strategy with an air source heat pumpalong with a water thermal storage system in a build-ing. [29] studies various energy demand prediction ma-chine learning models like feed-forward neural network,support vector machine, and multiple linear regres-sion. [30] proposes a systematic method to quantifybuilding electricity flexibility, which can be an assur-ing demand response resource, especially for large com-mercial buildings. [31] studies the effect of demand re-sponse on energy consumption minimization with theheat generation system in small residential builds in amore frigid climate.
Increasing demand for energy has shifted the focusof the power industry to alternative renewable sourcesof energy for power generation. Integrating renewablesources of energy into HEM systems can be beneficialfor both customers and power generation companies.Renewable sources of energy can fill in the excess de-mand for electricity by customers, thus moderating thepeak loads on the grids. Aghaei et al. carried out stud-ies on DR using renewable sources of energy [32]. [33]introduces a two-stage power-sharing framework. [34]talks about a cloud-based DSM method where con-sumer also has local power generation with batter-ies. [35] discusses the design for smart homes with theintegration of renewable energy sources for peak loadreduction and energy bill minimization. [36] introducesa real-time decentralized DSM algorithm that takes ad-vantage of energy storage systems (ESS), renewable en-ergy, and regularizing charging/discharging. [37] pro-poses a method to improve DSM by optimizing powerand spectrum allocation. [38] proposes a hierarchicalday-ahead DSM model with renewable energy. [39]discusses an Intelligent Residential Energy Manage-ment System (IREMS), which offers a model with in-house power generation using renewable sources of en-ergy. [40] proposes an algorithm to minimize consumercost by facilitating energy buying and selling betweenutility and residential community. [41] offers a multi-objective to demand response program by reducing the
energy cost of residential consumers and peak demandof the grid. [42] introduces collaborative energy trad-ing and load scheduling using a game-theoretic ap-proach and genetic algorithm. [43] propose a game-theoretic approach to minimize consumer energy costand discomfort in a heterogeneous residential neigh-borhood. [44] reduces the overall cost of the system byoptimizing the load scheduling and energy storage con-trol simultaneously with Lyapunov optimization. [45]proposes an intelligent residential energy managementsystem for residential buildings to reduce peak powerdemand and reducing prosumers’ electrics bills. [46] in-troduces an automated smart home energy manage-ment system using L-BFGS-B (Limited-memory Broy-den–Fletcher–Goldfarb–Shanno) algorithm with time-of-use pricing to optimize the load schedule.
[47] introduces an RL model to meet the overall de-mand with the current load, and the next 24 hrs loadpredicted information by shifting loads. [48] formeda fully automated energy management system by de-composing rescheduling loads over device clusters. [49]proposes the Consumer Automated Energy Manage-ment System (CAES), an online learning model thatestimates the influence of future energy prices andschedules device loads. [50] proposes a decentralizedlearning-based multi-agent residential DR for efficientusage of renewable sources in the grid. Lu [51] dy-namic pricing DR algorithm formulated as an MDP topromote service providers profit, reduce costs for con-sumers, and attain efficiency in energy demand andsupply. [52] introduced RL based scheduling of con-trollable load under a day-ahead pricing scheme.
This work encroaches into the territory of apply-ing DRL to DR. Existing deep reinforcement learning-based model for DR, which came out recently are[53,54]. [53] discusses an RL and DNN based algorithmmodeled on top of real-time incentives. A complete re-view of a collection of all control algorithms and theirassessment in DR strategies in the residential sector isgiven in [54]. It discusses machine learning-based pre-dictive approaches and rule-based approaches. Apply-ing deep reinforcement learning models to the settingof DR can be a lucrative field of research. The pro-posed model draws inspiration from DRL agents usedto play the game of Tetris. We have explored modelsof DRL automated agents playing the game of Tetrisand have tried to look for areas of similarity in theformulation of the reward function. [55] discusses thedesign of a DQN model for playing Tetris and presentsthe reward system of an automated Tetris agent.
3 Methods
Reinforcement learning has shown tremendous stridesin learning to take action in game environments sur-passing humans. Bringing RL capability to DR cre-ated the need to model DR into a game environment.An Atari game called Tetris falls in very close with thesimulation environment needed for DR. In Tetris, theplayer is asked to move blocks of different sizes and
3
Figure 2: Demonstrates the difference in block settling in game environment (left) and proposed DR simulation (right).Red, Green and Blue blocks aren’t supported by the grid base or another load block, thus they are allow to be brokendown and slid down to lower level in the simulation. Figure best viewed in color.
shapes in a 2D grid. Actions the player can take onthe block are rotation, moving left, right, or down. Theplayer is rewarded with points when a full line in thegrid is filled, and the game terminates when the blockplaced touches the maximum height of the grid. Weadapted this game environment to build a simulationto perform DR. The blocks in the game will be deviceloads in the DR simulation. The player will be replacedby an RL agent who will take action like moving loadblocks down, left, and right. Unlike solid blocks in thegame environment, the load blocks in the DR simula-tion are flexible solids, i.e., if part of the load in thegrid is not supported by the grid base or another loadblock, it is allowed to slide down to the lower level asshown in Figure 2. The agent reward is determinedby the load peak generated when a block is placed inthe simulation grid. A positive reward is given if itdoesn’t increase the current maximum load peak and anegative reward when the current maximum load peakincreases. The simulation ends when the load placedcrosses the maximum height, which motivates the RLagent to place more load in the simulation grid withoutgenerating peaks.
3.1 Simulation Design
The simulation is modeled to represent loads on atimescale of 24 hours. The simulation environmentconsists of 24 columns, each column representing anhour on the 24-hour clock. The height of each columndepicts the aggregate load on the electrical system atthat particular hour. Ideally the height of the simu-lation grid is decided by the maximum aggregate loadfrom the historical load profiles of the consumers in aspecific grid. As a proof of concept, a maximum of25kW of the aggregate load is set while training andtesting, this can be scaled according to the size of thepower grid. If the aggregate load at any hour exceedsthe constraints on maximum load, the simulation ter-minates.
3.2 Delineating States and Actions
The learning algorithm stores data in the form of (state,action, next state, reward) tuples. The state is an im-age of the current simulation screen. The action spaceis defined in terms of three actions: left, right and dropwhich imply left move, right move and dropping theblock onto the simulation respectively. All actions aresingle actions; i.e., for any state transition, only a sin-gle action can be performed, there are no cases where acombination of the individual actions can be performedfor a state transition. Taking action at any state causesthe game to transition to a new state that generates areward.
At any point of time if the agent decides on one ofthe actions the simulation moves to a new state. Aright action shifts the current load one cell (represent-ing timestamp) to the right and the new state is theload block shifted one block to the right. A left actionshifts the block of load one cell to the left and the sim-ulation transitions to a new state with the load blockshifted one cell to the left. A drop action results in theload block being placed on the load profile and a subse-quent state transition with the load block now placedon the load profile immediately below the load block.Actions in this sense are discrete and not continuous.At any point of time the state of the simulation is theload profile and the load block. This is captured onevery action. The state transitions are finite as therecan be finite shapes of blocks and they can be placed inon 24 timestamps. State transitions are deterministicin the the sense that given a state, block, action we canalways predict the next state.
3.3 Deep-Q-Learning (DQN)
DQN [16] agent based on the idea of using a neural net-work to approximate the below Q as shown in Equa-tion 1, and the pipeline of this method can be found in
4
Algorithm 1.
Qπ(s, a) = maxπ
Eπ[rt + γrt+1 + γ2rt+2+
.....|st = s, at = a](1)
where at is the action and st is the state at time t, rt isthe reward obtained by taking action at given state st,π denote policy function defines the learning agent’sway of behaving at a given time and γ is the discountfactor. The Q-function can be simplified as:
Q∗(s, a) = Es′ [r + γmaxa′
Q∗(s′, a′)|s, a] (2)
where s′ is the state generated by performing actiona in state s and a′ denoted the action taken in states′. The DQN model used in the proposed method isa Double DQN [56]. Double DQNs handles the prob-lem of the overestimation of Q-value. This consists oftwo networks, namely a policy network and a targetnetwork where all changes to the gradients are madeon the policy network, which is synced with the tar-get network at regular intervals of episodes. Episodeis the length of the simulation at the end of which thesystem ends in a terminal state. DQN network is usedto select the best action to take for the next state (theaction with the highest Q-value). The target networkis used to calculate the Q-value of taking that actionat the next state.
Figure 3: Network architecture used for agent. The policynetwork and target network have the same architecture.Convlayer consist of convolution block, batch normaliza-tion layer and ReLU activation. FC stands for fully con-nected layer.
The network consists of three convolution layers andone fully connected layer as shown in Figure 3. Allconvolution layers (CNN) use a 5 × 5 kernel and astride of 2 with batch normalization and ReLU (Rec-tified Linear Unit) as the activation function. The lasthidden layer is a fully connected (FC) layer with 192units, and the output layer consists of three units forthree actions. CNN is used over other network archi-tectures because CNN is memory efficient and best forfeature extraction. The number of parameters in FCdrastically increases with the increase in hidden layerswhen compared to CNN, thus increases computationand memory used. Designing a faster model is cru-cial for applications like load shifting, thus memoryefficient and fast neural networks like CNNs are used.According to the design of the proposed RL state, CNNis more apt as it is efficient in feature extraction fromraw data with spatial structures like simulation screens.We have also done an extensive analysis of how the se-lected CNN network over other affects the load shifting
problem in the proposed model, as shown in Table 6and Figure 6b. Even though the shallow network wasable to minimize cost, it creates high load peaks. Withbatch normalization, the model normalizes the inputlayer by adjusting and scaling the activation. Batchnormalization reduces the amount by what the hid-den unit values of the network shift around (covarianceshift), and this is proven in [57] to speed up the learn-ing process. RMSProp [58] is used for optimization inthe network.
For training, the network approximates the Q-valuesfor each action and take action with the maximum Q-value. The target network estimate the Q-values ofthe optimal Q-function Q* by using the Bellman Equa-tion 2, since Q-function for any policy obeys the Bell-man equation. Temporal difference error δ is computedby taking the difference of predicted Q-value and theoptimal Q-value computed from the Bellman equationusing the target network.
δ = Q(s, a)− (r + γmaxa′
Q∗(s′, a′)) (3)
By minimizing the loss between the Q-value and theoptimal Q-value, the agent arrives at the optimal pol-icy. Huber loss is used to minimize the error definedas below:
L =1
|B|∑bεB
L(δ) (4)
where batches B of experiences (knowledge acquired inthe past) where sampled from the agent’s replay bufferand L(δ) is defined as
L(δ) =
{12δ
2 |δ| ≤ 1
|δ| − 12 Otherwise
(5)
The Huber loss is robust to outliers. When the erroris small, it takes the definition of mean squared errorbut acts like the mean absolute error when the error issignificant.
3.4 Epsilon Greedy Policy and Experi-ence Replay
Epsilon greedy policy and Experience replay are twocrucial techniques that help the learning process of theRL agent drastically. Given the fact that the statespace is vast, the agent is enabled to explore the avail-able state space initially without taking too many pre-dicted actions from the network. When the agent takesa random step at any given state its called explorationand when it uses already accumulated experience tomake a predicted decision from the network is calledexploitation. Exploration and exploitation should notbe run exclusively. The agent explores many statesat first by taking random actions, but with each suc-cessive epoch, it increases the number of informed de-cisions exploiting the known information to maximizethe reward. The decay function γ used for this pur-
5
Algorithm 1: DQN algorithm
Input: D: Empty replay buffer,θ: Policy network parameters,θ−: Copy of θ, target netparamss,Nr: Replay buffer maximum size,Nb: Training batch size,N−: Target network replacement freq,M : Number of episodes,γ: Discount factor,εd: Decay rate of epsilon,εs: Starting value of epsilon,εe: Final value of the epsilon,Unif(D): Uniforming sampling from D,x: Empty frame sequence,A: Action space, sd: Steps done,grid: Matrix representing the state,S0: Initial state of the game.
Set sd ←− 1for episode ∈ {1,2,...,M} do
Set grid ←− S0
for t ∈ {0,1,2,...} doSet state s ←− x;Your command was ignored.Type I¡command¿ ¡return¿ to replace it withanother command,or ¡return¿ to continuewithout it.
Set
ε←− εe +εs + εe
esdεd
Set r ←− random number between 0 and 1if r > ε then
Set a ←− argmaxaQ(s, a; θ)else
Set a ←− random action from ASample next frame Xt from environment εgiven (s, a) and receive reward r and appendxt to x.
if |x| > Nr thenDelete oldest xtmin from x
endif |D| ≥ Nr then
Set s′ ←− x, and add transitiontuples(s, a, r, s′) to D, replace the oldesttuple
endSample a minibatch of Nb,tuples(s, a, r, s′) ∼ Unif(D)Construct target values, one for each the Nb
tuples:
yj =
{r if s’ is terminal
r + γmaxa′Q(s′, a′; θ−) otherwise
Do a gradient descent step with loss||yj −Q(s, a; θ)||2Replace target parameters θ− ←↩ θ every N−
steps.Set sd ←− sd+ 1Update grid based on previous grid and thenew load.if grid has a fully filled column then
Terminate episodeend
end
end
pose is the exponential decay function which is definedbelow:
γ = εe +εs + εe
esdεd
(6)
where sd is the total number of iterations till nowand εd is a hyperparameter controlling the rate of de-cay from εs to εe. One iteration translates to takingone action. Note that episodes and iterations have dif-ferent meanings in this context. Hyperparameters areparameters that are manually set by the practitionerand tuned according to a specific goal.
The Experience Replay [59] is used to avoid the agentfrom forgetting previous experiences as it trains onnewer states and to reduce correlations between ex-periences. In Experience Replay, a buffer of the oldexperiences is maintained and the agent is trained onit. By sampling batches of experience from the bufferat random, the correlation between the experiences canbe reduced. The reason for this is that if you were tosample transitions from the online RL agent as theyare being experienced, there would be a strong tempo-ral/chronological relationship between them. Duringour experiments, we fixed buffer size at 3 × 104. Weempirically found the optimal buffer size that worksbest for our model without allotting huge amount ofreserved memory space. We have done extensive anal-ysis on picking the right buffer size, which can be foundin Table 7 and Figure 6b.
3.5 Reward Systems
The proposed reward system consists of three impor-tant aspects, maximum height hmax, variance of theheight distribution var(h) and number of completelines l formed from the distribution. The reward sys-tem R is summarized as:
R = α1 ∗ var(h) + α2 ∗ l − α3 ∗ hmax (7)
The variance of the height distribution var(h) isgiven by:
var(h) =1
1 + var(ha)(8)
where var(ha) the variance of the height distributionafter taking the action.
The variance of load distribution reward var(h)would encourage the agent to shift load such that moreuniform distribution of height of the load profile canbe attained. Complete lines l reward complements theagent to increase the spread to have more completelines (rows) in the simulation. This reward componentenables the agent to decide to shift a load to either sideedges of the simulation. Maximum high reward com-ponent hmax helps the agent to avoid making a peakin the load profile as this reward contributes negativereward if the current maximum high increased. Otherhyperparameters used for the experiments are shownin Table 2.
6
Appliances Rated power (kWh) Preferred time Duration (hrs)Consumer 1Non-shiftable
Table 1: Daily load demand of five different consumers.Rated power 1.0, 0.5 indicate the power rating at first andsecond hour of the device.
Hyper parameters Valuesα1 10.0α2 0.76α3 0.5α4 0.2
Buffer size 3× 104
Learning rate 0.001
Table 2: Hyper parameters used in experiment.
Price (cents/kWh)Off-Peak Mid-Peak Peak Off-Peak
0-6hrs 6-15hrs 15-22hrs 22-24hrs6 9 15 6
Table 3: Monthly energy billing scheme.
The variance of load distribution encountered somecases where the agent realised that placing the blockson the same position also lead to increase in variancefor a sufficiently well spread out distribution. Due tothis, initially the agent would perform as expected byplacing blocks in a manner which caused the distribu-tion to spread out, but after the certain point it wouldstart placing all the blocks at the same position, re-sulting in the game terminating with poor results. Tocounter this, standard deviation was utilized to prop-erly scale the impact of distribution on the overall re-ward vis-a-vis the other reward parameters.
For simultaneous cost and peak minimization, an ad-ditional reward term c is introduced to the peak min-imization reward system. The cost is calculated withthe help of the piece-wise pricing adapted from [60] asshown in Table 3. As the price of the load scheduleincreases, the reward for the agent decreases; the termhelps the agent to shift load that results in lower billingprice, as shown in the Equation 9.
R = α1 ∗ var(h) + α2 ∗ l − α3 ∗ hmax − α4 ∗ c (9)
4 Simulation Results and Dis-cussion
The proposed RL model is tested in two different casestudies with five customers load data adapted from[18], as shown in Table 1. In case 1, the utility en-ergy bill is reduced with peak minimization objective.Case 2 reduces the utility energy bill by lowering peakand cost simultaneously. The models based on the pro-posed peak minimization technique showcase better re-sults than MILP.
4.1 Case 1
The objective of this case study is to minimize peakload demand with RL and thereby reducing the energybill of the consumer. The model is tested with oneobjective on five different consumers, as discussed inthe above sections. DR with RL (using DQN agent)scheduled load is compared with MILP scheduling forall consumers in Figure 4. Figure 4 shows how the sys-tem reacts to the aggregated load of all consumers. Theeffect of selecting the right reward function of the RLmodel for scheduling each consumer load is shown inTable 4 by comparing variance and standard deviationas a reward function. The variance reward system out-
Continuous curves have been used for better visualizationinstead of the discretized plot even though the load demand isonly captured at 24 timestamps.
7
(a) (b)
(c) (d)
(e) (f)
Figure 4: RL-DSM results of residential loads for five different consumers (a), (b), (c), (d) and (e) respectively with Case1 objective of peak minimization which are compared with MILP. (f) is the RL-DSM results of aggregated residentialloads for five different consumers to minimize daily peak load with Standard deviation(std RL) and Variance(var RL).The figure best viewed in color.
ConsumerTotal energy bill ($/month) Monthly savings ($)
without MILP RL with std RL with var MILP RL with std RL with varConsumer 1 81 77.85 76.95 74.25 3.15 4.05 6.75Consumer 2 92.25 90.5 82.8 81.9 1.35 9.45 10.35Consumer 3 87.75 89.55 87.75 79.65 -1.8 0 8.1Consumer 4 88.2 78.75 80.1 80.1 9.45 8.1 8.1Consumer 5 82.8 81 82.8 78.05 1.8 0 6.75
All 432 418.05 410.4 391.95 13.95 21.6 40.05
Table 4: Energy bill minimization with peak reduction (Case 1).
performed the standard deviation for all test consumerdata.
As shown in Table 4, the aggregated load reduced theoverall bill cost from $432 to $391.95, saving around9.27%. From the results, it can be inferred that theproposed method could reduce the monthly bill dra-matically when compared to other traditional methods
on single peak objective minimization.
4.2 Case 2
In this case study, two different objectives are min-imized simultaneously with RL. Here, the model re-duces the peak load and cost of the load schedule. This
8
(a) (b)
(c) (d)
(e) (f)
Figure 5: RL-DSM results of residential loads for five different consumers (a), (b), (c), (d) and (e) respectively with Case2 objective of peak and cost minimization. (f) is the RL-DSM results of aggregated residential loads for five differentconsumers to minimize daily peak load and cost. The figure best viewed in color.
Table 5: Energy bill minimization with peak and cost reduction (Case 2).
hybrid multi-objective function guarantee that the loadprofile does not have high peaks, and at the same time,the aggregated price of the load per day is minimum,as seen in Figure 5. It is observed that the loads frompeak hours have been moved to min peak or low peaktime to minimize high power demand. Taking load costinto account trains the model to understand that mov-
ing most loads from peak to non-peak hours is not en-tirely ideal for the consumer as it may not shift the loadto the time duration at lower prices. Adding the costfactor helps the agent to move the blocks to the lower-priced time duration, which currently has fewer loads.However, the experiments with single cost objective,which only considers cost and disregards peak mini-
9
(a) (b)
Figure 6: RL-DSM results of aggregated residential loads of five different consumer to minimize daily peak load and cost(a) with Shallow (After with Shallow) and Deep network (After with Deep) (b) using different memory buffer size like3000 (3k), 10000 (10k), 30000 (20k) and 50000 (50k). The figure best viewed in color.
ConsumerTotal energy bill ($/month) Monthly savings ($) Peak load (kW)
Before Shallow Net Deep Net Shallow Net Deep Net Shallow Net Deep NetConsumer 1 81 71.55 73.35 9.45 8.1 4 2Consumer 2 92.25 82.35 81 9.9 13.5 3 3Consumer 3 87.75 79.65 79.65 8.1 3.6 5 3Consumer 4 88.2 71.1 75.15 17.1 14.4 5.5 2.5Consumer 5 82.8 72.45 72.45 10.35 3.6 3 3
All 432 377.1 381.6 54.9 43.2 18.5 10.5
Table 6: Demonstration of deep CNN network’s efficiency compared to shallow network. Shallow Net: Shallow networkand Deep Net: Deep network.
mization entirely, shows that non-peak hours startedto get high peak loads as expected because the rewardsystem for the agent focused solely on cost minimiza-tion. This multi-objective function solves this issue.Adding peak minimization with cost minimization as-sures that the agent is not fixated on cost minimiza-tion, causing peaks to be formed during a time whenthe cost of electricity consumption is less. Thus, boththe parameters of the multi-objective function, peakand cost minimization are mutually beneficial for oneanother. Figure 5 shows how the system reacts to theaggregated load of all consumers.
As shown in Table 5, the aggregate load reducedbill cost from $432 to $381.6, saving around 11.66%.The current limitation of the proposed system is thatit does not support dynamic preferred time for eachdevice, and load demand is on the scale of 0.5kW. Thisis the reason that the daily test load demand shown inTable 1 has a 1hour/unit time scale and 0.5kW/unitpower demand scale to fit the proposed RL simulator.
As traditional methods like MILP does not have aholistic knowledge of the system, it can be only usedfor a fixed number of devices. If the traditional systemneeds to accommodate more devices at some point intime after deployment, the method needs to be rerun,and scheduling factors should be computed again. Theproposed methods are more flexible as it has the learn-ing capability to handle any load given to it. Even afterthe proposed RL model is deployed, there is no needfor any adjustment to add or remove devices from thesystem at any time instant. The time complexity ofthe proposed method is better than other traditionalmethods and its linearly proportional to number of de-
Figure 7: RL-DSM results on loads of 14 different devicesto demonstrate the scalability of the proposed method.
vices, unlike traditional methods that have exponentialgrowth in time concerning the number of devices to bescheduled. A trained RL agent always operates witha fixed number of floating-point arithmetic (FLOPs)and multiply-adds (MAdd) for any number of loads tobe scheduled. The number of FLOPs and MAdd onlydepend on the architecture of the network. This makesthe proposed model schedule hundreds of devices timeand space-efficient than traditional methods. The timecomplexity of RL can be defined as O(kp) and MILPwould be O(k2p) where p is the number of parametersassociated with the model and k is the number of de-vices to be scheduled. The space complexity of RL canbe formulated as O(p), whereas for MILP would beO(k2). The scalability of the proposed methodologyis demonstrated by scheduling 46 loads of 14 differentdevices but with power-hungry devices with RL agentas shown in Figure 7. Deploying trained RL agents in
Table 7: Demonstration of the effect of different memory buffer size like 3000 (3k), 10000 (10k), 30000 (20k) and 50000(50k) in RL-DSM for peak and cost minimization.
an embedded device is as easy as deploying traditionalmodels.
5 Conclusion
The exponential growth in demand for power in thehousehold has increased the stress on the power gridto meet its demand. DR can help a smart grid to im-prove its efficiency to meet the power need of the cus-tomer. This paper proposes a residential DR using adeep reinforcement learning method where both loadprofile deviation and utility energy bills are minimizedsimultaneously. An extensive case study with a singleobjective and multi-objective cases is conducted. Inboth cases, the proposed method outperformed the tra-ditional MILP method. This paper exhibited the po-tential of reinforcement learning for better smart gridoperations and tested all these cases on five differentconsumers and showcased the ability of the proposedmethod. In the future, this work can be extended to in-troduce variable preferred time for each device and im-prove the RL simulation grid to accommodate deviceswith lower power demand with a scale of 0.1kW. Thiswork can also be extended to schedule more granulartimed devices with less than 1hr to complete its task.In this work, renewable energy sources and energy stor-age are not considered. Designing an RL agent to man-age renewable energy and energy storage can bring outthe full potential of AI models in energy management.
References
[1] P. Yi, X. Dong, A. Iwayemi, C. Zhou, and S. Li,“Real-time opportunistic scheduling for residen-tial demand response,” IEEE Transactions onSmart Grid, vol. 4, pp. 227–234, March 2013.
[2] P. Palensky and D. Dietrich, “Demand side man-agement: Demand response, intelligent energysystems, and smart loads,” IEEE Transactions onIndustrial Informatics, vol. 7, pp. 381–388, Aug2011.
[3] C. River, “Primer on demand side management,”Report for the World Bank, 2005.
[4] G. Strbac, “Demand side management: Benefitsand challenges,” Energy Policy, vol. 36, no. 12,pp. 4419 – 4426, 2008. Foresight Sustainable
Energy Management and the Built EnvironmentProject.
[5] Y. Son, T. Pulkkinen, K. Moon, and C. Kim,“Home energy management system based onpower line communication,” IEEE Transactionson Consumer Electronics, vol. 56, pp. 1380–1386,Aug 2010.
[6] E. P. Act, “Energy policy act of 2005,” in USCongress, 2005.
[7] B. Severin, J. Michael, and R. Arthur, “Dynamicpricing, advanced metering and demand responsein electricity markets,” Center for the Study ofEnergy Markets, UC Berkeley, 2002.
[8] C. Kang and W. Jia, “Transition of tariff struc-ture and distribution pricing in china,” in 2011IEEE Power and Energy Society General Meeting,pp. 1–5, IEEE, 2011.
[9] M. H. Albadi and E. F. El-Saadany, “Demandresponse in electricity markets: An overview,”in 2007 IEEE power engineering society generalmeeting, pp. 1–5, IEEE, 2007.
[10] S. Caron and G. Kesidis, “Incentive-based en-ergy consumption scheduling algorithms for thesmart grid,” in 2010 First IEEE InternationalConference on Smart Grid Communications,pp. 391–396, IEEE, 2010.
[11] M. A. F. Ghazvini, J. Soares, N. Horta, R. Neves,R. Castro, and Z. Vale, “A multi-objective modelfor scheduling of short-term incentive-based de-mand response programs offered by electricity re-tailers,” Applied energy, vol. 151, pp. 102–118,2015.
[12] M. Yu and S. H. Hong, “Incentive-based de-mand response considering hierarchical electricitymarket: A stackelberg game approach,” AppliedEnergy, vol. 203, pp. 267–279, 2017.
[13] H. Aalami, M. P. Moghaddam, and G. Yousefi,“Demand response modeling considering inter-ruptible/curtailable loads and capacity marketprograms,” Applied Energy, vol. 87, no. 1,pp. 243–250, 2010.
11
[14] H. Zhong, L. Xie, and Q. Xia, “Coupon incentive-based demand response: Theory and case study,”IEEE Transactions on Power Systems, vol. 28,no. 2, pp. 1266–1276, 2012.
[15] R. S. Sutton, A. G. Barto, et al., Introduction toreinforcement learning, vol. 135. MIT press Cam-bridge, 1998.
[16] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu,J. Veness, M. G. Bellemare, A. Graves, M. Ried-miller, A. K. Fidjeland, G. Ostrovski, et al.,“Human-level control through deep reinforcementlearning,” Nature, vol. 518, no. 7540, p. 529, 2015.
[17] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves,I. Antonoglou, D. Wierstra, and M. Riedmiller,“Playing atari with deep reinforcement learning,”2013.
[18] B. Lokeshgupta and S. Sivasubramani, “Cooper-ative game theory approach for multi-objectivehome energy management with renewable en-ergy integration,” IET Smart Grid, vol. 2, no. 1,pp. 34–41, 2019.
[19] Z. Bradac, V. Kaczmarczyk, and P. Fiedler, “Op-timal scheduling of domestic appliances via milp,”Energies, vol. 8, no. 1, pp. 217–232, 2015.
[20] S. G. Hamed and A. Kazemi, “Multi-objectivecost-load optimization for demand side,”Information Networking (ICOIN), vol. 481,p. 485, 2016.
[21] P. Palensky and D. Dietrich, “Demand side man-agement: Demand response, intelligent energysystems, and smart loads,” IEEE transactions onindustrial informatics, vol. 7, no. 3, pp. 381–388,2011.
[22] P. Chavali, P. Yang, and A. Nehorai, “Adistributed algorithm of appliance schedulingfor home energy management system,” IEEETransactions on Smart Grid, vol. 5, pp. 282–290,Jan 2014.
[23] C. O. Adika and L. Wang, “Smart charging andappliance scheduling approaches to demand sidemanagement,” International Journal of ElectricalPower & Energy Systems, vol. 57, pp. 232–240,2014.
[24] S. Rahim, N. Javaid, A. Ahmad, S. A. Khan,Z. A. Khan, N. Alrajeh, and U. Qasim, “Exploit-ing heuristic algorithms to efficiently utilize en-ergy management controllers with renewable en-ergy sources,” Energy and Buildings, vol. 129,pp. 452 – 470, 2016.
[25] S. Salinas, M. Li, and P. Li, “Multi-objectiveoptimal energy consumption scheduling in smartgrids,” IEEE Transactions on Smart Grid, vol. 4,pp. 341–348, March 2013.
[26] E. Shirazi and S. Jadid, “Optimal residential ap-pliance scheduling under dynamic pricing schemevia hemdas,” Energy and Buildings, vol. 93, pp. 40– 49, 2015.
[27] Y. W. Law, T. Alpcan, V. C. S. Lee, A. Lo,S. Marusic, and M. Palaniswami, “Demand re-sponse architectures and load management algo-rithms for energy-efficient power grids: A sur-vey,” in 2012 Seventh International Conference onKnowledge, Information and Creativity SupportSystems, pp. 134–141, Nov 2012.
[28] F. D’Ettorre, M. De Rosa, P. Conti, D. Testi, andD. Finn, “Mapping the energy flexibility poten-tial of single buildings equipped with optimally-controlled heat pump, gas boilers and thermalstorage,” Sustainable Cities and Society, 2019.
[29] Y. Guo, J. Wang, H. Chen, G. Li, J. Liu, C. Xu,R. Huang, and Y. Huang, “Machine learning-based thermal response time ahead energy de-mand prediction for building heating systems,”Applied energy, vol. 221, pp. 16–27, 2018.
[30] Y. Chen, Z. Chen, P. Xu, W. Li, H. Sha, Z. Yang,G. Li, and C. Hu, “Quantification of electricityflexibility in demand response: Office buildingcase study,” Energy, vol. 188, p. 116054, 2019.
[31] B. Alimohammadisagvand, J. Jokisalo, andK. Siren, “Comparison of four rule-based demandresponse control algorithms in an electrically andheat pump-heated residential building,” AppliedEnergy, vol. 209, pp. 167–179, 2018.
[32] J. Aghaei and M.-I. Alizadeh, “Demand responsein smart electricity grids equipped with renew-able energy sources: A review,” Renewable andSustainable Energy Reviews, vol. 18, pp. 64 – 72,2013.
[33] S. Cui, Y.-W. Wang, J. Xiao, and N. Liu,“A two-stage robust energy sharing managementfor prosumer microgrid,” IEEE Transactions onIndustrial Informatics, 2018.
[34] M. H. Yaghmaee, M. Moghaddassian, andA. Leon-Garcia, “Autonomous two-tier cloud-based demand side management approach withmicrogrid,” IEEE Transactions on IndustrialInformatics, vol. 13, no. 3, pp. 1109–1120, 2016.
[35] N. Javaid, I. Ullah, M. Akbar, Z. Iqbal, F. A.Khan, N. Alrajeh, and M. S. Alabed, “An intelli-gent load management system with renewable en-ergy integration for smart homes,” IEEE Access,vol. 5, pp. 13587–13600, 2017.
[36] M. H. K. Tushar, A. W. Zeineddine, and C. Assi,“Demand-side management by regulating charg-ing and discharging of the ev, ess, and utilizing re-newable energy,” IEEE Transactions on IndustrialInformatics, vol. 14, no. 1, pp. 117–126, 2017.
12
[37] K. Ma, P. Liu, J. Yang, X. Wei, and C. Dou,“Spectrum allocation and power optimization fordemand-side cooperative and cognitive commu-nications in smart grid,” IEEE Transactions onIndustrial Informatics, vol. 15, no. 3, pp. 1830–1839, 2018.
[38] D. Li, W.-Y. Chiu, H. Sun, and H. V. Poor, “Mul-tiobjective optimization for demand side manage-ment program in smart grid,” IEEE Transactionson Industrial Informatics, vol. 14, no. 4, pp. 1482–1490, 2017.
[39] S. L. Arun and M. P. Selvan, “Intelligent residen-tial energy management system for dynamic de-mand response in smart buildings,” IEEE SystemsJournal, vol. 12, pp. 1329–1340, June 2018.
[40] T. Li and M. Dong, “Residential energy storagemanagement with bidirectional energy control,”IEEE Transactions on Smart Grid, 2018.
[41] W.-Y. Chiu, J.-T. Hsieh, and C.-M. Chen, “Paretooptimal demand response based on energy costsand load factor in smart grid,” IEEE Transactionson Industrial Informatics, 2019.
[42] B. Rajasekhar, N. Pindoriya, W. Tushar, andC. Yuen, “Collaborative energy management fora residential community: A non-cooperative andevolutionary approach,” IEEE Transactions onEmerging Topics in Computational Intelligence,vol. 3, no. 3, pp. 177–192, 2019.
[43] B. Rajasekhar and N. Pindoriya, “Decentral-ized energy management for a group of het-erogenous residential customers,” in 2015 IEEEInnovative Smart Grid Technologies-Asia (ISGTASIA), pp. 1–6, IEEE, 2015.
[44] T. Li and M. Dong, “Real-time residential-side joint energy storage management and loadscheduling with renewable integration,” IEEETransactions on Smart Grid, vol. 9, no. 1, pp. 283–298, 2016.
[45] S. Arun and M. Selvan, “Intelligent residen-tial energy management system for dynamic de-mand response in smart buildings,” IEEE SystemsJournal, vol. 12, no. 2, pp. 1329–1340, 2017.
[46] M. Martinez-Pabon, T. Eveleigh, and B. Tanju,“Optimizing residential energy management us-ing an autonomous scheduler system,” ExpertSystems with Applications, vol. 96, pp. 373–387,2018.
[47] I. Dusparic, C. Harris, A. Marinescu, V. Cahill,and S. Clarke, “Multi-agent residential de-mand response based on load forecasting,” in2013 1st IEEE Conference on Technologies forSustainability (SusTech), pp. 90–96, Aug 2013.
[48] Z. Wen, D. O’Neill, and H. Maei, “Optimal de-mand response using device-based reinforcementlearning,” IEEE Transactions on Smart Grid,vol. 6, no. 5, pp. 2312–2324, 2015.
[49] D. O’Neill, M. Levorato, A. Goldsmith, and U. Mi-tra, “Residential demand response using reinforce-ment learning,” in 2010 First IEEE InternationalConference on Smart Grid Communications,pp. 409–414, IEEE, 2010.
[50] I. Dusparic, A. Taylor, A. Marinescu, V. Cahill,and S. Clarke, “Maximizing renewable energy usewith decentralized residential demand response,”in 2015 IEEE First International Smart CitiesConference (ISC2), pp. 1–6, Oct 2015.
[51] R. Lu, S. H. Hong, and X. Zhang, “A dynamicpricing demand response algorithm for smartgrid: Reinforcement learning approach,” AppliedEnergy, vol. 220, pp. 220 – 230, 2018.
[52] F. Ruelens, B. J. Claessens, S. Vandael,B. De Schutter, R. Babuska, and R. Belmans,“Residential demand response of thermostaticallycontrolled loads using batch reinforcement learn-ing,” IEEE Transactions on Smart Grid, vol. 8,no. 5, pp. 2149–2159, 2017.
[53] R. Lu and S. H. Hong, “Incentive-based demandresponse for smart grid with reinforcement learn-ing and deep neural network,” Applied Energy,vol. 236, pp. 937 – 949, 2019.
[54] F. Pallonetto, M. D. Rosa, F. Milano, andD. P. Finn, “Demand response algorithms forsmart-grid ready residential buildings using ma-chine learning models,” Applied Energy, vol. 239,pp. 1265 – 1282, 2019.
[55] M. Stevens and S. Pradhan, “Playing tetris withdeep reinforcement learning,”
[56] H. van Hasselt, A. Guez, and D. Silver, “Deepreinforcement learning with double q-learning,”2015.
[57] S. Ioffe and C. Szegedy, “Batch normaliza-tion: Accelerating deep network training by re-ducing internal covariate shift,” arXiv preprintarXiv:1502.03167, 2015.
[58] C. Igel and M. Husken, “Improving the rproplearning algorithm,” in Proceedings of thesecond international ICSC symposium on neuralcomputation (NC 2000), vol. 2000, pp. 115–121,Citeseer, 2000.
[59] L. J. Lin, “Self-improving reactive agents based onreinforcement learning, planning and teaching,”Machine Learning, vol. 8, pp. 293–321, 1992.
[60] P. gas and electric company, “Residential time ofuse service.” https://www.pge.com/.
Supplementary Material:Intelligent Residential Energy Management System using Deep Reinforcement
Learning
(a) (b)
(c) (d)
(e) (f)
Figure 8: RL-DSM results of residential loads for five different consumers (a), (b), (c), (d) and (e) respectively withShallow (After with Shallow) and Deep network (After with Deep). (f) is the RL-DSM results of aggregated residentialloads for five different consumers to minimize daily peak load and cost. The figure best viewed in color.
14
(a) (b)
(c) (d)
(e) (f)
Figure 9: RL-DSM results of residential loads for five different consumer (a), (b), (c), (d) and (e) respectively usingdifferent memory buffer size like 3000(3k), 10000(10k), 30000(20k) and 50000(50k). (f) is the RL-DSM results of ag-gregated residential loads for five different consumers to minimize daily peak load and cost. The figure best viewed incolor.
