The rapid increase in population growth and energy consumption has brought about many environmental problems such as global warming (Weil et al., 2023) and energy crisis (Hafeez et al., 2020a). Among all energy consumption, household energy consumption is an important component (Zhang et al., 2023). To optimize the energy structure of households and reduce energy consumption, energy consuming equipment such as rooftop photovoltaics (PV), heat pumps, electric vehicle (EV), and batteries have been widely promoted. With the rapid increase in the number of distributed PV (Li et al., 2024) and EVs (Yin and Qin, 2022), home energy system management (HEMS) has become the most important aspect of achieving demand-side energy management in smart grids (Hafeez et al., 2021; Huy et al., 2023). The HEMS can make decisions for demand response based on current electricity prices, predicted photovoltaic output, user preferences, and device characteristics, achieving intelligent scheduling of home equipment and reducing electricity costs (Kikusato et al., 2019; Gomes et al., 2023). The HEMS is a key component in achieving zero-energy homes and has the potential for widespread application in residential distribution systems. The scheduling strategies used in HEMS mainly include real-time energy allocation, day ahead scheduling, and closed-loop energy management. Among them, day ahead scheduling can reduce computational complexity and improve computational efficiency, which is widely accepted and applied (Ren et al., 2024).

There are lots of related work have been conducted based on HEMS. Liu et al. in (Liu et al., 2022) proposes a HEMS for residential users that incorporates the uncertainty of data-driven results to achieve the best trade-off between electricity cost and the preference level. Tostada-Veliz et al. in (Tostado-Véliz et al., 2022) develops a HEMS that includes three novel demand response routines focused on peak clipping and demand flattening strategies. Chakir et al. in (Chakir et al., 2022) propose a management system for a future household equipped with controllable electric loads and an electric vehicle equipped with a PV–Wind–Battery hybrid renewable system connected to the national grid. However, these studies only consider the dispatch strategy of single type of load, which may not in line with real usage scenarios. In the real home energy system, there are multi-types of loads, such as dispatchable load and non-dispatchable load, all these types loads should be considered in the constructed system. To this end, Rehman et al. in (ur Rehman et al., 2022) proposed a holistic method to optimize the use of different types of home appliances according to the prosumers preferences and defined schedule. Dorahaki et al. in (Dorahaki et al., 2022) presents develop a behavioral home energy management model based on time-driven prospect theory incorporating energy storage devices, distributed energy resources, and smart flexible home appliances, which considers the dispatch of different types of appliances. Nezhad et al. in (Esmaeel Nezhad et al., 2021) proposes a new model for the self-scheduling problem using a home energy management system (HEMS), considering the presence of different types of loads, such as an air conditioner and EV. When temperature-controlled load such as air conditioner contained in the HEMS, the users’ comfort should be considered in the dispatch strategy. Song et al. in (Song et al., 2022) presents an intelligent HEMS with three adjustable strategies to maximize economic benefits and consumers’ comfort. Youssef et al. in (Youssef et al., 2024) proposes strategies that are evaluated in terms of consumer comfort, and cost, with waiting time used to assess user comfort. Once the users’ comfort is taken into account, the single objective optimization will change into a multi-objective optimization. It is difficult and important to balance performance of different objectives to obtain the optimal dispatch strategy in the multi-objective optimization. To this end, several studies (Ullah et al., 2021; Alzahrani et al., 2023) are proposed for tackling this problem.

Then, how to obtain the optimal dispatch strategy of the HEMS is a crucial problem (Xiong et al., 2024). Normally, the optimization-based methods such as stochastic programming method (SP) (Hussain et al., 2023) and robust optimization method (RO) (Wang et al., 2024) are utilized to solve the optimization problem of HEMS. Tostado et al. in (Tostado-Véliz et al., 2023a) develops a novel SP-based home energy management model considering negawatt trading. Kim et al. in (Kim et al., 2023) proposes an SP-based algorithm to reduce computation time while preserving the stochastic properties of generated scenarios based on the Wasserstein-1 distance. Nevertheless, the SP-based method requires both vast computational ability and accurate distribution of random variables that may not be realized in practice (Xiong et al., 2023a). In this context, the RO-based methods are widely applied. Tostado et al. in (Tostado-Véliz et al., 2023b) proposes a fully robust home energy management model, which accounts for all the inherent uncertainties that may arise in domestic installations. Wang et al. in (Wang et al., 2024) proposes a multi-objective two-stage robust optimization to address the inherent uncertainty of DES, aiming to concurrently realize energy savings, carbon emission reduction, and load smoothing. However, the optimization results calculated by RO method are usually conservative and utilize only one dispatch solution to deal with all uncertainties of whole dispatch period. To this end, the learning-based methods have been utilized to solve this problem (Hafeez et al., 2020b; Ben Slama and Mahmoud, 2023; Ren et al., 2024).

To bridge these gaps, this paper proposes an optimized scheduling model for home energy management to minimize the electricity cost with consideration of users’ comfort. Then, a novel deep reinforcement learning (DRL) based algorithm is utilized to deal with the uncertainties. The main contributions of this paper are summarized as follows:

1) This paper develops an optimized scheduling model for home energy management to schedule both interruptible load and uninterruptible load, which takes consideration of time-of-use price and users’ comfort. Different from the Refs. (Chakir et al., 2022; Tostado-Véliz et al., 2022), The optimized strategy for scheduling multi types of loads based on the time-of-use electricity price and real-time energy storage system charging status, which can reduce user electricity costs while ensuring users’ comfort.

The following sections are organized: The proposed system is described in detail in Section 2. The mathematical modeling and optimization algorithm are discussed in Section 3. Section 4 presents and analyzes the simulation results obtained for the proposed system. The paper concludes in Section 5.

To construct the model of PV, temperature and light radiation intensity are the key factors for determining the output of PV (Xiong et al., 2023b). These factors can be represented in the following model: P P V , t = P P V , r a t e d G G S T C 1 − k T c − T r (1) where P P V , r a t e d is the rated output of PV in the normal operating condition; T r is the rated temperature under normal test conditions. G , k and T c are the light radiation intensity, power temperature coefficient, and atmospheric temperature, respectively. The details of parameters of the PV model are shown in the Table 1.

The household electricity load can be divided into dispatchable load, non-dispatchable load, and temperature-controlled load based on the degree of controllability (Hafeez et al., 2020c).

Energy storage devices participate in scheduling through charging and discharging, balancing power fluctuations and improving system flexibility. This article reflects the remaining capacity of energy storage devices through the State of Charge (SOC), which can be expressed as: S O C B , t = S O C B , t − 1 + P c , t η c / c a p − P d , t / c a p ⋅ η d Δ t (8) 0 ≤ P c , t ≤ P c , ⁡ max 0 ≤ P d , t ≤ P d , ⁡ max (9) S O C B , ⁡ min ≤ S O C B , t ≤ S O C B , ⁡ max (10) where S O C B , t represents the SOC of the battery at the time-step t; P c , t and P d , t are the charge and discharge power of the battery at the time-step t; η c and η d are the charge and discharge efficiency at the time-step t; S O C B , ⁡ min and S O C B , ⁡ max are the minimum and maximum of the state of charge; c a p is the rated power of the battery. The details of parameters of the battery model are shown in the Table 2.

To meet the power balance needs of household residents and the demand for excess photovoltaic power grid, HEMS needs to interact with the power grid for energy exchange, which can be expressed as: P L , t = P M , t + P C S C , t + ∑ i = 1 m + n P A , i S A , t (11) P G , t = P L , t − P P , t − P B , t S B , t − P E , t S E , t (12) where P L , t is the total power of the load; P A , t , and P M , t represent the power of dispatchable loads and non-dispatchable loads, respectively; P G , t is the interaction energy with the power grid; S E , t is the EV switch state.

This paper aims to minimize the total cost with consideration of comfort, so the optimization objective can be formulated as: min ⁡ C G − β C c o m (13) C G , t = P G , t R b , t , P G , t ⩾ 0 C G , t = P G , t C P , t , P G , t < 0 (14) C c o m = 1 − t s i − T s i Δ T i (15)

Where C G , t is the cost generated by the interaction energy between the system and the power grid; R b , t is the real-time electricity price prediction information (RTP) for the day ahead; C P , t is the electricity price for photovoltaic surplus electricity; β is a weight coefficient, which aims to balance the energy cost saving and maintenance of user’s comfort during the optimization process. C c o m is the index of comfort; t s i represents the actual starting time of the electrical appliance; T s i represents the desired starting time; Δ T is the allowed working time.

When applying the DRL algorithm, the optimization problem should first be modeled as a Markov Decision Process (MDP), which can be expressed as follows:

State set S: the state set is composed of the state of agent at each time-step t, which can be represents S = s 1 , s 2 , . . . , s t . The state of agent at each time-step t can be denotes as: s t = P P V , t , P f i x , t , P i n , t , P u n , t , S O C B , t , S O C E V , t , T i , t , R b , t , C P , t (16) where P f i x , t is the non-dispatchable load at time-step t.

Action set A: the action set is composed of the action of agent at each time-step t, which can be represents A = a 1 , a 2 , . . . , a t . a t = α i n , t , α u n , t , α a i r , t , P S O C , t , P E V , t (17) where α i n , t , α u n , t , α a i r , t are the switching variables of interruptible load, uninterruptible load and air conditioning, respectively; P S O C , t is the action of battery at the time-step; P E V , t is the action of battery at the time-step.

Reward function R: The reward at time t r t represents an immediate reward, which is obtained when the agent executes action a t based on state information s t . The real-time reward can be formulated as: r t = − C G − β C c o m (18)

Transition Probability P: once the current information (such as a t , s t ) is determined, the probability of transitioning to the next state s t + 1 is fixed.

Then, the modeled MDP can be solved by proposed DDPG algorithm to obtain the optimal dispatch strategy, which is illustrated in Figure 2. The DDPG algorithm, as an advanced deep reinforcement learning algorithm, is very suitable for solving complex multidimensional optimization problems in continuous action spaces (Zheng et al., 2023). In the DDPG algorithm, the policy function maps the state to the expected output, while the critical function maps the state and action to the expected maximum output R _t, which maximizes the action value function Q ^π (s _t, a _t). The calculation formula for the action value function Q ^π (s _t, a _t) is as follows: Q π s t , a t = E π G s t , a t + γ E a t + 1 ∼ π Q π s t + 1 , a t + 1 (19)

The DDPG algorithm is based on the actor critic framework, which consists of two main parts (actor network and critic network), with each part containing two networks (i.e., the main network and the target network). The actor network adjusts the value of the parameters θ ^μ in the policy function μ(s|θ ^μ ) by fitting the current state to the corresponding actions. The critic network is used to adjust the value of the parameters θ ^Q in the action-value function Q (s,a|θ ^Q ).

The parameters θ ^Q in the critic network are updated by minimizing the value of the loss function ✓(θ ^Q ), which is expressed as follows: E s , a Q s t , a t | θ Q − y t 2 (20) where y t = r t s t , a t + γ Q s t + 1 , μ s t | θ μ | θ Q .

In the actor network, the parameters θ ^μ are updated through the policy gradient function as follows: ∇ θ μ J θ μ ≈ E s t ∼ ρ β ∇ θ μ Q s , a | θ Q a = μ s | θ μ ∇ θ μ μ s | θ μ = E s t ∼ ρ β ∇ a Q ( s , a θ Q a = μ θ s ∇ θ μ μ s | θ μ (21) where ρ represents the discount factor; β represents the specific strategy corresponding to the current policy π.

In order to improve the stability and reliability of the learning process of the DDPG algorithm, two different target networks are added to the actor network and the critic network, respectively. They are the target actor network μ' (s|θ ^μ' ) and the target critic network Q' (s, a|θ ^Q' ). In each iteration, the weight factors (θ ^μ' and θ ^Q' ) will be soft updated according to the following formulas: S o f t   u p d a t e θ Q ′ ← τ θ Q + 1 − τ θ Q ′ θ μ ′ ← τ θ μ + 1 − τ θ μ ′ (22) where τ represents the soft update coefficient, and τ<< 1.

The specific training process of the proposed algorithm is described in Algorithm 1, which is shown as below:

Algorithm 1

Training procedures of proposed DDPG method.

1: Input: states of agent s t .

To verify the effectiveness of the proposed method, a smart home energy system is constructed. The simulation period is set as 1 day with 24 h from 00:00–24:00. There are six dispatchable devices in the home, which are shown in the Table 3. Note that the superscript “*” in the first column of Table 1 indicates the household appliance is an uninterrupted load. The PV generation and non-dispatchable load are shown in the Figure 3. The capacity of battery is 3 kWh, while the charging/discharging efficiency is 0.95. The minimum and maximum of the state of charge S O C min and S O C max are 0.2 and 0.9. To meet comfort constraints, the indoor temperature must be limited between 25°C and 27°C when the air-conditioning is running. The simulation model is constructed in MATLAB 2018b and the training procedure of DRL method is conducted in Python based on a workstation computer with 32 GB RAM and Intel Core i9-10920X CPU.

To obtain the optimal dispatch strategy of HEMS, the DDPG algorithm is applied. The hyper-parameters of the agent are set as the Table 4 shown. The total training episodes is 8,000 for ensuring convergence of agent. Besides, the learning rate of actor and critic network are set as 0.002 and 0.001 for ensuring the exploring ability and decision-making ability, respectively. The soft update coefficient and batch size are set as 0.001 and 256 to stable the training process.

At each episode, the agent gets current state from the constructed home energy system, and then give the decided action back. The changes of reward of the applied DRL method during the whole training episodes is illustrated in the Figure 4. It can be obtained that the reward stays in a low range with an average value −21 in the first 2000 episodes, which indicates that the agent cannot finds the optimal policy for HMS dispatching. Then, the reward rises gradually to −14 and then converges to −13 after the ceaseless interaction between agents and environment, which means the agent can obtain better strategy for dispatching the system.

After the agent is well-trained, the optimal energy management strategy for HES can be obtained. The results of the dispatch optimization for devices are presented in Figure 5. The needs of non-dispatchable devices are satisfied first. Then, the dispatchable devices should be dispatch with consideration of the real-time electricity price and permitted working interval of each device. It can be observed that all the uninterruptable devices are scheduled at a relatively low price point for saving the total cost. For example, the work time-point of washing machine is scheduled at 19:00 and 20:00 caused by the low price. Thus, the dispatch strategy of the uninterruptible devices is quite reasonable.

Furthermore, the interruptible devices can be dispatchable at discontinuous time-point, whose dispatch strategy can be more flexible. When dispatching the interruptible devices, the system cost should be the first and only consider factor. It can be obtained that the EV and Ebike are scheduled to charge during the 00:00–06:00 cause the lower electricity price. Therefore, both the interruptible and uninterruptible devices can be reasonably scheduled after the agent is well-trained, which means that the proposed method can effectively realize the HEM optimal operation.

When dispatching the air-conditioning device, the comfort factor should be taken into account. The indoor temperature changes like a non-linear process when the air-conditioning working. Thus, the air-conditioning does not need to working continuous with consideration of cost saving. The dispatching result of air-conditioning and the indoor temperature are shown in Figure 6. Note that the comfort constraint only set between 00:00–07:00 and 18:00–24:00. It can be obtained that the air-conditioning is scheduled to work at 5 hours for keep the indoor temperature between 25°C–27°C. As the temperature curve shows, the indoor temperature always stays between 25°C and 27°C, which indicates the comfort constraint can be well limited.

Generally, the energy storage device can store electricity during lower electricity price periods and release it during higher prices to reduce system costs. Thus, an energy storage device is equipped in the paper. The SOC curve of the applied energy storage device is illustrated in Figure 7. It can be found that the energy storage device charging when electricity price is low and discharging when the price is high, which can effectively reduce the system cost. Hence, the results effectively demonstrate that the proposed approach can efficiently schedule the energy storage device in real-time to reduce the operating cost after the agent is well-trained.

The above results have verified the effectiveness of the proposed method. To further verify the effectiveness and progressiveness of the proposed method, the proposed method is separately compared with the optimization method based on stochastic programming (SP) and the optimization method based on deterministic optimization (DO) (Alzahrani et al., 2023). The difference between SP and DO is that DO only consider optimization problems in deterministic scenarios, which does not consider uncertainties of PV and loads.

The optimization results of the three algorithms are shown in Table 5. Compared to traditional optimization methods, the proposed method can better cope with the uncertainty of PV output and load demand to achieve better optimization results. It can be obtained that the proposed method can achieve the lowest total cost compared with other two method, which the total operation cost can be reduced by 21.9% at least. The proposed method can reasonably schedule the different types of appliances for reducing the cost of purchasing electricity and improving revenue from selling electricity. Besides, the proposed can maintain the highest comfort for the home users by reasonably dispatching the switching time of air-conditioning. The DO method solves the modelled optimization problem under deterministic conditions, and the final cancelled optimization effect is not significantly different from the optimization effect of the proposed method. This also fully demonstrates the effectiveness of the proposed method. However, the DO method cannot address the issue of output uncertainty and is not applicable to actual operating conditions. Therefore, the proposed method is more suitable for optimizing the operation of the HES in uncertain environments.