Load forecasting has consistently been a vital concern for the power industry (Li et al., 2023b), as forecasting data enables power generation and load management departments to bolster their performance and reliability. In addition to economic and environmental considerations, load forecasting serves the following essential functions.

1) Comprehending load profiles allows power companies to devise rational electricity demand plans for customers, make economically prudent decisions, and mitigate risks for the organization.

The outcomes of load forecasting techniques are influenced by various factors. To obtain accurate predictions, it is crucial to consider the relevant factors of the dataset and use them appropriately. Numerous variables may affect the load forecasting performance. Here are some factors related to load forecasting.

For a long time, researchers have been dedicated to improving load forecasting techniques. However, when it comes to the specific modeling of load forecasting, there are still some obstacles.

First, weather is a key factor when performing load forecasting. Since the weather cannot be accurately estimated, it is impossible to accurately determine its impact on the load. Sudden weather changes can have significant effects on the expected load characteristics.

Second, the variety of meters utilized by consumers considerably influences load forecasting performance. Consumers employ an array of meters, encompassing smart and conventional meters, each with distinct measurement frequencies. As meter measurement frequencies and customer consumption behavior diverge, employing combined data for load forecasting may result in significant prediction errors.

Third, to further refine load forecasting, a series of other complex factors can be considered, but this adds to the difficulty of accommodating multiple variables, rendering the selection of an appropriate load forecasting model extremely challenging.

Fourth, since power systems may experience faults, power outages, and other intermittent events during dynamic operation, load forecasting models cannot account for such sudden occurrences, which also affect the load forecasting performance.

Fifth, consumer electricity demand is influenced by changes in economic market conditions or tariff changes. Although these economic factors significantly impact load forecasting outcomes, they are often overlooked by existing load forecasting methodologies.

In constructing a load forecasting model, the time dimension emerges as a critical factor influencing forecasting performance. Dynamic patterns can be discerned from historical load time series data. Consequently, employing neural networks adept at handling time series data can effectively extract inherent feature information and augment model accuracy. Long short-term memory (LSTM) networks, a unique variant of recurrent neural networks (RNNs), exhibit a natural advantage in processing sequential data (Hochreiter and Schmidhuber, 1997), as shown in Figure 1. The LSTM network manipulates the cell state through internal input gates, output gates, and forget gates, ultimately yielding their hidden state, as demonstrated in the ensuing equations: i t = s i g m o i d W x i x t + W h i h t − 1 (1) o t = s i g m o i d W x o x t + W h o h t − 1 (2) f t = s i g m o i d W x f x t + W h f h t − 1 (3) c t = f t ⊙ c t − 1 + i t ⊙ tanh W x c x t + W h c h t − 1 (4) h t = o t ⊙ tanh c t (5) where x t ∈ R n represents the input of the LSTM layer at time t ; i t , o t , f t , c t , and h t ∈ R m denote the input gate state, output gate state, forget gate state, cell state, and hidden layer state at time t , respectively; W x i , W x o , W x f , and W x c ∈ R m × n are all learnable parameter matrices. The symbol ⊙ denotes element-wise multiplication.

While the LSTM network has exhibited remarkable proficiency in managing time series data, the advent of the attention mechanism facilitates the extraction of pertinent information among features (Ruan et al., 2023c), thereby augmenting the model’s learning capacity and accuracy. Consequently, this article incorporates a TPA mechanism, grounded in the LSTM network, to bolster the load forecasting model’s ability to learn from historical load time series data.

First, a one-dimensional convolutional neural network (1-D CNN) layer is used to extract the feature learning capability of the LSTM network’s hidden state. Let h 1 , … , h t ∈ R m × t represent the hidden states of the LSTM layer, where dimension m represents the number of features and dimension t represents the time steps. The hidden states in the past t − 1 steps, i.e., H = h 1 , … , h t − 1 ∈ R m × t − 1 , are processed by the one-dimensional convolution operation, as follows: H i , j C = ∑ l = 1 T H i , t − T − 1 + l × C j , l (6) where the convolution operation H C ∈ R n × k is configured with k convolution kernels C j ∈ R 1 × T . The convolution kernels are applied along the row vectors of the hidden state matrix H to compute the convolution, extracting the temporal pattern matrix H C within the visual field of the convolution kernels. H i , j C represents the result value of processing the i th row vector of H with the j th convolution kernel.

Subsequently, a scoring mechanism is employed to evaluate the relevance between the hidden state h t and the row vectors of the convolutional temporal pattern matrix H C , as follows, s H i c , h t = H i c W a h t (7) where H i C ∈ R 1 × k represents the i th row vector of H C ; W a ∈ R k × m is the attention mapping matrix in the scoring mechanism.

By applying the Sigmoid activation function to the scoring mechanism, the attention coefficient α i is obtained, which represents the relevance between h t and H i C , making it easier to compare multivariate associations: α i = s i g m o i d s H i c , h t (8)

Based on the obtained attention coefficients, performing attention-weighted summation and addition operations yields the output under the TPA mechanism: h t ′ = W h h t + W v ∑ i = 1 n α i H i C T (9) where both W h ∈ R m × m and W v ∈ R m × k are learnable parameter matrices for the TPA layer, and h t ′ ∈ R m represents the hidden state after being processed by the LSTM layer and the TPA layer.

An overall model for the TPA-LSTM-based short-term load forecasting considering multi-regional factors is described in Figure 2, encompassing three modules. Module 1 employs TPA-LSTM to learn from historical load data, initially establishing a load baseline. Module 2 assimilates various factors from distinct regions, such as climate and economy, utilizing fully connected layers (FCLs) to learn diverse climate conditions, including temperature, humidity, wind speed, precipitation, and economic factors like market stability, electricity price adjustments, load control, and industrial growth. Module 3 constitutes the date information learning module, which examines the influence of varying seasons and typical days on electric load and integrates the output of Module 2 through a concatenation operation to learn the impact of date information on the electric load across various regions. Ultimately, the three modules enter the fusion layer and yield the final load profile in the form of a fully connected layer.

The determination of hyperparameters can be accomplished through a combined implementation of random search and k -fold cross-validation methodologies. Initially, a predefined search space is established to encompass the range of potential values for each hyperparameter. From this search space, a series of random samples is generated to explore and discover the possibly optimal combination of hyperparameters. Subsequently, in order to enhance the robustness of the load forecasting model, the entire dataset is divided into k subsets. During each iteration of training, the model is trained k times using different subsets as the training set and one subset as the validation set. This process ensures that each subset serves as the validation set exactly once throughout the iterations. Following the completion of k iterations, the model’s performance metrics are averaged over the training process. Finally, the set of hyperparameters that yields the best performance is selected for implementation in the load forecasting model.

The employment of the proposed load forecasting model is illustrated in Figure 3. The process begins with the careful preparation of the dataset, followed by the construction of the load forecasting model as demonstrated in Figure 2. Subsequently, the hyperparameters of the load forecasting model are selected by employing a combination of random search and k -fold cross-validation techniques. The finalized hyperparameters are then implemented in the load forecasting model, which undergoes training for practical application.

To evaluate the performance of the proposed load forecasting model, a historical load dataset encompassing all regions in Panama from 2015 to 2020 is employed for simulation. The data has a time granularity of 1 h and includes total load (MWh), temperature (°C), relative humidity (%), liquid precipitation (L/m2), wind speed (m/s), school day indicator (0/1), holiday indicator (0/1), and holiday index (integer) for three cities in Panama. The dataset is divided into training, validation, and test sets with a non-overlapping partition ratio of 8:1:1. The whole dataset starts from 3 January 2015, and ends by 27 June 2020. Accordingly, the sample sizes of the training, validation, and test sets are 1596, 199, 200, respectively.

In addition, three prevalent deep learning models serve as comparative models, as shown in Table 1. The MLP model learns climate information, economic factors, and date information to predict the load profile for the next 24 h. Based on the learning of external factors such as climate information and date information, the LSTM and GRU models learn from the historical load profile for the past week to predict the load profile for the next 24 h. Their model structure is similar to that shown in Figure 2, with the only difference being that they use LSTM and GRU instead of TPA-LSTM to process time-series data. The TPA-LSTM model predicts the load profile for the next 24 h by learning the impact of external factors on future loads while simultaneously extracting the features of historical load curves based on the temporal pattern attention mechanism.

During the model training process, standard deviation normalization transformation is applied to all data features to mitigate the influence of feature units on prediction outcomes. The mathematical expression for this transformation is as follows, x ∼ i k = x i k − x ¯ i σ x i (10) where x i k and x ∼ i k represent the original and transformed values of the k th sample in the i th feature of data, respectively; x ¯ i and σ x i represent the mean and standard deviation of the i th feature of data, respectively.

After all data is transformed, the models can be trained, with the mean squared error (MSE) as the training loss function, as follows, L o s s = 1 N ∑ i = 1 N y i − y ^ i 2 (11) where N is the total number of samples for training; y i and y ^ i are true value and prediction of the i th sample.

The training process of the four models is shown in Figure 4. It is clear that all models converge at the end of the training.

For visualization, the last week of data in the test set is selected to compare the predictive performance over the four models. These predicted results are denormalized to the normal scale, as shown in Figure 5. It is evident that the MLP model, which only focuses on climate information, economic factors, and date information, cannot accurately predict the load profile. The LSTM and GRU, two special types of RNNs, can fit the load profile to a certain extent based on the extraction of external factors, but still have significant errors. However, the TPA-LSTM model can not only capture the changes in the load itself but also pay attention to the impact of external factors on the load. The attention mechanism can focus on high-value features, thereby accurately predicting the electric load.

To comprehensively evaluate the performance of the four models, the mean absolute percentage error (MAPE), mean absolute error (MAE), and root mean squared error (RMSE) are used as indicators to statistically analyze the predictions of the four models on the test set, as follows, M A P E = 1 N ∑ i = 1 N y i − y ^ i y i (12) M A E = 1 N ∑ i = 1 N y i − y ^ i (13) R M S E = 1 N ∑ i = 1 N y i − y ^ i 2 (14)

The statistical results are shown in Table 2. It can be seen that the proposed TPA-LSTM model demonstrates superior predictive performance in all indicators. The reason is that this model not only takes into account the external factors for load variations, but also considers the influence from the historical load profile. More importantly, it employs the TPA mechanism that can identify valuable features and eliminate irrelevant ones to improve the model performance.

In summary, the model proposed in this article processes historical load data based on the TPA mechanism to establish a baseline for load forecasting. It also uses fully-connected layers to extract the impact of external factors (such as regional climate information, economic factors, and date information on the load), thereby accurately predicting the load curve. According to the MAPE statistics, the model has an error of only 4.41%, making it suitable for use in actual load forecasting models.