Leveraging sanitized data for probabilistic electricity market prediction: a Singapore case study

Abstract

In deregulated electricity markets, predicting price and load is a common practice. However, market participants and shareholders often seek deeper insights into other system statuses associated with price prediction, such as power flow and market share of generation companies (GenCos). These insights are challenging to obtain using purely data-driven methods. This paper proposes a physics-based solution for the probabilistic prediction of market-clearing outcomes, using real sanitized offer data from the National Electricity Market of Singapore (NEMS). Our approach begins with approximating the generator offers that have been historically cleared. Using this pool of offer data, we propose a probabilistic market-clearing process. This process allows for the probabilistic prediction of market prices. By considering the power system network and its constraints, we also naturally obtain probabilistic predictions of power flow and market shares. We validate our approach using actual NEMS data. Our findings show that while the overall performance of price prediction is comparable to existing methods, our proposed method can also provide probabilistic predictions of other associated system operating conditions. Furthermore, our method enables scenario studies, such as the impact of demand-side participation and the penetration of rooftop photovoltaic (PV) systems on the Uniform Singapore Energy Price (USEP).

1 Introduction

The past few decades have witnessed the liberalization of electricity markets all over the world (Bunn et al., 2021; Yang et al., 2019), including the National Electricity Market of Singapore (NEMS) (Services, 2021; Energy Market Authority of Singapore, 2018). As a fundamental aspect of these markets, electricity price prediction has been the subject of extensive research (Zhao et al., 2008; Abedinia et al., 2017; Wan et al., 2017; Monte et al., 2018; Nowotarski and Weron, 2018; Brusaferri et al., 2019; Chai et al., 2019; Afrasiabi et al., 2020; He et al., 2020; Hong et al., 2020; Li et al., 2020; Fraunholz et al., 2021; Taylor, 2021; Uniejewski and Weron, 2021; Meng et al., 2022; Heidarpanah et al., 2023). However, most existing tools prioritize price prediction, often overlooking the comprehensive insights that additional system parameters, such as power flow and market share of generation companies (GenCos), can offer. The repercussions of such an oversight can be manifold. Without these nuanced insights, prediction models could inadvertently misjudge critical supply–demand imbalances at specific nodes. This, in turn, can lead to potential inaccuracies in price forecasts. GenCos, if left uninformed about these pivotal parameters, might grapple with challenges in streamlining their offer-making decisions, inadvertently paving the way for market inefficiencies. Furthermore, the lack of these comprehensive insights might foster an environment ripe for market volatility, with speculative bidding amplifying price fluctuations.

Over the past 2 decades, prediction techniques have significantly advanced (Zhao et al., 2008; Abedinia et al., 2017; Wan et al., 2017; Monte et al., 2018; Nowotarski and Weron, 2018; Brusaferri et al., 2019; Chai et al., 2019; Afrasiabi et al., 2020; He et al., 2020; Hong et al., 2020; Li et al., 2020; Fraunholz et al., 2021; Taylor, 2021; Uniejewski and Weron, 2021; Meng et al., 2022; Zhou et al., 2022; Heidarpanah et al., 2023). Point forecasting techniques, for instance, have been widely adopted in this field (Abedinia et al., 2017; Fraunholz et al., 2021; Hong et al., 2020; Heidarpanah et al., 2023). In the NEMS, the market operator publishes point forecasts for demand and the Uniform Singapore Energy Price (USEP) (Energy Market Company, 2023). Probabilistic forecasting, however, goes a step further. It reflects the probabilistic and heteroscedastic nature of electricity price and load, providing predictions in the form of intervals (Zhao et al., 2008; Wan et al., 2017; Taylor, 2021; He et al., 2020), quantiles (Uniejewski and Weron, 2021), and densities (Chai et al., 2019; Brusaferri et al., 2019; Li et al., 2020; Afrasiabi et al., 2020). Zhao et al. (2008) and Wan et al. (2017) construct optimal prediction intervals. Taylor (2021) proposes expectile-bounded intervals. He et al. (2020) introduce a method based on a deep neural network model. To address the vulnerability of quantile regression averaging (QRA) to low-quality predictors, a regularized variant of QRA is proposed (Uniejewski and Weron, 2021). Density forecasting, which provides comprehensive uncertainty information, has its own advantages. Chai et al. (2019) propose a reliable strategy for constructing predictive densities oriented toward continuous ranked probability scores. A Bayesian deep learning-based technique is developed by Brusaferri et al. (2019). Li et al. (2020) combine density probabilistic load forecasts to enhance the performance of the final probabilistic forecasts. Afrasiabi et al. (2020) construct a deep neural network (DNN) model from historical data to directly predict the probability density function (PDF) of residential loads based on past time series.

Many AI-based models inherently focus on identifying patterns in historical data, often sidelining real-world grid considerations, such as transmission capacities, voltage limits, and other network-related constraints. This data-driven approach can lead to predictions that, while statistically accurate, are operationally infeasible, particularly in complex scenarios or during events that stress the grid. Furthermore, these models typically struggle to derive additional system parameters, such as local marginal price (LMP), power flow, and the market share of GenCos, which are crucial for a comprehensive understanding of market dynamics. The “black-box” nature of these models also presents challenges in interpretability, making it difficult to understand the rationale behind certain predictions.

In contrast, simulation-based prediction methods, which account for the intricacies of the power system network, offer a way to obtain these additional system parameters. However, there exists limited literature on this topic. Ji et al. (2017) provide a forecasting method from the vantage point of a system operator who has access to system operating conditions. In a short-term forecasting algorithm proposed by Zhou et al. (2011), supply-offer behaviors are assumed static. Quadratic cost function and lossless power flow are also assumed. Bo and Li (2009) investigate the impact of load uncertainty on LMP forecasting. In these studies, GenCos’ offers are often assumed or assigned as fuel-cost-based functions due to the sanitization of generator offer data in a typical electricity market.

One approach to this challenge is to approximate offers from accessible historical data. In Durvasulu and Hansen (2018), generator types are identified and clustered using only publicly available data in the PJM market. Market-based cost functions are then fitted and used in optimal power flow (OPF) calculations to obtain the marginal cost. However, this approach becomes more complicated due to the variability of available data across different markets. For instance, in markets such as the NEMS, the available dataset is more limited. Not only is the generator ID sanitized, but so is the offer quantity. LMP is also not available, as detailed in Figure 3, Section 2.

The proposed methodology is validated using real NEMS data. To the best of our knowledge, this work is the first to provide a probabilistic prediction for price, market share, and power flow using real-world sanitized data.

The framework for typical prediction methods is shown in Figure 1A. Figure 1B shows how we address the four points mentioned previously. To tackle the data availability issue (point 1), the first step is to estimate historically cleared offers based on accessible data, for which we develop an optimization method. In the second step, we sort the offer data estimated from the first step using a clustering technique. We derive clusters of functions, representing possible offers using regression, and then assign them to generators, each with associated conditional probabilities. The obtained offer curves enable the calculation of optimal power flow (OPF), addressing point 2. In the final step, we propose a conditional probabilistic market-clearing process. With load forecast as the input and using Monte Carlo simulation, we can obtain probabilistic estimates of price along with other market-clearing outcomes, addressing point 3. Since our methodology is physics-based, we can conduct scenario studies with assumed system parameters, addressing point 4.

FIGURE 1

The remainder of this paper is structured as follows: Sections 2–4 detail the proposed methodology against the backdrop of the NEMS. A solution for the historical offer estimation with sanitized information is proposed in Section 2. The formulation of conditional offer functions is presented in Section 3. A probabilistic clearing process for market prediction is proposed in Section 4. In Section 5, actual NEMS data are used to validate the methodology. Its performance is benchmarked against both classical and advanced prediction methods. In Section 6, future scenarios are studied. Section 7 concludes the paper.

2 Historical offer estimation

2.1 Market basics

In the NEMS, the following data are available either publicly or through subscription (Energy Market Company, 2023; Energy Market Authority, 2023): historical USEP and load for each area, registered information of generating units, overhaul schedule (i.e., generator’s availability) and its marginal price (i.e., offer price cleared historically), daily market share of GenCos, and load forecasts. The 230 kV and 400 kV transmission network lines of the Singapore power system are available in Limited (2016), Joseph et al. (2021), and Global Energy Network Institute, 2023.

Figure 2 shows forms of offers and pricing throughout the market-clearing process, as well as their data availability in the NEMS. Like many modern electricity markets, the NEMS uses nodal pricing. This means that, subject to the physical properties and constraints, electricity price varies across transmission nodes. Unique to the NEMS, while generators are paid the nodal price (local marginal price or LMP), consumers are charged a uniform price to prevent locational disadvantages. This is referred to as the Uniform Singapore Energy Price or USEP.

FIGURE 2

Definition (Uniform Singapore Energy Price): USEP is the weighted average of LMP over all the nodes that withdraw energy:where and are LMP of node and demand at period , respectively. is the total number of buses in the system.

2.2 Approximating historically cleared offers

An offer in the context of electricity markets consists of a price and its corresponding quantity. As illustrated in Figure 2, in the NEMS, only cleared offer price and historical USEP are available. As outlined in Figure 1, our initial step is to approximate the offers that have been cleared historically. To achieve this, we formulate an optimization problem. The objective of this problem is to approximate the historical quantity (as shown in the second block of Figure 2) in such a way that the resulting USEP estimation closely matches the recorded values (2). In this paper, the terms “offer” and “price–quantity pair” are used interchangeably.

Minimize:where and are the actual and estimated USEP at the th period, respectively.

In the setup of this problem, and in (1) are known, and the LMP is determined by the marginal price and quantity of the generators. Therefore, (1) becomes

In this paper, represents the price ($/MWh), while is the quantity (MW). The LMP at bus is defined as the price to serve the next MW of load at that location as expressed in (4):where is the total production cost of all generators. The estimated LMP also depends on the generators’ accepted offers:where and denote vectors of accepted prices and estimated quantities cleared of all generators at as expressed in (6) and (7), respectively:where and represent the price and estimated quantity cleared historically, respectively, for the th generator from the th GenCo at period . represents the total number of GenCos in the system, and is the number of generators in the th GenCo.

In addition to the typical constraints for DC power flow calculations, such as power limits and balance, and branch limits, the function is also subjected to an additional constraint—GenCos’ market shares. In the NEMS, the historical data of each GenCo’s daily market share are known, as represented by (8):where is the total daily production of GenCo . It is worth noting that only the output of GenCos, rather than individual generators, is given. A GenCo typically owns multiple generators, hence the summation in (8). Furthermore, if generators under the same GenCo have the same type and capacity, and their cleared prices are the same, we assume that they share the same quantity, as expressed in (9):

As can be seen from the aforementioned formulation, the optimization aims to estimate historically accepted offer quantity so that the resultant USEP for each period closely matches the actual USEP, while observing all transmission limits and additional market constraints. Given the typically large number of generators and their capacities, there could be many solutions to . (8) and (9), along with network constraints, help narrow down the solution space. If a solution cannot be found, (9) is relaxed.

2.3 Solution

The solution to the aforementioned problem is illustrated in Figure 3. In the present study, we have chosen to employ the self-adaptive differential evolution (SaDE) algorithm, as detailed in Qin et al. (2009), given its demonstrated efficiency in optimization tasks. One of the primary advantages of SaDE over traditional evolutionary algorithms (EAs) is its intrinsic ability to converge more rapidly. This accelerated convergence is achieved through SaDE’s unique capability to adaptively select learning strategies and fine-tune control parameters throughout the evolutionary process (Qin et al., 2009; Qin and Suganthan, 2005). Unlike many other algorithms that require manual parameter tuning, SaDE’s self-adaptive mechanisms render it more user-friendly and often more robust across a variety of optimization challenges. By dynamically balancing exploration and exploitation phases, SaDE reduces the risk of premature convergence, often a challenge in traditional EAs. Furthermore, the algorithm’s adaptability means that it is less sensitive to initial parameter settings, thereby offering a level of flexibility seldom found in conventional differential evolution approaches. This dynamic and adaptive nature of SaDE ensures a holistic and efficient optimization process, making it a compelling choice for our research.

FIGURE 3

For corroboration, basic EA options, such as the genetic algorithm (GA), are also employed. The power flow can be solved using MATPOWER (Zimmerman et al., 2011). In this problem, the offer quantity is the variable, while the price is given. Therefore, the cost/offer functions for generators are constant functions within each iteration.

3 Conditional probabilistic offer functions

Once the estimated offer quantity for each period is determined, we form the historical data pool of offers. Assuming that a generator would not significantly change its offer strategy, each data point () represents a possible offer the generator might make in the future. However, those data points cannot be directly used in market-clearing or OPF calculation, where offer functions are required.

Figure 4 plots the historical price–quantity pairs of a typical generator (the 8th) in the NEMS. It can be observed that most of the time, the cleared offer price is under 300 Singapore dollars per MWh. However, there is also a possibility that the price may exceed 1,000 $/MWh. In this paper, “$” denotes the Singapore dollar.

FIGURE 4

Second, generators choose their offer submissions based on various market conditions. As a result, these multiple choices of offers need to be modeled. In this part of our model, we form offer clusters and regress offer functions. Each cluster/function is associated with its respective probability. The framework for this step is summarized in Figure 5, with the details provided in the following subsections.

FIGURE 5

3.1 Clustering

To reflect the decision-making process of GenCos, the offers obtained in the first step (Section 2) are classified into clusters. Each cluster represents a group of price–quantity pairs from which a generator might choose its offer. For instance, as depicted in Figure 4, the offers made by generator 8 can be clustered into four groups.

There are many clustering techniques available. In this paper, we use -means clustering (Arthur and Vassilvitskii, 2006). The probability of selecting an offer from cluster is proportional to the number of data points in that cluster, as shown in (10).where is the probability of cluster and is the number of data points in the cluster.

After clusters are formed, an offer function is used as a representation of each cluster.

3.2 Conditional offer-making

Variables that influence electricity prices are listed in Chai et al. (2019). Unlike the Nordic system (Chai et al., 2019), the Singapore power system primarily relies on domestic generation, with no presence of hydropower and nuclear power. Among the accessible data in the NEMS, conditions affecting generators’ offer-making include demand, generation, time of the day, and day of the week. Under these conditions, (10) becomeswhere is the probability for an offer to be chosen from the th cluster, is the total capacity of generation, is the load demand, is the period of the day, and is the number of data points in cluster under conditions. With this conditional filter applied, the data points are refined and better represent offers made/cleared under conditions. Figure 6 shows offer data filtered with conditions.

FIGURE 6

Within each filtered cluster of the dataset, conditional offer functions can be obtained using either linear or polynomial regression. In this paper, linear regression is used, as illustrated in Figure 6. For this combination of conditions (Figure 6A), the probabilities associated with the four offer functions (from bottom to top) are 94.51%, 4.38%, 1.09%, and 0.36%, respectively. In addition, the set of offer curves varies with the conditions. For example, the offers made by the same generator during an early period of the day could only exist in the lowest cluster, as shown in Figure 6B.

FIGURE 7

4 Probabilistic market-clearing

The concept of probabilistic market-clearing is integral to our approach. Instead of relying on deterministic values, we employ a probabilistic method that accounts for uncertainties inherent in the electricity market. The crux of our probabilistic market-clearing method lies in the utilization of Monte Carlo simulation. This simulation leverages the conditional offer functions and their respective probabilities, which we previously derived. The process involves multiple iterations, each time selecting offer functions based on their conditional probabilities and subjecting them to random sampling.

For each iteration, the selected offer functions play a pivotal role in market-clearing. Here, we calculate the LMP and USEP using Eq. 3. This iterative method ensures that the resulting solution is stable and reflective of market dynamics. To ensure convergence and computational efficiency, we have set specific criteria: the simulation halts when the change in variance across 1,000 consecutive iterations is less than 0.1%. An upper limit of 1,000,000 iterations is set to prevent excessive computation.

The primary objective of market-clearing in the NEMS, in the absence of demand-side bidding, is to minimize the total cost, represented as . For the detailed computation of power flow and network losses, we employ MATPOWER (Zimmerman et al., 2011), a widely acknowledged tool in the domain. For an in-depth understanding of the market-clearing process, especially its formulation in the NEMS, readers are directed to refer to Zhou et al. (2016) and Gao et al. (2017).

5 Numerical study

Data from the NEMS spanning from 1 January 2019 to 30 September 2019 are used for verification. During this period, there were 46 registered generators in the NEMS, with their capacities ranging from 4.8 to 431 MW. The total capacity was 9.51 GW, and the peak load was 7.24 GW. There were 48 trading/dispatch periods in a day. The proposed methodology is used for day-ahead market prediction. A 3-week dataset is selected for testing, while the remaining datasets are used for modeling.

5.1 Historical offer estimation

The proposed methodology for historical offer estimation (Section 2) is implemented in the MATLAB environment. Offer quantities for 12,096 periods (269 days) are estimated. Given the computationally intensive calculation involved, the high-performance computing platform of Nanyang Technological University (Nanyang Technological University, 2021) is used. A total of 400 CPU (AMD EPYC™ 7702 2.0 GHz) cores/workers are deployed in parallel. Using the SaDE algorithm (Qin et al., 2009), it takes approximately 40 h to converge. It should be noted that this step of offer approximation is carried out offline, as the estimated offers serve as the basis for the following steps and would not change once calculated.

Figure 8 shows the histogram of the values of the objective function (1). The average of the USEPs for the 9-month period is 98.28 $/MWh. The mean error between the calculated and actual USEPs ((1)) is 0.084 $/MW (i.e., 0.084% of the average USEP); 99.52% of the cases result in errors less than 1%.

FIGURE 8

5.2 Comprehensive market prediction results

With the estimated historical price–quantity pairs, we can determine offer functions with conditional probabilities (Section 3), followed by USEP forecasting using probabilistic market-clearing (Section 4). The number of clusters is set to 7. The prediction results for a typical day and a day with price spikes are shown in Figures 9A, B, respectively.

FIGURE 9

For a typical day in the NEMS, the USEP is approximately 100 $/MWh in 2019. However, there are possibilities to witness price spikes, especially during the periods from 10:30 to 12:00 and 13:00 to 14:00. The distribution of the prediction is more converged during the early hours. This is because, historically, the prices during early hours are most likely to settle approximately 100 $/MWh.

Generally speaking, the prediction errors mainly stem from errors in 1) the offer estimation (Section 2) and 2) the clustering, filtering, and linear regression (Section 3). The benchmark for USEP prediction and its comparison with other methods are discussed in the following subsection.

The prediction results for daily market share are shown in Figure 10. The predicted results for typical normal days are close to the mean values (Figure 10A), whereas the prediction is more scattered for a day with price spikes (Figure 10B). This is because generators’ offers are more likely to settle in lower price tiers on a normal day, leading to more converged results, as shown in Figure 10A. This is consistent with the observations in price prediction, as shown in Figure 9. It is important to remember that in the NEMS, only daily market share is available (according to (8)). The calculated market shares for each period and each generator (Section 2) could differ from the actual market shares.

FIGURE 10

5.3 Performance benchmarking

The performance of USEP prediction is evaluated against both classical and advanced prediction models. These include a basic empirical unconditional density forecast model (EU), a classical time-series model—generalized autoregressive conditional heteroskedastic model (GARCH) with the Gaussian error distribution (Jónsson et al., 2014), and three AI models—an extreme learning machine logistic continuous ranked probability score-based ensemble model output statistics (ELC-EMOS) (Chai et al., 2019), a regularized quantile regression averaging method which utilizes the least absolute shrinkage and selection operator (LQRA) (Uniejewski and Weron, 2021), and a combined model (Combine) (Li et al., 2020).

The continuous ranked probability score (CRPS) is used for benchmarking. The CRPS measures the closeness of forecast distribution to the corresponding observation and is one of the popular indices for measuring the performance of probabilistic prediction (Chai et al., 2019; Li et al., 2020; Afrasiabi et al., 2020; Matheson and Winkler, 1976; Hersbach, 2000; Gneiting et al., 2007; Gneiting and Raftery, 2007; Zhang et al., 2020). The same NEMS data are used for training and testing, respectively. However, network constraints are not considered in the benchmark models. For quantile forecast, quantiles 0.01 to 0.99 are estimated in steps of 0.01.

Table 1 presents a comparative analysis of daily average CRPS for selected testing cases, bifurcated into normal days and days with price spikes. Among the selected 21 testing cases, two groups—normal days and days with spikes—are formed and benchmarked. Among them, 11 are normal days in which the daily average USEP varies from 87.30 to 107.61 $/MWh, and 10 days with price spikes in which the daily average varies from 112.01 to 520.58 $/MWh. The smaller the CRPS score, the better the performance.

5.3.1 Classical methods

Among the classical methods, the EU stands out for its simplicity, relying on a constant empirical distribution. Information such as , , and in Section 3.2 is not applied in the EU. However, its performance lags behind, as evidenced by the highest CRPS scores. The GARCH model shows improvement, attributing its better performance to its heteroscedastic modeling (Jónsson et al., 2014).

5.3.2 AI-based methods

Turning to AI-based methods, they collectively outperform their classical counterparts. The combined method slightly edges out others in both normal days and days with spikes. This indicates the potential of AI in capturing intricate patterns and nuances in the data.

5.3.3 Proposed method

Our proposed method outperforms, especially during days with price spikes. While it competes closely with AI-based methods on normal days, it takes a clear lead during volatile days. When comparing the proposed method, for normal days, our proposed method is on par with AI-based methods, while AI methods have slight edges. The average CRPSs for ELC-EMOS, LQRA, Combine, and the proposed method are 7.89, 8.15, 7.51, and 8.16 $/MWh, respectively. As discussed in Section 3.1 and Section 3.2, increasing the number of clusters can be one solution to improve the results. The offer information is distilled and represented by a limited number of linear functions. For a given number of explanatory information (i.e., conditional filter , , and ), an increased cluster number means less loss of information due to regression. However, more historical data are also required accordingly.

For days with spikes, on the other hand, the proposed method takes a clear lead in terms of CRPS. The average CRPSs for ELC-EMOS, LQRA, Combine, and the proposed method are 34.80, 33.95, 32.63, and 30.22 $/MWh, respectively. This can be mainly attributed to the following reasons: first, USEP in Singapore context is a form of local marginal pricing that considers the grid’s network property, which conventional AI-based prediction models do not take as the input. Compared with normal days in which offers are cleared at similar prices, the network’s effect on LMP becomes pronounced when more generators’ offers are cleared at high price points. As a result, the LMP on the bus increases. As the weighted average of LMPs (1), USEP also increases.

The advantage of our method can be ascribed to two key factors:

Network influence on LMP: The inherent network properties of the Singapore grid play a pivotal role in shaping USEP, especially during high-clearance offers. Traditional AI models (Wan et al., 2017; Chai et al., 2019; Wan et al., 2014) often overlook this constraint. Our method, however, integrates these network effects, leading to enhanced prediction accuracy.

Data constraints on AI models: While AI-based methods typically thrive on large datasets, our study was constrained to 9 months of data. This limitation could pose challenges in tuning AI models, potentially leading to suboptimal performance.

The results underscore the importance of considering network properties and constraints in electricity market predictions. While AI offers promise, its efficacy is often contingent upon data availability. Our proposed method, which seamlessly blends these considerations, emerges as a robust solution, especially during market volatilities.

6 Scenario study

6.1 Impact of demand-side participation

The proposed method also has the advantage of being able to study future scenarios with varied system parameters. For instance, the NEMS is actively considering consumers’ participation in the price discovery process, such as demand-side bidding (Energy Market Authority, 2013). To study the impact, fixed-rate pricing schemes for demand-side bidding are assumed. Apart from the pricing, another influencing factor is the penetration level of the dispatchable load, defined here as the proportion in relation to the annual peak load. In this case, 1% of the penetration level is equivalent to 72.37 MW. The dispatchable load is assumed to be distributed across the whole network in proportion to demand on each node. In our model, it is treated as generators with negative output. A price floor of 300 $/MWh for demand-side bidding in the NEMS is designed to address the potential gaming issue (Energy Market Authority, 2013).

By varying the price and the penetration level of the dispatchable load in the system, the changes in the average USEP can be obtained, as shown in Figure 11. It can be seen from the figure that USEP drops with the demand-side’s participation. For a given amount of dispatchable load, the reduction in USEP decreases with the increase in the bidding price. For instance, with 1% penetration and 300 $/MWh bidding price, the USEP reduction is 1.30 $/MWh (1.32% of the average USEP). Increasing the price up to 500 $/MWh, the USEP reduction remains similar. The impact of demand-side bidding dwindles when its price is between 600 and 1,000 $/MWh. Beyond 1,000 $/MWh, its effect on USEP reduction diminishes. The higher the bidding price, the smaller the chance it can be cleared. This observation becomes more noticeable with higher penetration levels.

FIGURE 11

The second observation concerns the penetration level. Given the bidding prices, the USEP continuously reduces as the capacity of dispatchable load increases, but not always in a linear way. When the price is low, for example, 300 $/MWh, the USEP reduction is almost in a linear relationship with the penetration level. As the price increases, especially approximately from 600 to 900 $/MWh, the USEP reduction does not grow in proportion with the penetration level. The explanation is that at lower prices, bids from the demand-side compare favorably with generators. However, at higher price levels, offers from traditional generators start to compete despite increased participation of flexible load. From the aforementioned results, the flexible load can hardly be cleared when the bidding price is higher than 1,000 $/MWh.

6.2 Impact of rooftop PV penetration

Singapore’s access to renewable energy sources is limited, with solar photovoltaics (PVs) being one of the few options. Given the country’s land constraints, rooftop PV installations have become a popular solution. We obtained power measurement data from a typical commercial building equipped with solar panels. This building consumes approximately 60 MWh of energy daily, and its rooftop PV system has a capacity of 1 MWp. Commercial loads account for 35% of the Singapore’s total demand, with the majority located in the central area.

We studied the impact of rooftop PV systems on USEP by assuming that similar PV systems are installed on other commercial buildings. The penetration level of rooftop PV, defined as the installed capacity, ranges from 0 to 1 MWp per 60 MWh commercial load. Offer strategies from GenCos are assumed to remain the same. Table 2 shows the average daily USEP reduction under different rooftop PV penetration levels.

The table reveals that an increase in PV capacity indeed lowers the electricity price by reducing demand. The price drop is approximately proportional to the PV penetration level. With the highest coverage (1.0 MWp per 60 MWh of demand), the PV system can support up to 2.46% of the total daily demand on a sunny day (solar radiation intensity ranging from 800 to 1000 W/m² around solar noon, a typical intensity for clear days in Singapore’s equatorial climate (The National Environment Agency, 2023)). In our experiments, this translates into a modest reduction (0.93%) in the daily USEP.

The limited impact can be attributed to three contributing factors: 1). PV can only operate during daylight hours (approximately 10 h), leaving the USEP unchanged for the remaining periods. Additionally, sunlight availability is not always guaranteed; 2). the penetration of PV is still relatively limited when installed solely on commercial buildings’ rooftops; and 3). the USEP is the weighted sum of LMP across all nodes, which means the overall impact is diluted on average.

7 Conclusion

In essence, this work offers a robust and adaptable toolkit for electricity market prediction, promising expansive applications and avenues for future exploration.

8 Future work

While our methodology provides a comprehensive approach to predicting GenCo’s offer-making decisions, it is worth noting that we have based our predictions on a set of factors including demand, generation, time of the day, and day of the week. In real-world scenarios, GenCos often base their decisions on a myriad of factors, especially in situations leading to price hikes. For instance, external factors, such as weather patterns, can significantly influence energy demand, especially with the increasing penetration of renewable energy sources. Similarly, broader economic indicators can hint at potential changes in energy consumption patterns.

To enhance the predictive accuracy of our model, future research endeavors should delve deeper into integrating these additional factors. Incorporating weather data, for example, can provide insights into potential changes in demand due to temperature variations. Similarly, studying economic indicators can give a clearer picture of how broader economic trends might influence energy consumption and, consequently, GenCo offers. While our study has been tailored to the unique intricacies of the Singaporean electricity market, it would be invaluable to apply this methodology to other electricity markets. This would not only test the model’s universal applicability but also highlight potential adjustments required to cater to different market dynamics and structures.

References

• 1

  AbediniaO.AmjadyN.ZareipourH. (2017). A New feature selection technique for load and price forecast of electrical power systems. IEEE Trans. Power Syst.32 (1), 62–74. 10.1109/tpwrs.2016.2556620

  • CrossRef
  • Google Scholar
  • View reference in article

• 2

  AfrasiabiM.MohammadiM.RastegarM.StankovicL.AfrasiabiS.KhazaeiM. (2020). Deep-based conditional probability density function forecasting of residential loads. IEEE Trans. Smart Grid11 (4), 3646–3657. 10.1109/tsg.2020.2972513

  • CrossRef
  • Google Scholar
  • View reference in article

• 3

  AltmanN. S. (1992). An introduction to kernel and nearest-neighbor nonparametric regression. Am. Stat.46 (3), 175–185. 10.2307/2685209

  • CrossRef
  • Google Scholar
  • View reference in article

• 4

  ArthurD.VassilvitskiiS. (2006). k-means++: the advantages of careful seeding. Stanford.

  • Google Scholar
  • View reference in article

• 5

  BoR.LiF. (2009). Probabilistic LMP forecasting considering load uncertainty. IEEE Trans. Power Syst.24 (3), 1279–1289. 10.1109/tpwrs.2009.2023268

  • CrossRef
  • Google Scholar
  • View reference in article

• 6

  BrusaferriA.MatteucciM.PortolaniP.VitaliA. (2019). Bayesian deep learning based method for probabilistic forecast of day-ahead electricity prices. Appl. Energy250, 1158–1175. 10.1016/j.apenergy.2019.05.068

  • CrossRef
  • Google Scholar
  • View reference in article

• 7

  BunnD. W.InekweJ. N.MacgeehanD. (2021). Analysis of the fundamental predictability of prices in the British balancing market. IEEE Trans. Power Syst.36 (2), 1309–1316. 10.1109/tpwrs.2020.3015871

  • CrossRef
  • Google Scholar
  • View reference in article

• 8

  ChaiS.XuZ.JiaY. (2019). Conditional density forecast of electricity price based on ensemble ELM and logistic EMOS. IEEE Trans. Smart Grid10 (3), 3031–3043. 10.1109/tsg.2018.2817284

  • CrossRef
  • Google Scholar
  • View reference in article

• 9

  DurvasuluV.HansenT. M. (2018). Market-based generator cost functions for power system test cases. IET Cyber-Phys. Syst. Theory Appl.3 (4), 194–205. 10.1049/iet-cps.2018.5046

  • CrossRef
  • Google Scholar
  • View reference in article

• 10

  Energy Market Authority (2023). Energy market authority. [Online]. Available at: https://www.ema.gov.sg/.

  • Google Scholar

• 11

  Energy Market Authority (2013). Implementing demand response in the national electricity market of Singapore - final determination paper.

  • Google Scholar

• 12

  Energy Market Authority of Singapore (2018). Intermittency pricing mechanism for intermittent generation sources in the national electricity market of Singapore: final determination paper.

  • Google Scholar

• 13

  Energy Market Company (2023). Energy market Company. [Online]. Available at: https://www.emcsg.com/.

  • Google Scholar

• 14

  FraunholzC.KraftE.KelesD.FichtnerW. (2021). Advanced price forecasting in agent-based electricity market simulation. Appl. Energy290, 116688. 10.1016/j.apenergy.2021.116688

  • CrossRef
  • Google Scholar

• 15

  FriedmanJ. H.BentleyJ. L.FinkelR. A. (1977). An algorithm for finding best matches in logarithmic expected time. ACM Trans. Math. Softw.3 (3), 209–226. 10.1145/355744.355745

  • CrossRef
  • Google Scholar

• 16

  GaoY.ZhouS.ShuZ.BengG. H. (2017). “Market clearing model for Singapore electricity market incorporating transmission loss,” in 2016 Asian Conference on Energy, Power and Transportation Electrification (ACEPT 2016), Singapore, 25-27 October 2016 (IEEE).

  • Google Scholar

• 17

  Global Energy Network Institute (2023). Global energy network Institute. [Online]. Available at: http://www.geni.org/.

  • Google Scholar

• 18

  GneitingT.BalabdaouiF.RafteryA. E. (2007). Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. B Stat. Methodol.69 (2), 243–268. 10.1111/j.1467-9868.2007.00587.x

  • CrossRef
  • Google Scholar

• 19

  GneitingT.RafteryA. E. (2007). Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc.102 (477), 359–378. 10.1198/016214506000001437

  • CrossRef
  • Google Scholar

• 20

  HeH.LuN.JiangY.ChenB.JiaoR. (2020). End-to-end probabilistic forecasting of electricity price via convolutional neural network and label distribution learning. Energy Rep.6, 1176–1183. 10.1016/j.egyr.2020.11.057

  • CrossRef
  • Google Scholar

• 21

  HeidarpanahM.HooshyariporF.FazeliM. (2023). Daily electricity price forecasting using artificial intelligence models in the Iranian electricity market. Energy263, 126011. 10.1016/j.energy.2022.126011

  • CrossRef
  • Google Scholar

• 22

  HersbachH. (2000). Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecast15 (5), 559–570. 10.1175/1520-0434(2000)015<0559:dotcrp>2.0.co;2

  • CrossRef
  • Google Scholar

• 23

  HongY. Y.TaylarJ. v.FajardoA. C. (2020). Locational marginal price forecasting using deep learning network optimized by mapping-based genetic algorithm. IEEE Access8, 91975–91988. 10.1109/access.2020.2994444

  • CrossRef
  • Google Scholar

• 24

  JiY.ThomasR. J.TongL. (2017). Probabilistic forecasting of real-time LMP and network congestion. IEEE Trans. Power Syst.32 (2), 831–841. 10.1109/tpwrs.2016.2592380

  • CrossRef
  • Google Scholar

• 25

  JónssonT.PinsonP.MadsenH.NielsenH. A. (2014). Predictive densities for day-ahead electricity prices using time-adaptive quantile regression. Energies (Basel)7 (9), 5523–5547. 10.3390/en7095523

  • CrossRef
  • Google Scholar

• 26

  JosephA.GaoZ.KongX.XuN. Z.NginH. T. (2021). Dynamic modelling and simulation of Singapore transmission network (230/400kV). Technology Disclosure, (NTU Ref: 2021-305).

  • Google Scholar

• 27

  LiT.WangY.ZhangN. (2020). Combining probability density forecasts for power electrical loads. IEEE Trans. Smart Grid11 (2), 1679–1690. 10.1109/tsg.2019.2942024

  • CrossRef
  • Google Scholar

• 28

  LimitedS. P. P. A. (2016). Offering circular. Singapore.

  • Google Scholar

• 29

  MathesonJ. E.WinklerR. L. (1976). Scoring rules for continuous probability distributions. Manage Sci.22 (10), 1087–1096. 10.1287/mnsc.22.10.1087

  • CrossRef
  • Google Scholar

• 30

  MengA.WangP.ZhaiG.ZengC.ChenS.YangX.et al (2022). Electricity price forecasting with high penetration of renewable energy using attention-based LSTM network trained by crisscross optimization. Energy254, 124212. 10.1016/j.energy.2022.124212

  • CrossRef
  • Google Scholar

• 31

  MonteiroC.Ramirez-RosadoI. J.Fernandez-JimenezL. A. (2018). Probabilistic electricity price forecasting models by aggregation of competitive predictors. Energies (Basel)11 (5), 1074. 10.3390/en11051074

  • CrossRef
  • Google Scholar

• 32

  Nanyang Technological University (2021). High performance computing centre (HPCC). Singapore: Nanyang Technological University. [Online]. Available at: https://ntuhpc.org/.

  • Google Scholar

• 33

  NowotarskiJ.WeronR. (2018). Recent advances in electricity price forecasting: a review of probabilistic forecasting. Renew. Sustain. Energy Rev.81 (2016), 1548–1568. 10.1016/j.rser.2017.05.234

  • CrossRef
  • Google Scholar

• 34

  QinA. K.HuangV. L.SuganthanP. N. (2009). Differential evolution algorithm with strategy adaptation for global numerical optimization. IEEE Trans. Evol. Comput.13 (2), 398–417. 10.1109/tevc.2008.927706

  • CrossRef
  • Google Scholar

• 35

  QinA. K.SuganthanP. N. (2005). “Self-adaptive differential evolution algorithm for numerical optimization,” in 2005 IEEE Congress on Evolutionary Computation Vol. 2, Edinburgh, UK, 2-4 September 2005, 1785–1791.

  • Google Scholar

• 36

  ServicesS. P. (2021). Open electricity market. [Online]. Available at: https://www.openelectricitymarket.sg.

  • Google Scholar

• 37

  TaylorJ. W. (2021). Evaluating quantile-bounded and expectile-bounded interval forecasts. Int. J. Forecast.37 (2), 800–811. 10.1016/j.ijforecast.2020.09.007

  • CrossRef
  • Google Scholar

• 38

  The National Environment Agency (2023). Meteorological service Singapore. [Online]. Available: http://www.weather.gov.sg.

  • Google Scholar

• 39

  UniejewskiB.WeronR. (2021). Regularized quantile regression averaging for probabilistic electricity price forecasting. Energy Econ.95, 105121. 10.1016/j.eneco.2021.105121

  • CrossRef
  • Google Scholar

• 40

  WanC.NiuM.SongY.XuZ. (2017). Pareto optimal prediction intervals of electricity price. IEEE Trans. Power Syst.32 (1), 817–819. 10.1109/tpwrs.2016.2550867

  • CrossRef
  • Google Scholar

• 41

  WanC.XuZ.WangY.DongZ. Y.WongK. P. (2014). A hybrid approach for probabilistic forecasting of electricity price. IEEE Trans. Power Syst.5 (1), 463–470. 10.1109/tsg.2013.2274465

  • CrossRef
  • Google Scholar

• 42

  YangJ.ZhaoJ.WenF.DongZ. (2019). A model of customizing electricity retail prices based on load profile clustering analysis. IEEE Trans. Smart Grid10 (3), 3374–3386. 10.1109/tsg.2018.2825335

  • CrossRef
  • Google Scholar

• 43

  ZhangH.LiuY.YanJ.HanS.LiL.LongQ. (2020). Improved deep mixture density network for regional wind power probabilistic forecasting. IEEE Trans. Power Syst.35 (4), 2549–2560. 10.1109/tpwrs.2020.2971607

  • CrossRef
  • Google Scholar

• 44

  ZhaoJ. H.DongZ. Y.XuZ.WongK. P. (2008). A statistical approach for interval forecasting of the electricity price. IEEE Trans. Power Syst.23 (2), 267–276. 10.1109/tpwrs.2008.919309

  • CrossRef
  • Google Scholar

• 45

  ZhouF.HuangZ.ZhangC. (2022). Carbon price forecasting based on CEEMDAN and LSTM. Appl. Energy311, 118601. 10.1016/j.apenergy.2022.118601

  • CrossRef
  • Google Scholar

• 46

  ZhouQ.TesfatsionL.LiuC. C. (2011). Short-term congestion forecasting in wholesale power markets. IEEE Trans. Power Syst.26 (4), 2185–2196. 10.1109/tpwrs.2011.2123118

  • CrossRef
  • Google Scholar

• 47

  ZhouS.ShuZ.TanK.GooiH. B.ChenS.GaoY. (2016). “Study of market clearing model for Singapore’s wholesale real-time electricity market,” in 2016 IEEE International Conference on Power System Technology (POWERCON 2016), Wollongong, Australia, September 2016 (IEEE).

  • Google Scholar

• 48

  ZimmermanR. D.Murillo-SánchezC. E.ThomasR. J. (2011). MATPOWER: steady-state operations, planning, and analysis tools for power systems research and education. IEEE Trans. Power Syst.26 (1), 12–19. 10.1109/tpwrs.2010.2051168

  • CrossRef
  • Google Scholar