Solar Irradiance Nowcasting for Virtual Power Plants Using Multimodal Long Short-Term Memory Networks

The rapid penetration of photovoltaic generation reduces power grid inertia and increases the need for intelligent energy resources that can cope in real time with the imbalance between power generation and consumption. Virtual power plants are a technology for coordinating such resources and monetizing them, for example on electricity markets with real-time pricing or on frequency reserves markets. Accurate short-term photovoltaic generation forecasts are essential for such virtual power plants. Although signi ﬁ cant research has been done on medium-and long-term photovoltaic generation forecasting, the short-term forecasting problem requires special attention to sudden ﬂ uctuations due to the high variability of cloud cover and related weather events. Solar irradiance nowcasting aims to resolve this variability by providing reliable short-term forecasts of the expected power generation capacity. Sky images captured in proximity to the photovoltaic panels are used to determine cloud behavior and solar intensity. This is a computationally challenging task for conventional computer vision techniques and only a handful of Arti ﬁ cial Intelligence (AI) methods have been proposed. In this paper, a novel multimodal approach is proposed based on two Long Short-Term Memory Networks (LSTM) that receives a temporal image modality of a stream of sky images, a temporal numerical modality of a time-series of past solar irradiance readings and cloud cover readings as inputs for irradiance nowcasting. The proposed nowcasting pipeline consists of a preprocessing module and an irradiance augmentation module that implements methods for cloud detection, Sun localization and mask generation. The complete approach was empirically evaluated on a real-world solar irradiance case study across the four seasons of the northern hemisphere, resulting in a mean improvement of 39% for multimodality.


INTRODUCTION
VPP (Virtual Power Plants) are emerging as a key solution for power grid decarbonization and the integration of photovoltaic and wind power generation.Yet the inclusion of these renewable generation resources to VPPs remains limited, partly due to the uncertainties involved in their production (Liu et al., 2021).Mahmud et al. (2020) emphasize the need for VPPs to have a capability to cope with sudden weather changes impacting PV production as well as other energy consuming assets under the same VPP.Yu et al. (2019) review the major categories of uncertainty that a VPP needs to manage.They identify market prices and consumption demand as uncertainties that are interdependent with renewable generation uncertainty.A common way to manage the uncertainty of photovoltaic generation is to invest in significant battery storage capacity Liu et al. (2019), but such resource intensive investments could be reduced by better predictions of the photovoltaic generation.A VPP must have a capacity to operate in realtime and to adjust plans that were made earlier based on forecasts of PV generation and other assets (Fan et al., 2020).The adjustments can be made either to ensure compliance with bids that were made ahead of time, or to allow the VPP to perform a second round of trading on real-time markets (Qiu et al., 2017).In either case, an accurate short-term photovoltaic generation forecast is needed to support the real-time operations of a VPP.
Generation and consumption in a power grid must be balanced at all times in order to maintain the grid frequency close to a rated frequency.Excess generation causes overfrequency and excess consumption causes under-frequency.Frequency reserves are services purchased by the grid operator to react to such situations.Providers of reserve resources commit to increasing production or decreasing consumption in underfrequency situations, and to decrease production or increase consumption in over-frequency situations.An example of photovoltaic participation in frequency reserves is curtailing the production in over-frequency situations.Subramanya et al. (2021) report of such a commercially operational installation, managed by a VPP and participating in Finnish frequency reserve markets.Thus, frequency reserves are another category of applications benefitting from photovoltaic generation forecasts.Subramanya et al. address the problem with day-ahead forecasts for the purpose of bidding on the day-ahead frequency reserve markets.However, there is a need for further work on more accurate real-time forecasts that allow the VPP to address actual frequency deviations as they occur.Such forecasts would be applicable to work on dynamic planning of a VPP participating on frequency reserves (Javadi et al., 2019).
This paper is motivated by the need for development of realtime capabilities to VPPs, for example to participate on real-time electricity markets, to react to frequency deviations when participating on frequency reserves markets, or simply to cope with commitments that were made ahead of time based on uncertain photovoltaic generation predictions.A capability for real-time forecasting of photovoltaic generation, "nowcasting," is developed for these purposes.Artificial Intelligence (AI) techniques such as classic neural networks Al-Dahidi et al. (2019) and more recently Long Short-Term Memory Networks (LSTM) Li et al. (2020), Stüber et al. (2021) have performed well for medium-term solar forecasting.These applications of LSTM are based on time-series data, but the short-term forecasting context could significantly benefit from the additional consideration sky image data.The processing of diverse kinds of inputs by a machine learning model is known as the multimodal approach Baltrusaitis et al. (2019), which is applied in this paper to consider time-series data alongside sky image data.In this paper, following research contributions are reported to advance the state-of-the-art on photovoltaic nowcasting: 1 An AI pipeline for nowcasting that receives a temporal image modality of a stream of sky images, a temporal numerical modality of a time-series of past solar irradiance readings and cloud cover readings as inputs, followed by a data preprocessing module and an irradiance augmentation module leading up to the irradiance nowcasting module. 2 A novel method for cloud detection, Sun localization and mask generation in the irradiance augmentation module that computes the impact of cloud cover on irradiance intensity.3 A multimodal approach based on two LSTMs for the image and time-series modalities in the irradiance nowcasting module that can deliver real-time capabilities to VPP. 4 Finally, the complete pipeline study is evaluated across the four seasons of the northern hemisphere, using a real-world solar irradiance case study conducted by the NREL SRRL labs.

Nowcasting for VPP With Photovoltaic Generation
A resource intensive way to manage photovoltaic integration is to invest in large-scale battery resources to complement the photovoltaic generation capacity, and one line of research is to investigate the optimal sizing of battery storages for this purpose e.g., (Kong et al., 2019;Kosmadakis et al., 2021;Muñoz-Rodríguez et al., 2021).A more intelligent and less resource intensive approach is to employ a VPP Thavlov and Bindner (2015) to coordinate diverse energy resources such as photovoltaic generation, smart loads and electric vehicles.A typical planning horizon for a VPP is day-ahead, in which case the VPP requires forecasts of its photovoltaic generation as well as all the resources under its control for this time horizon (Xing et al., 2021).In this context, real-time pricing usually means hourly changing prices that are known day ahead Nge et al. (2019), but other shorter-term real-time pricing approaches exist and shall be introduced below.A minority of research considers a shorter timeframe in which the photovoltaic generation nowcasting could be exploited.Recognizing the inevitable errors in predicting wind power generation day ahead, Hu et al. (2021) propose a 2-stage dispatch, day-ahead and realtime, to exploit the more accurate short-term forecast; a similar approach is applicable to photovoltaic dispatch if short-term solar irradiation forecasts are available.Das et al. (2021) propose to optimize photovoltaic generation and battery storage use in the context of hourly changing electricity prices.Elkazaz et al. (2020) consider a household with local photovoltaic generation and battery storage, and propose a scheme consisting of day-ahead optimization as well as real-time control to manage the errors caused to the optimization by uncertain forecasts.Elkazaz et al. (2020), Das et al. (2021), Hu et al. (2021) are all examples of methods that use real-time measurements as the input to an algorithm that handles the error of medium-term renewable generation forecasts, so an obvious line of further research would be to exploit nowcasting.Nefedov et al. (2018) exploit electric vehicle batteries for short-term storage of surplus photovoltaic energy, updating plans as vehicles arrive or if they leave unexpectedly.Wu et al. (2020) and Seyyedeh Barhagh et al. (2020), present similar approaches with the additional capability to consider dynamic electricity prices.Kolodziejczyk et al. (2021) operate a system consisting of photovoltaic generation and battery storage on a real-time electricity market with hourly changing prices.An example of a close to real-time electricity market is Singapore's half-hourly spot market (Zhang et al., 2021); at this time-scale, photovoltaic generation nowcasting exploiting sky image analysis begins to be advantageous.A few works address a very short-term scenario, in which the photovoltaic generation nowcasting approach is most beneficial.In particular, Larsen et al. (2017) consider a virtual power plant with smart loads participating on a real-time market with prices changing at 5 min intervals.
Photovoltaic generation is at the same time a problem and a potential solution for power grid frequency reserves.In fact, increased photovoltaic penetration without VPPs is a part of the problem, as it decreases the grid inertia Su et al. (2020), Zhang et al. (2020) and increases the demand for frequency reserves (Karbouj et al., 2019).This increased demand is specifically due to the short-term variability of photovoltaic generation (Sinsel et al., 2020).A straightforward approach is to invest in battery storage that is dedicated for frequency reserves (Koller et al., 2015).However, a VPP can be more economical for managing uncertainties and optimizing a set of diverse resources with different characteristics.Forecasting the grid frequency involves considerable uncertainty which is compounded by other uncertainties such as photovoltaic generation Mashlakov et al. (Mashlakov et al., 2021).As the future grid frequency deviations are not known even in the short term, it is the frequency reserve resource provider's duty to ensure that the reserve resource capacity is available and ready to be activated in the event of a frequency disturbance (Subramanya et al., 2021).Several innovative solutions have been proposed for this purpose, exploiting the specific characteristics of the intelligent energy resources that are being exploited as frequency reserve resources.Angenendt et al. (Angenendt et al., 2020) identify an alternative way to exploit batteries for supporting local photovoltaic generation.Such batteries could be traded on frequency reserves in the low photovoltaic generation season.Other authors simply rely on smart loads to cover the demand for frequency reserves caused by large photovoltaic penetration (Conte et al., 2021).Bagheri-Sanjareh and Nazari (2020) present a comprehensive approach for coordinating photovoltaics with other smart energy resources for the purpose of providing frequency reserves.In the case of these and other systems for providing frequency reserves by coordinating intelligent energy resources, whenever photovoltaic generation is part of the mix, the nowcasting of such generation can support the planning of the coordination on a short time horizon, e.g., 5, 10 or 15 min.This is especially beneficial for reserve resources that need several minutes to be ramped up.Finally, it is noted that frequency reserve markets are traditionally operated by transmission system operators, but the emergence of photovoltaic microgrids is leading to the emergency of real-time frequency response being managed at the level of the microgrid Shi et al. (2015) or cluster of collaborative microgrids (Qazi et al., 2021).

Related Work in Solar Irradiance Nowcasting
In recent work, several approaches have been proposed for solar irradiance nowcasting, using sky cameras, satellite imagery, ground sensor networks, and time-series of historical PV generation.
Chen et al. ( 2019) proposed a spatio-temporal PV nowcasting method using a sensor network with predictor preselection.It enabled preselection in different cloud movement scenarios; stationary, and ramp, and interpolate cloud information to provide consistent PV nowcasts.Zheng et al. (2020) proposed a method to predict the output of solar power plants based on time series forecasting, primarily focusing on multiple regions.They used a Long Short-Term Memory Network (LSTM) model for the forecasting and multiple LSTM structures were compared to determine the final prediction model with sensitivity analysis.Additionally, particle swarm optimization was employed to improve the prediction by optimizing the parameters used for the LSTM   2019) have reported a hybrid forecasting method to forecast 1-min averaged solar irradiance using all-sky images and past irradiance measurements.Detection of clouds and cloud motion estimation were done using pre-existing methods with the addition of a post-processing procedure.After that, the predicted cloud information was used to infer the solar irradiance by forecasting the clear-sky index using the proposed method that uses a simple deterministic model.Zhang et al. (2018) proposes a deep photovoltaic nowcasting approach using sky images and historical PV power values.Several CNN based models were trained to learn the relationship between sky appearance and future PV power output.Feng and Zhang (2020) proposes a CNN based approach for intra-hour solar forecasting exclusively using sky images.This contains a set of models that generate fixed-step global horizontal irradiance (GHI) in parallel.
Infrared (IR) technology has also been applied in related work on sky imaging based solar irradiance forecasting.Mammoli et al. (2019) proposed short-term solar forecasting by merging far-field images from multiple longwave infrared sensors.Terrén-Serrano and Martínez-Ramón (2021) presented a short-term solar forecasting approach based on processing of global solar irradiance and ground-based infrared sky images.Both approaches focus on the cloud behavior and the latter also categorized the sky condition of an infrared image into four main cloud formations, clear-sky, cumulus, stratus, and nimbus.Apart from forecasting, infrared is also used in solar position identification which is an important aspect that even helps with forecasting.Nespoli and Niccolai (2020) proposed an approach for solar position identification using infrared thermal allsky images of which the sunspot is defined clearer than color images.However, to capture infrared images a special camera system is specified.
As delineated above, several statistical, machine learning, deep learning, infrared-based and hybrid methods have been proposed in recent literature to address the technical challenges of solar irradiance nowcasting.However, none of these approaches consider the multimodality of the problem domain in terms of the need for development of real-time capabilities for VPP.The significance of cloud detection, Sun localization and mask generation in computing the impact of cloud cover on irradiance intensity as a necessary augmentation phase prior to the consideration of its multimodality is a further limitation of existing work.Drawing on this context, a new approach that delivers an AI pipeline consisting of preprocessing, augmentation, and a multimodal deep learning model, is presented in the following section.

PROPOSED METHODOLOGY
The proposed approach for solar irradiance nowcasting is illustrated in Figure 1.Given the multimodal nature of this approach, it receives two input modalities, a temporal image modality of a stream of sky images, a temporal numerical modality that consists of a time-series of past solar irradiance readings and cloud cover readings.Streams from both modalities are input to a data preprocessing module, which conducts missing data handling and redundancy reduction for images and temporal sequence alignment, normalization, and range correction for the solar irradiance readings.The preprocessed sky image stream is funneled through an irradiance augmentation module, which further refines the irradiance information using cloud detection, Sun localization and mask generation techniques.The irradiance augmentation module receives a further time-series of solar angle readings.This augmented sky image stream, the preprocessed irradiance readings and cloud cover data converge in the final phase for irradiance nowcasting.The irradiance nowcasting module is composed of two LSTM Networks, irradiance readings and cloud cover data LSTM (irLSTM) and sky image LSTM (siLSTM).The irLSTM was designed with four 2-dimensional layers of 128, 128, 64, 64 nodes and three fully connected layers of 64, 32, 16 nodes.The siLSTM was designed with five 2-dimensional Conv-LSTM layers of 256, 256, 128, 64, 32 nodes that generates a flattened 1-dimensional output which is then integrated with a repeat vector of 32 nodes, followed by a time distributed dense layer of 32 nodes that is again flattened prior to final fully connected layers of 64, 32 and 16 nodes.This deep learning architecture was formulated using a grid optimization process, which is considered as the state of the art in global optimization (Bergstra et al., 2012).Each module is further elaborated in the following subsections.

Data Preprocessing Module
Scaling and normalization are the initial preprocessing steps conducted to ensure that variations in magnitude, range and units of measurement do not have a negative impact on the  model architecture or the learning process of the proposed multimodal approach based on deep learning in LSTM networks.For example, input features of high magnitude are allocated with large weights resulting in unstable models during the learning and testing phases.Temporal sequence alignment, normalization and range correction techniques were used to address these variations in numerical timeseries data streams of solar irradiance and cloud cover readings.In the nowcasting prediction module, the final output was reverted to the original scale and magnitude for evaluation and application.
The sky image data stream was initially checked for missing data due to maintenance and technical faults.The missing days are removed entirely to maintain the accuracy of the model.Then, the raw image data are saved in JPEG file format with the size of 288 × 352.A raw image consists of the sky region and other segments of the surroundings or the machinery involved.In redundancy reduction, all images were cropped to 270 × 270 (as shown in Figure 2) to focus on the area of interest, which is the sky region.
A correlation analysis was performed using the Pearson correlation coefficient to determine features of importance from the time series of irradiance readings.Hour of the day, total cloud cover, and opaque cloud cover were identified as the most correlated to the actual solar irradiance reading.Figure 3A illustrates this correlation, here it can be observed that the presence of clouds leads to a decrease in the level of irradiance.
Figure 3B depicts solar irradiance in the Northern hemisphere across the 12 months of a calendar year.During winter months, irradiance is low and during summer, irradiance reaches the peak.This seasonal fluctuation impacts the predictive accuracy of an irradiance nowcasting approach.To overcome this, the dataset was divided into four seasons and trained a separate model for each season.

Irradiance Augmentation Module
Following the preprocessing phase, the irradiance augmentation phase further refines the irradiance information using cloud detection, Sun localization and mask generation techniques.Each technique is described below.

Cloud Detection
Red Blue Ratio (RBR) method was utilized for cloud detection (Li et al., 2019).First, the images were separated into the red, blue, and green channels.Then, the pixel-wise ratio of the red and the blue channels were calculated which reduces the 3-channel image to a 2-D array with ratio values.After this, commonly a single threshold is used for cloud detection.However, in this approach three thresholds were used where each threshold detects clouds with different intensities, to increase the cloud details that can be extracted from the image.As shown in Figure 3, when a lower threshold is used, all the clouds with lower intensity are detected and when a higher threshold is used, only the clouds with higher intensity are detected.In the final image, all the outputs which were acquired using different thresholds were concatenated giving each layer a value depending on the threshold where the layer acquired from the highest threshold gets 255. Figure 4 illustrates the input and the outputs of each stage of the cloud detection technique.

Sun Localization and Mask Generation
A new algorithmic method is proposed for Sun localization using the zenith angle, the azimuth angle and their corresponding astronomical details.Here the zenith angle is considered as the angle between the direction of the Sun and the zenith (directly above).Also, the azimuth angle is defined as the angle between the horizontal direction of the Sun and the reference (usually north but sometimes taken as south).Figure 5 illustrates the zenith angle and the azimuth angle.The coordinates of the Sun on the image (x, y) are calculated using Eq1, 2 and 3, l S p sin (α) (1) Here, the R is the radius of the hemisphere that the image has covered.Next, the Sun localization is used for solar mask generation, which is used to make the region closer to the Sun more effective since the clouds close to the Sun has the most impact when it comes to nowcasting.The solar mask is a gradient from 1 to 0, with 1 being directly where the Sun is located and then gradually decreasing to zero across the image.Finally, element-wise multiplication is used to apply the mask to the processed image.Figure 6 illustrates the estimation of the solar location (B) on samples representing different times of the day and the year (A) and the corresponding solar mask (C). Figure 7 illustrates the process in terms of outputs at each stage, and how the solar mask is merged with the preprocessed image.

Irradiance Nowcasting Module
As described earlier, the irradiance nowcasting module is based on LSTM networks.LSTM is a type of deep Recurrent Neural Network (RNN).RNNs are designed for sequential data with internal memory states and feedback connections.An LSTM unit is illustrated in Figure 8.It mainly consists of four parts, namely, cell, input gate, output gate and forget gate.The cell stores the feedback values while the gates control the information flow to the cell.Each of these cells can be considered as neurons of a standard feedforward neural network where the activation of a weighted sum is calculated.In that sense, if the activations for the input, output and forget gates are considered as i t , o t , and f t , respectively, there are calculated using Eq 4, 5 and 6.W represents the input weights and U represents the recurrent connection weights of the cell or the three gates.The cell output c t and the upward output h t is calculated using Eq. 7, 8 and 9, Here, σ represents the sigmoid function and in both Figure 8 and the equations, ⊗ denotes the matrix multiplication.Furthermore, h t goes through an activation function, before it is considered as the output of the unit.
The proposed irradiance nowcasting module is composed of multimodal LSTM networks Hochreiter and Schmidhuber (1997), irradiance readings LSTM (irLSTM) and sky image LSTM (siLSTM).Although LSTM has been used in solar forecasting Zhang et al. (2018), Kong et al. (2020), Zheng et al. (2020), the multimodal LSTM approach and the learning architecture that is proposed in this paper are novel contributions.Conv-LSTM is a variant of the LSTM network which is specifically designed to learn the spatial information of data.Conv-LSTM is designed for multi-dimensional data in which the all-sky solar image dataset can be learned.The matrix multiplications at each gate in the LSTM are replaced by the convolutional operations in the Conv-LSTM to capture the underlying spatial information in data.
The model architecture of the multimodal LSTM networks used for nowcasting is illustrated in Figure 9.For time series data, four LSTM layers with 128, 128, 64, and 64 units were used and three dense layers with 64, 32, and 16 units were integrated at the end.For image data, five 2D Conv-LSTM layers with 256, 256, 128, 64, and 32 units were used.Then the output was flattened, and a repeat vector was integrated before the LSTM layer with 32 units.Following a time distributed dense layer with 32 units the output gets flattened again before the final dense layers with 64, 32 and 16 units.This deep learning architecture was formulated using a grid optimization process (Bergstra et al., 2012).Finally, the output is irradiance nowcasting with a prediction horizon of 10 min.

EXPERIMENTS AND RESULTS
The proposed approach was experimented on a solar irradiance case study conducted by the NREL SRRL labs (Stoffel and Andreas, 1981).The solar irradiance data, considered is the global horizontal hemispheric shortwave irradiance, measured    continuous epochs.For the LSTM layers, the sequence size of six steps (1 h) was used.Hence, the rolling window method was used for training as well as the evaluations.The hardware specifications of the experimental setup are composed of eighth Gen.Intel i7 processor, 16 GB random access memory and NVIDIA GeForce GTX 1050 Ti graphics card.
Three evaluation metrics were used to validate all models.They are, Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and normalized RMSE (nRMSE).MAE measures an average model accuracy giving equal weights to the errors that are calculated at each observation.Considering Ai and Fi as the actual and the forecasted values respectively, for n observations, Eq 4 calculates the MAE.

MAE
The RMSE takes the residuals between the forecasted values and the actual value to compare prediction errors, giving higher weights to large errors.Considering A i and F i as the actual and the forecasted values respectively, for n observations, Eq 5 calculates the RMSE.
The nRMSE represents the RMSE to the actual range of the predicted variable.It is used to facilitate the comparison between models developed for data with different scales.Also, it can be interpreted as a fraction of the overall range that is typically resolved by the model.Considering the X bar as the mean of the actual variable, nRMSE is calculated using Eq. 6.

nRMSE RMSE X
Data from 3 years were used for the evaluation, January 2018 to December 2020, following 2-fold cross validation procedure.To evaluate and demonstrate the advantages of using both time series and image data, the irLSTM and siLSTM models were tested separately.The architectures are illustrated in Figure 9 and additionally for each model a dense layer with 1 unit was integrated following the dense layer with 16 units.Also, a univariate LSTM model with the same architecture as of irLSTM was tested as the benchmark model.Table 2 shows the experimental results of the models, separately for "BM," "irLSTM" and "siLSTM," as well as the "multimodal" model.Here "BM" represents the benchmark model.
Based on Table 1, the proposed multimodal model surpasses the benchmark model and the irLSTM model also outperforms the benchmark in terms of the error metrics demonstrating the positive effect of the added time series features.Then, it can be observed that a considerable performance gain could be achieved by using both irLSTM and siLSTM models as multimodal model, with some fluctuations only during the spring season.Especially during the winter and autumn, 14.28 and 10.8% improvements in nRMSE were recorded.The multimodal model for the spring and the summer produced nRMSE of 28.74 and 32.16%, respectively.The model performed better during the winter and the autumn scoring nRMSE values of 21.38 and 21.05%, respectively.
Table 3 presents the model performance of the multimodal model with image data before and after the Sun masking.For each season, the performance of the model was improved with the Sun masking.Especially during spring and summer, the nRMSE of the models were improved by 8.41 and 6.32%, respectively.
Figure 10 illustrates the actual solar irradiance vs the prediction of the benchmark and the multimodal model, for 4 days during winter.As depicted by the "total cloud cover," Day 1 is all-day cloudy with a percentage close to 100% and Day 4 is sunny throughout the day with percentages close to zero.During Day 1, the predictions from the multimodal model manage to get closer to the actual value.During day 4, multimodal model predicts closer to the actual value but a bit lower during the peak.Then, during both day 2 and day 3, cloud cover is recorded a lower value and increases during the latter part of the day, where the irradiance reading is significantly affected.It is observed that model produces lower values at the peaks during this period, while maintaining the prediction closer to the actual value.
Also, the uncertainty of the prediction of the multimodal model is also explored.The method proposed in Gal and Ghahramani (2015) which utilizes dropout for representing the model uncertainty was used.The measurement of uncertainty was defined as the variance of the values that the model predicts with dropout for 50 iterations.A dropout ratio of 0.16 was selected in a way that the percentages of the actual values in each confidence interval are closer to the confidence interval itself.This was carried out to make sure that the model confidence interval is neither overestimated nor underestimated.Nonetheless, the 'variance' in Figure 10 shows that the proposed model has a lower uncertainty during cloudy conditions than during clear sky conditions.

CONCLUSION
In this paper, the challenge of solar irradiance nowcasting in virtual power plants is addressed using a novel approach that leverages the multimodality of irradiance data streams.This approach receives a temporal image modality of a stream of sky images, a temporal numerical modality of a time-series of past solar irradiance readings and cloud cover readings as inputs into an AI pipeline composed of preprocessing and irradiance augmentation modules followed by a multimodal LSTM network architecture that nowcasts solar irradiance within a prediction horizon of 10 minutes.These short burst predictions inform the upstream decision-making of real-time pricing and frequency reserves optimization in VPP.The complete approach was empirically evaluated across the four seasons of the northern hemisphere using a real-world solar irradiance case study.The results confirm that our approach is more effective than single modalities and outperforms current benchmark models.As future work, we intend to formulate and integrate further modalities such as weather, climate and mobility that improve the predictive accuracy of solar irradiance nowcasting.Also, automated machine learning methods offer a promising approach for optimizing the deep neural network architecture, and their application is a topic for further work.One relevant starting point for such work is the extensive discussion of Neural Architecture Search (NAS) methods in the survey (He et al., 2021).
models.Kong et al. (2020) proposed a series of novel deep learning approaches for short-term PV forecasting based on whole-sky-images.They developed both static and dynamic architectures composed of Convolutional Neural Networks (CNN), LSTM, ConvLSTM and PredNet deep learning models.Caldas and Alonso-Suárez (

FIGURE 1 |
FIGURE 1 |The proposed approach for solar irradiance nowcasting.

FIGURE 3 |
FIGURE 3 | (A) Solar irradiance behavior with cloud cover (B) Mean solar irradiance for each month.

FIGURE 4 |
FIGURE 4 | Cloud detection using the Red Blue Ratio method.

FIGURE 7 |
FIGURE 7 | Pictorial representation of the process for merging the sun mask with preprocessed image output.

FIGURE 9 |
FIGURE 9 | Model architecture of the multimodal LSTM networks.

FIGURE 10 |
FIGURE 10 | Actual irradiance, irradiance nowcasting, cloud cover and uncertainty measurement across 4 days during winter.

TABLE 3 |
Comparison of nowcasting results of the multimodal model with before and after the Sun masking.