This paper addresses the pressing need for an accurate solar energy prediction model, which is crucial for eficient grid integration. We explore the influence of the Air Quality Index and weather features on solar energy generation, employing advanced Machine Learning and Deep Learning techniques. Our methodology uses time series modeling and makes novel use of power transform normalization and zero-inflated modeling. Various Machine Learning algorithms and Conv2D Long Short-Term Memory model based Deep Learning models are applied to these transformations for precise predictions. Results underscore the efectiveness of our approach, demonstrating enhanced prediction accuracy with Air Quality Index and weather features. We achieved a 0.9691 R2 Score, 0.18 MAE, 0.10 RMSE with Conv2D Long Short-Term Memory model, showcasing the power transform technique's innovation in enhancing time series forecasting for solar energy generation. Such results help our research contribute valuable insights to the synergy between Air Quality Index, weather features, and Deep Learning techniques for solar energy prediction.
4,5
In the modern world, it has become increasingly clear that eliminating fossil fuels is one of the huge
requirements to achieve a carbon-neutral future. The Working Group III Special Report on Renewable
Energy Sources and Climate Change Mitigation (SRREN) [19] suggests that consumption of fossil fuels
accounts for the majority of anthropogenic GHG emissions worldwide. It states that CO2 consumption
had risen to over 390 ppm, which was around 39% above pre-industrial levels by the end of 2010. In
the race to an eficient energy ecosystem, solar energy is a promising renewable resource, but its
intermittent nature poses challenges for integration into the grid. A recent study shows that India lost
29% of its utilizable global horizontal irradiance potential due to air pollution in the period between
2008 and 2018 [
In order to shed some light on the inconsistent patterns of solar generation data, a number of regression models were initially utilised to predict the per-hour generation of solar power. We thus benchmarked a number of regression models, of which the chief ones were Linear Regression, Lasso, Ridge, ElasticNet, and ensemble models like RandomForest and XGBoost. These models utilize diferent Event, Lucknow, India. ∗Corresponding author.
CEUR
ceur-ws.org
methodologies which were compared to determine the model with optimal performance to provide
valuable insights into the future of solar generation.[
The novel contributions of this work can be summarized as follows:
This study hopes to investigate and optimise the performance of the aforementioned machine learning algorithms and provide a clear and accurate picture of solar generation by considering weather features and AQI data which are theorized to have an impact on the fluctuation of this data. It is envisaged that the methodologies employed in this paper will contribute considerably to painting a clearer picture of the sporadic nature of solar power and its influencing factors. The thorough benchmarking and prediction pipeline provides significant benefits for the eficient and sustainable utilization of solar resources by stakeholders, contributing to the adoption and use of practices that ultimately power our society into a clean energy future.
Various studies have used machine learning algorithms to increase the understanding and
improvement of solar power forecasting models. Chuluunsaikhan et al. [
Zhou et al. [
Additionally, Jia et al. [
Along with machine learning models, there were a lot of studies that suggested the use of deep
learning methods for predicting solar power generation. Application of models like CNN’s and Recurrent
Neural Network (RNN)’s exhibits the efectiveness of these deep learning techniques in capturing
complex patterns and dependencies in solar generation signified by Lee et al. [
The Zero-Inflated Poisson (ZIP) model is employed to capture the possibility of extra observed
zeros relative to the Poisson distribution, a common feature in count data. Using these insights we also
utilized zero-inflated models in our research. Yeom et al. [
To summarize, the reviewed papers have considerably contributed to solar power generation using machine learning and deep learning techniques. Their research provided observations that helped us build our research on and further enhance solar forecasting by utilizing AQI, time series-based data, exploring novel approaches, and other diferent approaches to making solar power forecasting more reliable and accurate.
In order to accurately train our model on features that would help it efectively predict solar
generation, we needed a dataset that had high granularity, solar generation, irradiance, and
meteorological data. We thus utilized the UNISOLAR Solar Generation Dataset [
It was noticed that the solar data generated for 15-minute intervals had a number of irregularities in the number of data collections per day. Switching to one-hour intervals eliminated this problem and made the data intervals regular. Another reason for the use of 1 hour intervals was to reduce the noise caused due to various external factors like equipment sensitivity, cloud cover and other atmospheric factors.
For our study, we implemented 70:30 dataset split for training and testing respectively. This approach with 70% of the dataset dedicated to training, allows our model to efectively learn the complex temporal dependencies and relationships inherent in solar data, thereby enhancing its predicting accuracy. The remaining 30% served as an independent testing set to determine the eficacy of our proposed model on unseen data.
(a) (b)
Initially, a regression-based approach was utilized to predict the solar power generation based on the factors present. However, this did not provide adequate information regarding the relationship between these factors and solar power generation.
This prompted us to try out a time series-based approach as we also had chronological data. This is because a time series model is designed to capture patterns in sequential and discrete data points over a set period in time. We made use of solar generation at a particular moment in time to predict the generation at some time in the future as this type of approach would help us identify trends and seasonality to make more accurate predictions. We decided to shape our prediction by creating a column that would predict solar generation values 24, 48 and 72 hours out and compared the efectiveness of these models.
The Machine Learning models used for generating and comparing solar power generation for a Time Series approach were: • Linear Regression
It is a statistical model used to establish a linear relationship between a dependent variable and one or more independent variables. It allows for the prediction of coeficients that decide the strength of the established linear relationship. It is used to predict the line of best fit which minimizes the distance between actual values and predicted values.
= 0 + 1 1 + 2 2 + … + + (1) • Gradient Boosting Regression
It is an ensemble model which combines multiple weak prediction models like decision trees, and takes the strongest combination of predictions to build a strong predictive model. Gradient ̂ = ∑ ( ) = 0( ) +∑ ℎ ( ) • Random Forest Regression
It is a supervised learning algorithm used for regression-based tasks. It combines concepts of decision trees and ensemble learning to make precise predictions. It also reduces overfitting due to an element of randomness which restricts individual trees from memorizing the training set data provided to it. The decision of the algorithm is made by aggregating the decisions of individual trees using majority voting.
Boosting works by repeatedly training weak models to fit the negative gradient of the ongoing prediction’s loss model, which helps improve its accuracy.
=1 =1
=1 ̂ = 1 ∑ ( ) =1 =1 (2) (3) (4) • XGBoost Regression
Extreme Gradient Boosting is an advanced gradient boosting algorithm that is widely used due to the high accuracy of predictions. It achieves this by iteratively adding weak models to the ensemble and ensuring they fit according to the current prediction. It also uses features like column and row subsampling to further improve its performance.
̂ = ∑ ( ) = 0( ) +∑ ℎ ( ) • Random Forest Regression + XGBoost Regression
Building upon the positive results displayed by Lokesh et al [20], we decided upon an ensemble model consisting of a Random Forest Regressor and a XGBoost Regressor, considering that their strengths are complementary in the sense that both are ensemble learners and use boosting and bagging respectively. RFR can handle non-linear relationships between data points and XGBoost can capture subtle patterns and temporal dependencies. • ConvLSTM2D
In this subsection, we delve into the specifics of the ConvLSTM2D model, a hybrid architecture that merges CNNs and LSTM networks. This unique architecture is designed specifically for analyzing spatio-temporal data, where understanding both spatial relationships and temporal dynamics is crucial for accurate predictions.
The architecture of the ConvLSTM2D model, illustrated in Figure 2, consists of layers that facilitate the processing of spatio-temporal data. At its core are ConvLSTM units, which extend the traditional LSTM cells by incorporating convolutional operations within the recurrent structure. This innovative design enables the model to efectively capture both spatial and temporal dependencies within the data. The utilization of ConvLSTM units makes the model adept at handling sequences of spatio-temporal observations, a characteristic essential for tasks such as video analysis, weather forecasting, and motion prediction.
The training procedure of the ConvLSTM2D model follows a systematic approach tailored to exploit its full potential. Initially, standard data preprocessing techniques, similar to those employed for other machine learning models, are applied. However, due to the unique architecture of ConvLSTM2D, additional reshaping of the data is necessary to conform to its input shape requirements. The input data is reshaped into a 5-dimensional tensor, accommodating batches of sequences, each comprising 2D matrices over time. This reshaping operation enables the model to interpret the input data as a spatio-temporal sequence, facilitating efective learning of complex patterns. Furthermore, to ensure robust training and prevent issues stemming from skewed distributions, a power transformation is applied to scale input features appropriately.
This normalization step fosters homogeneous feature scales, thereby preventing any individual feature from disproportionately influencing the learning process.
During the training phase, the model is optimized using the Adam optimization algorithm, with a predefined learning rate. The optimization process involves iteratively adjusting the model’s internal parameters to minimize the mean squared error loss between predicted and actual values. This iterative optimization enables the model to discern intricate patterns within the data and refine its predictive capabilities accordingly.
The ConvLSTM2D model was chosen for its inherent capability to capture spatio-temporal dependencies efectively, a critical requirement in our analysis. Through rigorous experimentation and evaluation, it consistently outperformed alternative models, exhibiting superior performance in terms of predictive accuracy and generalization capabilities. Its hybrid architecture, which integrates both convolutional and recurrent operations, enables it to discern complex patterns within spatio-temporal data, making it particularly well-suited for the tasks at hand.
An analysis of the solar power data showed that the distribution was such that an unusually high number of values amounted to zero, this was likely due to negligible solar generation due to the intermittent nature of sunlight received by the solar panels. These zeros overshadowed the other values in the distribution by a fair amount.
The histogram shown in Figure 3 highlights this zero-inflated data of our initial dataset. It also provides insights into the range of values of solar energy generation and their respective frequencies.
Thus, on noticing this skewness of the solar energy generation data, we decided to switch to a
zero-inflated model which essentially diferentiates between structural zeros (due to nighttime and
genuine absences of solar generation) and zero inflation (additional factors that may influence the
probability of observing a zero, such as cloudy days or equipment failures). According to a study,
previous research indicates that if excessive zero is not accounted for, an unreasonable fit for both the
the zero-inflated nature of our dataset.
zeros and nonzero counts will occur (Perumean-Chaney et al. 2013). ZI (Lambert 1992) and hurdle
models (Mullahy 1986; Heilbron 1994) have been developed to model zero-inflation when the regular
count models such as Poisson or negative binomial are unrealistic [
Tweedie Distribution The Tweedie distribution(Figure 4) is characterized by the following components: • Random Variable: Let be the random variable representing the observed value. • Mean Parameter: is the mean parameter of the distribution, indicating the average value of . • Power Parameter: is the power parameter of the distribution, controlling the variance structure.
It determines the shape of the distribution and can take any positive value, excluding 1. • Normalizing Constant: (, )
is the normalizing constant that ensures the Probability Density Function (PDF) integrates to 1 over the support of . It accounts for the specific value of and and is essential for properly defining the distribution.
The probability density function of the Tweedie distribution is thus given by: ( ; , ) = −1 exp ((1−) 1−
) (, ) exp (−
)
(, )
The Tweedie distribution encompasses various shapes, ranging from heavy-tailed with excess zeros (for < 1 ) to symmetric and Gaussian-like (for 1 < < 2 and > 2 ). It includes special cases such as the Poisson distribution (when = 1 ) and the gamma distribution (when = 2 ). The choice of determines the specific characteristics of the Tweedie distribution, including its skewness, tail behavior, and overall shape. This was suitable for our zero-inflated distribution, by using a modified version of the above called a Zero Inflated Tweedie (ZIT) Model which accounts for the excess zeros using a separate inflation component. with probability with probability 1 − log (
) = 0 + 1 1 + 2 2 + … + log() = 0 + 1 1 + 2 2 + … +
) = 0 + 1 1 + 2 2 + … + where: - represents the response variable (count variable with excess zeros). - represents the positive count variable. - represents the probability of excess zeros. - represents the mean parameter. - represents the dispersion parameter. - 1, 2, … , represent the predictor variables for the mean equation. - 1, 2, … , represent the predictor variables for the dispersion equation. - 1, 2, … , represent the predictor variables for the zero-inflation equation. - 0, 1, 2, … , represent the coeficients for the mean equation. - 0, 1, 2, … , represent the coeficients for the dispersion equation. - 0, 1, 2, … , represent the coeficients for the zero-inflation equation.
Now in order to accurately and eficiently apply a customized zero-inflated model to our data and test out various regression models in Python, we utilized H2O, a Java-based software for data modeling and general computing.
H2O is an abstracted distributed processing engine that allows for simple horizontal scaling in order to provide solutions faster and more eficiently. When it comes to model application, it consists of numerous estimator functions like H2OGradientBoostingEstimator and H2ODeepLearningEstimator served using REST API abstraction, each of which consists of a plethora of hyperparameters for deep customization. For the scope of our project, we have chosen H2OGradientBoostingEstimator, H2OXGBoostEstimator, H2ODeepLearningEstimator, and H2ORandomForestEstimator. The value of the parameters distribution and tweedie_power were set to ‘tweedie’ and ‘1.5’ respectively for the models and other parameters were optimized using GridSearchCV. For the Deep Learning Zero-inflated Model (ZIM), 4 hidden layers were chosen of neuron count 100, 100, 50 and 50 respectively. Transforming our data using a zero-inflated model resulted in a marked improvement in our solar power prediction with a reduced MAE and RMSE.
It is a data transformation and scaling technique which is another way to tackle the skewness of the solar generation data; we found that using PowerTransformer was a significantly better fit for our dataset than the zero-inflated model. This scaling method is applied feature-wise to make the data more Gaussian or Gaussian-like which is inherently assumed by regression-based prediction models. It is used when dealing with non-constant variance. There are 2 diferent methods of performing the power transform, namely the Box-Cox transform and the Yeo-Johnson transform.
The Yeo-Johnson power transform is given by the formula: = ⎧(( + 1) − 1) /, if ≠ 0, ≥ 0 ⎪ln( + 1), if = 0, ≥ 0 ⎨− ((|| + 1) 2− − 1) /(2 − ), if ≠ 2, < 0 ⎪ ⎩− ln(|| + 1), if = 2, < 0
Here, represents the original variable, and is a parameter that determines the type of power transform applied. The transformed variable is the result of applying the Yeo-Johnson power transform. The aforementioned Yeo-Johnson power transform was thus applied to the data, and the resulting distribution was notably found to resemble a Tweedie distribution.
The solar energy generation data points are now normalized to make the distribution more Gaussian as shown in Figure 5.
For the prediction of solar energy generation using multiple methodologies, we have found that the Power Transformed data led to the most accurate prediction in comparison to Regular Time Series and Zero-Inflated models.
Power Transformation of data is a particular method that stands out in comparison to the rest. This is due to solar energy generation being dependent on various factors like temperature, seasonality, time of day, and air quality of the region. These data points have non-linear relationships with each other. The improvement compared to the prior models can be attributed to the power transform method’s ability to make the data more Gaussian like. This addresses the inherent skewness present in solar data, leading to an increase in performance across the board.
ConvLSTM2D models outperform normal regression models as it combines convolutional operations and LSTM memory cells, allowing for the modeling of both spatial and temporal dependencies in data. It also helps capture long term dependencies eficiently in the time series data This makes ConvLSTM2D well-suited for tasks where both spatial and sequential information are important, such as solar power generation forecasting. Figure 7 shows the Loss vs Epoch plot for the ConvLSTM2D model displaying the training progress.
For evaluating the performance of the models on the gathered data, various statistical metrics were
considered, of which ultimately R2, MAE, and RMSE were used. Our key metric was R2, as the coeficient
of determination R2 is generally a better indicator of regression model performance when compared to
other metrics[
The range of target variables was between 0 and 320 for regular time series and Zero-Inflated models. For the Zero Inflated Model, we performed five-fold cross-validation, and there was no significant variance in the validation accuracy from the training accuracy.
The range of target variables was between -0.9 and 1.8 after Power Transform was applied.
After comparing all three methodologies, regression models, and variation in prediction time, the best combination of factors respectively has been tabulated in Table V.
The solar power generation data when plotted monthly follows a specific pattern that can be attributed to the seasonal cycle of the Australian landmass, where the dataset was sourced from. The generation is noted to be maximum from November to February which coincides with the summer months in Australia and reaches its minimum during the months of May to August which are the winter months. This can be explained due to the sun being directly overhead during summer, leading to longer days and more exposure to solar radiation. This phenomenon reverses during the winter months as is shown by the histogram (Fig. 8).
Our time-series based solar power prediction models also capture this phenomenon as seen in the graphs plotted as shown in Fig. 9 and Fig. 10. There is a drop in solar production during the winter months of 2020 and the peak production is reached in the summer of 2021.
This study investigates the use of various machine learning algorithms to efectively determine the future solar generation in a region by utilizing a time series approach. The chief models employed were Linear Regression, Lasso, Ridge, ElasticNet, ensemble models like RandomForest and XGBoost, and deep learning models like ConvLSTM2D. Later, on conferring, we decided to switch to a time-series based approach due to the seasonal fluctuation in the solar data. However, on noticing the skewness of the solar energy generation data which had a high number of zeros, we decided to switch to a zero-inflated model which helps ascertain the diference between the true zero data points and the inflated zeros. This approach immediately yielded a higher accuracy of solar prediction with a lower mean standard error. Another way to tackle the skewness of the solar generation data was to try diferent scaling techniques; we found that using PowerTransformer was a significantly better fit for our dataset than the zero-inflated model. This scaling method is applied feature-wise to make the data more Gaussian or Gaussian-like which is inherently assumed by regression-based prediction models. In conclusion, this study investigates the use of machine learning algorithms incorporating AQI and climate factors to provide more accurate solar generation forecasts. Considering seasonal variations in the solar data, the time series-based method was modified. In addition, a zero-inflated model and scaling techniques were used to address the skewness of the solar generation data. The findings provide valuable insights for solar stakeholders, contributing to the adoption and use of sustainable energy sources.
Solar energy generation forecasting is a dynamic field that will always develop and demand exploration. There is a lot of scope in the future to improve the accuracy of solar forecasting. To make this forecasting scalable and generalized for all geographical regions, the dataset can be collected from diferent regions and time periods since our research primarily utilizes data from a specific region and time zone. Additionally, data fusion and feature engineering can be done to enhance the power of forecasting. Data fusion is the practice of collecting data from multiple sources like satellite imagery and weather stations which can be combined together for better results. There was a limitation on the dataset which we faced during implementation which was the fact that the AQI data we used wasn’t from a data station at the exact geographical location of the data collection but rather another data station away from the solar site. Thus in order to increase accuracy one can take AQI data from a data station at maximum proximity to the data collection center in order to eliminate susceptibility to geographical errors. Another possibility can be to collect our own data to create a localized dataset which would likely be more accurate since we can customize it and remove any bottlenecks faced earlier by having utilised an external dataset. Moreover, feature engineering can be employed to extract more meaningful features from the data available, hence increasing the model performance. Features like cloud cover, dust, and further manners of seasonality can also be taken into consideration for future research.
The authors have not employed any Generative AI tools. 2021 Jul 5;7:e623. doi: 10.7717/peerj-cs.623. PMID: 34307865; PMCID: PMC8279135. [19] O. Edenhofer et al., Eds., Renewable Energy Sources and Climate Change Mitigation: Special Report of the Intergovernmental Panel on Climate Change. Cambridge: Cambridge University Press, 2011.