U-Net and LSTM-based Neural Networks

Abstract

ICYRIME 2023: 8th International Conference of Yearly Reports on explanation of the system we developed is illustrated in Workshop Proceedings

-

Weather Forecasting changes in solar wind properties accurately is crucial for predicting space weather, as it significantly impacts the majority of space operations and the telecommunication system. To meet this challenge, we introduce an architecture that combines U-Net’s capabilities for segmenting coronal holes from high-resolution sun images with the predictive abilities of Long Short-Term Memory (LSTM) and ConvLSTM models. This architecture predicts solar wind density using sun surface images obtained from the AIA 193 Å dataset (provided by NASA) and historical electron and proton density data from the OMNI and ELM2 datasets (also provided by NASA), covering the entire year 2012. Our findings demonstrate the system’s ability to generate reliable coronal hole segmentation maps and achieve good accuracy in forecasting solar wind density.

1. Introduction

The solar wind is a dynamic flow of charged particles in a plasma state, originating from the Sun’s corona. This stream of particles emanated from expansive luminous areas known as coronal holes, overcomes the Sun’s gravitational force thanks to its elevated thermal energy and spreads all over the universe[ 1 ]. Composed primarily of electrons and protons, this solar wind significantly influences the conditions of the entire solar system. While the Earth’s magnetic field shields the majority of this wind, excessive strength can lead to geomagnetic storms that are particularly dangerous, especially for astronauts and spacecraft, and can cause disruptions in power grids, interfere with satellite communications, and even lead to notable incidents such as the 1989 blackout in Quebec caused by a high-velocity solar wind. Other historical around 774-775 AD, underscore the immense impact of solar wind variations. The former disrupted telegraph communications, while the latter, studied through the analysis of Carbon 14 in the polar ice, highly surpassed the intensity of the 1859 storm. We can only imagine the consequences of such magnetic storms in today’s world nEvelop-O (C. Napoli) Section 4. Subsequently, the evaluation of our architecture is presented in Section 5. Finally, we summarized the content of the article and we outlined the possible viable improvements that can be made in Section 6. events, like the solar storm 1859 and the Miyake event in the absence of detailed solar surface features. and the mobility of citizens [ 2 ]. In a society heavily re- the description of the dataset employed in this experiliant on electricity, this could significantly disrupt our CEUR

ceur-ws.org

2. Related Works

Network (CNN) architecture is employed to solve a sim

ple regression problem that given as input a single RGB In the realm of works focusing on segmenting coro- image of the Sun, predicts the corresponding solar wind nal holes, the field of image segmentation has experi- speed that will be registered at the L1 point. Briefly, enced significant advancements in recent years, with a CNN [ 16 ] is a network consisting of layers applying the U-Net model emerging as a powerful, versatile, and convolutional operations to detect patterns, features, or widely adopted architecture used across various domains objects within images. such as medical imaging, remote sensing, and astron- Another notable work was proposed by the authors of omy. Originally developed for tasks like biomedical im- [ 17 ] that presented WindNet, a CNN-LSTM framework age segmentation (e.g., cell and tissue segmentation from that uses a pre-trained GoogleNet architecture [ 18 ] as a microscopy images), the U-Net architecture features a feature extractor. In specifics, a Long Short-Tem Memory classic encoder-decoder structure characterized by its (LSTM) is an extension of a Recurrent Neural Network symmetric U-shape. The encoder component typically (RNN) [ 19 ] intended to capture and combine long- and follows a traditional CNN architecture, incorporating suc- short-term dependencies in data sequences. WindNet cessive convolutional layers and pooling layers to reduce was created to look into the optimal combination of delay spatial dimensions while capturing high-level hierarchi- and history values to predict the solar wind speed. cal features from input images. On the other hand, the In [ 20 ], to predict the solar wind propagation delay decoder utilizes upsampling layers to restore the spatial between the Lagrangian point L1 and the Earth, some resolution of feature maps. This restoration is achieved classic machine learning methods are employed (e.g., Ranthrough skip connections directly linked to the encoder, dom Forest Regression (RF) [ 21 ], Gradient Boosting (GB) enabling precise localization by providing detailed infor- [ 22 ], and Linear Regression (as ordinary least square mation from the original input images. In the realm of regression presented in [ 23 ]) and their performance is solar physics, researchers have explored adaptations of compared to classic physics models such as the flat or the U-Net model for segmenting solar features such as vector delay methods. The GB turned out to be the best sunspots or coronal holes from high-resolution imagery. model. Despite its high accuracy, the evident limitation These adaptations often involve fine-tuning the network of solving this particular problem lies in the small time to address specific challenges presented by solar images, interval to be predicted, a few seconds, which is the time such as the varying intensity and appearance of sunspots necessary for the Solar Wind to travel from the L1 point against a dynamic background. and the Earth.

In [ 9, 10 ], the authors trained a U-Net neural network In ref. [ 24 ], linear prediction functions were used to using daily SDO/AIA 193 Å solar disc images and corre- forecast the solar wind speed at 1 AU up to four days sponding coronal hole segmentation maps from 2010 to in advance by using solar images with a 1-hour time 2017 provided by the Kislovodsk Mountain Astronomi- resolution. In detail, through a thresholding process, cal Station. They evaluated this model using data from the active areas in the central meridional slice of the Sun 2017 to 2018 and compared it with other semi-automatic were extracted, as it is part of the sun that is most directly segmentation procedures. The authors found that U- facing the Earth, and used as input to the empirical modelNet outperformed the other algorithms used for coronal based. hole segmentation, demonstrating higher generalization In [ 25 ], the authors compared the performance of empower and accuracy. pirical, hybrid empirical-physics-based, and fully physics

A similar approach is presented in [ 11 ], where a cus- based coupled corona-heliosphere models over 8 years tomized U-Net architecture named SCSS-NET was devel- of solar wind observations. They found that the empiroped to segment solar corona structures from Sun im- ical baseline schemes produce the “best” predictions of ages. This system was benchmarked against established solar wind parameters in near-Earth space, at least in algorithms such as the Spatial Possibilistic Clustering terms of the Mean Squared Error (MSE). However, even Algorithm (SPoCA) [ 12 ], the Coronal Hole Identifica- if the physics-based approaches still require some further tion via Multi-thermal Emission Recognition Algorithm parameterization, with continued refinement, they can (CHIMERA) [ 13 ], and the Region Growth algorithm [ 14 ]. potentially outperform empirical schemes in terms of The SCSS-NET model demonstrated promising segmen- prediction. This is especially true when it comes to the tation results comparable to these methods. However, its integration of transient structures, the drivers of major performance is contingent upon the accuracy of reference space weather disturbances. annotations.

Regarding the diferent architectures introduced to solve the solar wind properties forecasting, most of them are focused on predicting the solar wind speed. For instance, in [ 15 ], a straightforward Convolutional Neural

3. Dataset

casting solar wind density, we created an additional dataset by combining the coronal hole segmentation We employed the AIA 193 Å dataset from NASA’s Solar maps obtained by our customized U-Net architecture Dynamics Observatory (SDO) space telescope to train with two tabular datasets. The first one is the OMNI the U-Net architecture for coronal hole segmentation. dataset, which is an hourly resolution multi-source The SDO’s Atmospheric Imaging Assembly (AIA) instru- dataset of near-Earth solar wind’s magnetic field and ment captures solar images across various ultraviolet plasma parameters, such as the IMF (magnitude and vecand extreme ultraviolet wavelengths, including the 193 tor), flow velocity (magnitude and vector), flow pressure, Ångström (Å) wavelength, crucial for studying coronal proton density, alpha particle to proton density ratio, structures like coronal holes, coronal loops, and active and more. The second one is the ELM2 (EESA Low Elecregions. tron Moments) dataset, which comes from the WIND

To generate the ground truth coronal hole segmen- 3-D Plasma experiment that makes measurements of the tation maps, we performed manual thresholding on all full 3-D distribution of suprathermal electrons and ions. images from 2011 to 2012, with a temporal resolution of Since all these datasets have coinciding timestamps, we 6 hours, using the following algorithm: first, we down- merged the information available for the year 2012 with scaled the original high-resolution images to 256 × 256 a time interval of 6 hours into a single dataset. It is impixels to balance feature detail and computational load. portant to highlight that while generating this dataset Next, we converted them to grayscale and manually se- we make a strong approximation. Even if the solar wind lected a threshold to enhance coronal hole visibility. Sub- speed is extremely variable, we associated the coronal sequently, we inverted the binary images obtained to hole segmentation maps with the tabular data acquired highlight segmented coronal holes and applied a circular two days later. mask representing the sun’s shape to isolate the seg- The pre-processing of this latter dataset involved difmented coronal holes. The complete procedure for ex- ferent operations. First of all, we removed the constant tracting coronal hole segmentation maps is illustrated in values from the two tabular datasets, since they do not Fig. 1. give valuable information to the training process. SubseTo train the LSTM-based models responsible for fore- quently, we removed the outliers by eliminating values exceeding 5 times the standard deviation for each respective feature. Finally, since we had to deal with diferent physical quantities having diferent orders of magnitude, we normalized the data in the range [ 0,1 ] based on the maximum and minimum values present in the dataset for each feature.

4. Methodology

The full pipeline of our system, illustrated in Fig. 2, consists of two main modules. The segmentation module extracts the binary coronal hole segmentation maps from the history of high-resolution images of the sun. Subsequently, the prediction module forecasts solar wind density by integrating the historical coronal hole segmentation maps with the historical solar density data.

4.1. Segmentation Module

4.2. Prediction Module

The LSTMs and their variants stand out as the most widely utilized architectures for learning from sequential data and forecasting future states. LSTMs excel in analyzing the evolution of coronal holes over time, efectively handling the nonlinear and unpredictable aspects of their movement and morphological changes. Consequently, the prediction module incorporates two networks: a ConvLSTM architecture for analyzing historical binary coronal hole segmentation maps and a standard LSTM architecture for processing historical solar wind density data. We have stacked five layers for both of these networks and included dropout layers (with a dropout rate of 0.1) after each LSTM layer except the last one. To predict electron and proton densities, we flattened the output of the ConvLSTM and concatenated it with the output of the standard LSTM. This combined output is then passed through a linear feed-forward neural network, composed of three linear layers, responsible for the outcome of our system.

5. Results

For validating our segmentation module, we employed two types of error functions: the Intersection over Union (IoU) and the Dice coeficient. The IoU measures the overlap between the predicted segmentation and the ground truth divided by the union: = (1)

+ +

The Dice coeficient, also known as F1 score, is a statistical tool that measures the similarity between two sets of data: = 2 (2)

2 + +

Here, TP (true positive) represents correctly segmented pixels, FP (false positive) denotes predicted object mask pixels not matching the ground truth, and FN (false negative) indicates ground truth object mask pixels not associated with predicted pixels.

Based on these error metrics, our segmentation model has demonstrated excellent performance, achieving an IoU of 0.93 and a Dice score of 0.95.

Regarding the prediction module instead, we leveraged

the Mean Squared Error (MSE) function and we achieved a value of 15.29.

An example of the prediction capabilities of our system is shown in Fig. 4.

6. Conclusion

In this study, we have developed and evaluated a comprehensive approach for forecasting solar wind density, addressing the critical need for accurate space weather predictions. Our system distinguishes itself for the balance between computation load and the very high precision in accurately segmenting coronal holes from highresolution sun images and the good forecasting capabilities of the prediction module that efectively combines the LSTM and ConvLSTM networks. Moving forward, we propose future improvements by substituting the convolutional layers with Vision Transformers [ 26, 27, 28 ]. This enhancement strategy aims to further elevate the accuracy and robustness of our system, paving the way for more precise solar wind density forecasts and improved space weather predictions. Furthermore, another improvement to enhance the performance of our system involves expanding the dataset used to train our customized U-Net architecture. Specifically, we aim to increase the volume of manually generated coronal hole segmentation maps within the dataset. By incorporating a more extensive and diverse dataset, we anticipate boosting the generability power of the network. This expansion strategy is expected to enable the neural network to generalize better to unseen data and variations in coronal hole structures. Improved generalization can also benefit the prediction module by achieving better results and reducing forecasting errors.

Acknowledgments

This work has been developed at is.Lab() Intelligent Sys

tems Laboratory at the Department of Computer, Control, and Management Engineering, Sapienza University of Rome (https:// islab.diag.uniroma1.it). The work has also been partially supported from Italian Ministerial grant PRIN 2022 “ISIDE: Intelligent Systems for Infrastructural Diagnosis in smart-concretE”, n. 2022S88WAY - CUP B53D2301318, and by the Age-It: Ageing Well in an ageing society project, task 9.4.1 work package 4 spoke 9, within topic 8 extended partnership 8, under the National Recovery and Resilience Plan (PNRR), Mission 4 Component 2 Investment 1.3—Call for tender No. 1557 of 11/10/2022 of Italian Ministry of University and Research funded by the European Union—NextGenerationEU, CUP B53C22004090006.

References

[1] F. Bonanno , G. Capizzi , A. Gagliano , C. Napoli , Optimal management of various renewable energy sources by a new forecasting method , 2012 , pp. 934 - 940 . doi: 10 .1109/SPEEDAM. 2012 . 6264603 .

[2] N. Brandizzi , S. Russo , G. Galati, C. Napoli , Addressing vehicle sharing through behavioral analysis: A solution to user clustering using recency-frequencymonetary and vehicle relocation based on neighborhood splits , Information (Switzerland) 13 ( 2022 ). doi: 10 .3390/info13110511.

[3] O. Ronneberger , P. Fischer , T. Brox , U-net: Convolutional networks for biomedical image segmentation, in: Medical image computing and computerassisted intervention-MICCAI 2015: 18th international conference , Munich, Germany, October 5- 9 , 2015 , proceedings, part III 18 , Springer, 2015 , pp. 234 - 241 .

[4] N. N. Dat , V. Ponzi , S. Russo , F. Vincelli , Supporting impaired people with a following robotic assistant by means of end-to-end visual target navigation and reinforcement learning approaches , volume 3118 , 2021 , pp. 51 - 63 .

[5] G. De Magistris , R. Caprari , G. Castro, S. Russo , L. Iocchi , D. Nardi , C. Napoli , Vision-based holistic scene understanding for context-aware humanrobot interaction 13196 LNAI ( 2022 ) 310 - 325 . doi: 10 .1007/978- 3- 031 - 08421- 8_ 21 .

[6] S. Hochreiter , J. Schmidhuber , Long short-term memory , Neural computation 9 ( 1997 ) 1735 - 1780 .

[7] V. Ponzi , S. Russo , V. Bianco , C. Napoli , A. Wajda , Psychoeducative social robots for an healthier lifestyle using artificial intelligence: a case-study , volume 3118 , 2021 , pp. 26 - 33 .

[8] X. Shi , Z. Chen , H. Wang , D.- Y. Yeung , W.-K. Wong, W.-c. Woo, Convolutional lstm network: A machine learning approach for precipitation nowcasting , Advances in neural information processing systems 28 ( 2015 ).

[9] E. A. Illarionov , A. G. Tlatov , Segmentation of coronal holes in solar disc images with a convolutional neural network , Monthly Notices of the Royal Astronomical Society 481 ( 2018 ) 5014 - 5021 .

[10] S. Pepe , S. Tedeschi , N. Brandizzi , S. Russo , L. Iocchi , C. Napoli , Human attention assessment using a machine learning approach with gan-based data augmentation technique trained using a custom dataset , OBM Neurobiology 6 ( 2022 ). doi: 10 . 21926/obm.neurobiol. 2204139 .

[11] Š. Mackovjak , M. Harman , V. Maslej-Krešňáková , P. Butka , Scss-net: solar corona structures segmentation by deep learning , Monthly Notices of the Royal Astronomical Society 508 ( 2021 ) 3111 - 3124 .

[12] C. Verbeeck , V. Delouille , B. Mampaey , R. De Visscher, The spoca-suite: Software for extraction, tection 14126 LNAI ( 2023 ) 3 - 16 . doi: 10 .1007/ characterization, and tracking of active regions and 978- 3- 031- 42508- 0_1. coronal holes on euv images , Astronomy & Astro- [27] A. Dosovitskiy , L. Beyer , A. Kolesnikov , D. Weisphysics 561 ( 2014 ) A29 . senborn, X. Zhai, T. Unterthiner , M. Dehghani,

[13] T. M. Garton , P. T. Gallagher , S. A. Murray , Auto- M. Minderer , G. Heigold, S. Gelly , et al., An image is mated coronal hole identification via multi-thermal worth 16x16 words: Transformers for image recogintensity segmentation , Journal of Space Weather nition at scale , arXiv preprint arXiv:2010.11929 and Space Climate 8 ( 2018 ) A02 . ( 2020 ).

[14] A. Tlatov , K. Tavastsherna , V. Vasil'eva , Coronal [28] G. De Magistris , M. Romano , J. Starczewski, holes in solar cycles 21 to 23, Solar Physics 289 C. Napoli , A novel dwt-based encoder for human ( 2014 ) 1349 - 1358 . pose estimation, volume 3360 , 2022 , pp. 33 - 40 .

[15] H. Raju , S. Das , Cnn-based deep learning model for solar wind forecasting , Solar Physics 296 ( 2021 ) 134 .

[16] S. Albawi , T. A. Mohammed , S. Al-Zawi , Understanding of a convolutional neural network , in: 2017 international conference on engineering and technology (ICET) , Ieee, 2017 , pp. 1 - 6 .

[17] V. Upendran , M. C. Cheung , S. Hanasoge , G. Krishnamurthi, Solar wind prediction using deep learning , Space Weather 18 ( 2020 ) e2020SW002478 .

[18] C. Szegedy , W. Liu, Y. Jia , P. Sermanet , S. Reed , D. Anguelov , D. Erhan , V. Vanhoucke , A. Rabinovich , Going deeper with convolutions , in: Proceedings of the IEEE conference on computer vision and pattern recognition , 2015 , pp. 1 - 9 .

[19] R. M. Schmidt , Recurrent neural networks (rnns): A gentle introduction and overview , arXiv preprint arXiv: 1912 . 05911 ( 2019 ).

[20] C. Baumann , A. E. McCloskey , Timing of the solar wind propagation delay between l1 and earth based on machine learning , Journal of Space Weather and Space Climate 11 ( 2021 ) 41 .

[21] L. Breiman , Random forests, Machine learning 45 ( 2001 ) 5 - 32 .

[22] J. H. Friedman , Greedy function approximation: a gradient boosting machine , Annals of statistics ( 2001 ) 1189 - 1232 .

[23] F. Pedregosa , G. Varoquaux , A. Gramfort , V. Michel , B. Thirion , O. Grisel , M. Blondel , P. Prettenhofer , R. Weiss , V. Dubourg , et al., Scikit-learn: Machine learning in python , the Journal of machine Learning research 12 ( 2011 ) 2825 - 2830 .

[24] T. Rotter , A. Veronig , M. Temmer , B. Vršnak , Realtime solar wind prediction based on sdo/aia coronal hole data , Solar Physics 290 ( 2015 ) 1355 - 1370 .

[25] M. Owens , H. E. Spence , S. McGregor , W. Hughes , J. Quinn , C. Arge , P. Riley , J. Linker , D. Odstrcil , Metrics for solar wind prediction models: Comparison of empirical, hybrid, and physics-based schemes with 8 years of l1 observations, Space Weather 6 ( 2008 ).

[26] A. Alfarano , G. De Magistris, L. Mongelli , S. Russo , J. Starczewski , C. Napoli , A novel convmixer transformer based architecture for violent behavior de-