Towards Very-Low Latency Storm Nowcasting through AI-Based On-Board Satellite Data Processing

Abstract

Satellite-based Earth Observation (EO) is a key technology for applications like emergency management, civilian security, and environment and resource monitoring. Demands on amount, type and quality of remote-sensing satellite data and eficient methods for data analysis have increased sharply in recent years. However, the use of satellite-based image products for scenarios which require very low-latencies, such as rapid meteorological and civil security applications, is still limited by the bottleneck created by the classical EO data chain, which involves the acquisition, compression, and storage of sensor data onboard the satellite, and its transfer to ground for further processing. Onboard processing ofers a promising solution to reduce the latencies between data acquisition and product delivery to the end user. The H2020 EU project EO-ALERT (http://eo-alert-h2020.eu) implements this approach through the development of a next-generation EO data processing chain that moves optimised key elements from the ground segment to onboard the satellite. In this article, the feasibility of the concept is demonstrated using EO-ALERT's extreme weather nowcasting product as an example. The system is able to detect and track convective storms and Overshooting Tops, and to send the processed information to ground, within 5 minutes of the observation.

Keywords

• eol>Earth Observation
• On-Board Processing
• AI
• Low latency
• Nowcasting
• Satellite Architecture

1. Introduction

generated directly onboard the spacecraft and transmitted to ground and to the End User with very low latency.

In many EO scenarios including environment and re- While the EO-ALERT concept and architecture enables source monitoring, emergency management and civilian a wide range of low-latency earth observation products, security, EO products are only useful if available in a two use-case scenarios are developed to proof the feasibilvery short time period. However, the use of satellite EO- ity of the approach: Ship detection and extreme weather based image products for rapid meteorological and civil nowcasting. The ship detection scenario is motivated security applications is still limited by the bottleneck cre- by the European Maritime Safety Agency’s (EMSA) vesated by the classical EO data chain, which involves the sel detection service and ofers possible applications for acquisition, compression, and storage of sensor data on- monitorization of illegal fishing, illegal immigration, and board the satellite, and its transfer to ground for further in search and rescue missions. processing. This introduces long latencies until product In this work, the second application, meteorological delivery to the end user. nowcasting for early warnings of convective storms, is

The H2020 EU project EO-ALERT [ 1, 2 ] (http://eo-alert- used to demonstrate the capabilities of the EO-ALERT h2020.eu) a collaboration of several partner organizations product and its novel data processing chain in a realistic (DEIMOS Space (Leader), DLR, OHB Italy, Politecnico scenario. The article is organized as follows: Section 2 di Torino, TU-GRAZ), addresses this problem through gives an overview of the state of convective storm nowthe development of a next-generation EO data process- casting. In Section 3, the EO-ALERT processing chain is ing chain that moves optimised key elements from the presented as a solution for low latency storm nowcastground segment to onboard the satellite. Applying opti- ing. Sections 4 and 5 present the design and results of the mised Machine Learning (ML) methods, EO products are detection algorithms for convective storm and Overshooting Top (OT) detection. Section 6 briefly summarizes the conclusion of this work.

2. Background

Deep moist convection processes cause damaging efects like heavy rainfall and large hail, strong wind gusts, wind shear, lightning, tornadoes, etc. Those, in turn, produce negative efects like flash floods, power cuts, damaged to move EO data processing elements from the ground crops, etc. which can negatively afect human lives as segment to the satellite and execute image processing well as produce high economic losses [ 3 ]. Overshooting and machine learning algorithms onboard (O/B). Relying tops (OTs) are deep convective storm updrafts able to rise solely on image data acquired O/B, processing can be above the storms’ equilibrium level in the tropopause started as soon as a cloud cell is visible in the acquired region. OTs are directly related to hazardous weather image (Figure 1). By applying AI-based image processat the Earth’s surface such as heavy rainfall, damaging ing, storms can thus be detected before they are seen in winds, large hail, and tornadoes [ 4 ]. An early detection radar data on ground. Alerts for detected convective cells of these kinds of phenomena is beneficial in many senses. are then sent to ground on-the-fly before the actual raw

Meteorological forecasters use non-hydrostatic nu- data is transmitted. The alert information can trigger and merical prediction models (NWP) to forecast convective complement further analysis done on ground by other storm occurrence some days in advance, but the accu- existing solutions. racy these NWP provide does not allow to know their exact location, time and intensity level. For this reason, nowcasting tasks are crucial in these cases. Nowcasting systems of the national meteorological services are mainly fed by remote sensing data like radar and satellite images, and derived products.

The international meteorological community is continuously putting a great deal of efort into the understanding of convective phenomena and the improvement Figure 1: Data availability for forecasting and nowcasting at of nowcasting tools. To that purpose, EUMETSAT has a diferent stages of storm evolution. dedicated Satellite Application Facility (SAF) to provide nowcasting tools (NWCSAF). In particular, the NWCSAF Rapid Developing Thunderstorms – Convection Warning (RDT-CW) product is devoted to the detection, tracking and forecasting of intense convective systems and rapidly developing convective cells [ 5 ].

The RDT-CW is a mainly satellite-based product that provides information on the storm location, size, speed, followed and future trajectory, cloud top cooling rate and severity information, among many other useful descriptive parameters. RDT-CW also performs OT detection.

To compute all this information, RDT-CW uses, besides satellite imagery at diferent times, NWP fields, lightning information and other NWCSAF product outputs.

RDT-CW has been both calibrated and validated against lightning information, and needs between three and nine minutes to be processed (depending on the region processing size and the number of NWCSAF products used as input), once raw data has been downloaded to ground, image generation has reached Level 1.5 and its radiances are available.

The proposed novel satellite processing chain opti

mises the classical EO processing chain in a number of critical aspects and has implications on several technological areas, including high-speed avionics, Flight Segment/Ground Segment (FS/GS) communications, O/B compression and data handling and O/B image generation and processing (Figure 2). In contrast to the classical EO data processing chain this approach does not rely on the transfer of raw data to ground and thus greatly reduces the amount of data transmitted. Together with the EO-ALERT onboard data compression and high data rate communication links, this allows for very low latency product delivery. EO-ALERT has a goal latency of less than 1 min and requires a maximum latency below 5 min for both Synthetic Aperture Radar (SAR) and optical image products, including those from LEO and GEO satellites.

The Hardware (HW) design is implemented as a hybrid solution that uses both Commercial Of-The-Shelf (COTS) and space-qualified components [ 6, 7 ]. COTS are used in conjunction with mitigation techniques to increase robustness of the design against radiation effects, whereas space-qualified components are used for the critical functions. This choice allows keeping weight, volume and cost of the Payload Data Processing Unit (PDPU) low with respect to an all space-grade design and it takes advantage of the state-of-the-art technology and processing power of the latest COTS components. Processing boards are based on the Xilinx Zynq US+ ZU19EG MPSoC featuring a quad core ARM processor and a large Field-Programmable Gate Array (FPGA).

3. EO-ALERT Processing chain for rapid civil alerts

The low latencies required for early warnings of convec

tive storms obtained from meteorological nowcasting and very short-range forecasting are limited by the classical EO data chain, which involves the acquisition, compression, and storage of sensor data onboard the satellite, and its transfer to ground for further processing by products like the RDT-CW. The EO-ALERT project proposes

Ground truth data for training and testing of the ML

algorithm is generated from OPERA weather radar netFigure 2: EO-ALERT’s next generation satellite processing work maximum reflectivity data [ 8 ]. First, radar images chain for rapid civil alerts modifies the classical data chain corresponding to cloud-free days are used for the cre(black, top) based on raw data compression and transfer, by ation of a clutter map. Radar echoes with an intensity new innovative key elements and data flows (red, bottom). less than that locally defined by the clutter map are classified as spurious echoes caused by EM interferences and removed from radar images. The algorithm described in

The proposed processing chain is verified and evalu- Steiner et al. [ 9 ] is then applied to detect mature conated for the Ship detection and Extreme Weather scenario, vective cells in each radar image. After re-projecting using relevant EO sensor data. The Extreme Weather sce- OPERA images to the MSG grid, convective labels are nario will be described in detail in the following. assigned based on the spatial overlap between OPERA convective cells and EO-ALERT candidate cells detected in SEVIRI IR10.8µm images (see 4.2). Only cells within 4. Convective storm detection the OPERA radar network’s coverage (Figure 3 top left) and a latitude below 55∘ 0′N are included. The ground 4.1. Data truth Phase of Life (PoL; Convective Initiation, Mature, A key prerequisite for the development of AI/ML algo- Decaying) is assigned to each candidate cell based on the rithms for the O/B processing chain is the availability of temporal evolution of radar reflectivity, as illustrated in data sets representative of the data to be used onboard Figure 4: A cell which is convective in the radar image (generally uncompressed L0, L1). Optical image genera- stays ‘Mature’ until radar reflectivity decreases below tion from raw data is performed by the O/B processing a threshold (ℎ = 35), after which it is considchain and tested for the ship scenario. The EO-ALERT ered ‘Decaying’. Analogously, going backward in time, dataset for the extreme weather (EW) scenario has been the PoL changes from ‘Mature’ to ‘Convection Initiation’ created from MSG High Rate SEVIRI Level 1.5 data, which when passing below a fixed threshold ( ℎ = 35). is obtained from the EUMETSAT Data Centre and cor- Cells which are not discriminated as convective in the responds to 164 days in 62 periods of one or more con- radar image at any step of their evolution are considered secutive days between 2016 and 2018. Images have size non-convective, while cells at any PoL are considered 1192pxl x 639pxl with a ground sampling distance of 3 convective. km/px, covering a total area of 6.855.192 km² contain- The data set is split into train, validation and test set. ing the European continent. Of the 12 available SEVIRI A summary of the number of dates, images, detected

4.2. Image processing algorithm

• Intra-channel diferences between diferent acquisition times (e.g., min(010.8) − min(− 1510.8)) • Inter-channel diferences for same acquisition times (e.g., min(010.8) − min(06.2)).

Gradient Boosting Decision Tree (GBDT) ensemble classifiers are used for discrimination of convective/nonconvective cells, Phase-of-Life-classification and Overshooting Top detection. For convective discrimination and PoL, separate GBDT models were trained for each configuration of available historical information. PoLclassification is performed using a one-vs-rest scheme only on those cells which have previously been discriminated as ‘convective’.

4) Alert Generation: Finally, alerts are created for cells which have been classified as convective. These alert messages, which contain the comprehensive characterization details of the detected storms, are transferred to ground where they can then be evaluated by the enduser.

Image processing and cell discrimination follows a multistep algorithmic approach (Figure 5) inspired by the RDTCW product. The processing steps are:

1) Candidate Cell Extraction: Radiances from SEVIRI IR10.8µm images are converted to brightness temperature images. Temperature minima with a temperature diference between cell top and cell base greater than 6∘ are detected, the cell boundaries corresponding to each minimum are found, and the candidate mask is created.

2) Candidate Cell Tracking: In order to gather information on the evolution and movement of cells, their trajectory is followed over subsequent acquisition times.

Cells are matched to those found in the previous image based on spatial overlap in subsequent candidate maps.

Ambiguities (splitting, merging of cells) as well as the disappearance and formation of cells are handled. Figure 5: Extreme weather processing steps: Top left: SEVIRI 3) Candidate Cell Discrimination: Each cell is char- IR10.8µm image. Top right: Candidate mask. Bottom left: acterized by its corresponding brightness temperatures Cell tracking. Bottom right: Cell discrimination. in 5 infrared channels in SEVIRI imagery and their respective evolution over time, and cell features for ML classification are created from historical information from up to 4 past and the present acquisition times 4.3. Results ( = − 60, − 45, − 30, − 15, 0 min), combining: • Cell area (in the candidate mask) • Statistics on the Brightness Temperatures (BT) in 5 Channels (Minimum, Maximum, Mean)

For performance and latency evaluation the extreme

weather algorithm was executed on the EO-ALERT EW test set, setting a 366pxl x 366pxl region of interest (ROI) covering approximately 1, 2 x 1062 centered on the Iberian Peninsula. Results are presented for convective/non-convective discrimination.

5. Overshooting Top Detection

The dataset used for overshooting top (OT) classification is the one described in [12]. This dataset consists of two days, 20th June 2013 and 29th July 2013. The first date contains 1365, the second 446 OTs.

The dataset has been divided into 3 sets for training, validation and testing. Training is performed over the data from the entire June 20th, validation over the data from July 29th before 16:10 and testing over the data from July 29th after 16:10.

5.2. Algorithm

Overshooting Top detection is based on the work by Kim et al. [ 13 ]. Feature extraction is performed over the

IR Ch09 (10.8µm) from the candidate regions (extracted

Detection performance. Results in terms of Proba- from step 3 of the convective storm detection algorithm bility of Detection (POD), False Alarm Ratio (FAR) and presented in Section 4.2) where within a region of pixels F1-score are shown in Table 2. Performance improves a standard deviation filter and a center pixel diference (i.e., POD increases, FAR decreases) with each additional filter are applied. With these two filters and a subtraction time step available, reaching POD=0.82 and FAR=0.14 for between Ch05 (6.2µm) and Ch09 (10.8µm), Ch07 (8.7µm) cells with fully available history. When combining the re- and Ch09 (10.8µm), and Ch10 (12.0µm) and Ch09 (10.8µm), sults for all history configurations (POD=0.68, FAR=0.20), classification is carried out using a GBDT classifier. A performance is still compatible with the operational RDT non-maximum suppression algorithm is applied to group product. The result for “RDT v2018 (OPERA)” is ob- close regions where the same OT has been detected. tained by validating RDT convective cells versus the OPERA-derived ground truth. This is not the data RDT- 5.3. Results CW is originally calibrated on. Due to these diferences in the ground truth data and the classification strategy OT detection is illustrated in Figure 6. In the left picture [ 5, 10, 11 ] the comparison should be considered as qualita- the ground truth of [ 12 ] can be found, and in the right tive. Results shown for the RDT v2011 correspond to the one, the prediction from the EO-ALERT OT detection oficial RDT validation campaign for the verification set- algorithm. True positives are represented in green, false ting “Moderate Lightning Hypothesis, Statistical element positives in red and false negatives in orange. trajectory” [ 11 ]. This setting is similar but not identical to the EO-ALERT “Discrimination” scheme, and values are also reported for purpose of qualitative comparison.

The results suggest that for convective discrimination, the EO-ALERT EW prototype product is compatible and can compete with the RDT-CW operational product.

Latencies. Processing is performed in a dual-board scheme on only one processing board. Table 3 shows the Figure 6: OT detection algorithm results. time elapsed for candidate cell extraction, tracking and discrimination. Assuming additional transfer delays and management tasks, it is possible to have the products ready to be sent to ground in 6 seconds.

Detection performance results from areas containing

an OT which have been obtained over all detected candidate cells are shown in Table 4. The complete discrimination result, including convection and OT detection, is illustrated in Figure 7.

Convection Working Group (cwg.eumetsat.int) for sharing their Overshooting Top database.

6. Conclusions

This paper provides a detailed overview of the EO-ALERT

EW Scenario as a realistic application example of the EO-ALERT data processing and communication pipeline, which provides low-latency nowcasting of convective storms by performing machine learning-based EO satellite image analysis directly O/B the satellite. The modular storm detection system consisting of candidate convective cell extraction, tracking and ML-based discrimination of convective storms and overshooting tops obtains promising qualitative (comparison with the RDTCW product) and quantitative (validation against OPERA radar data and CWG OT database) results. Results from hardware testing show that the demanding objective of providing EO products with a latency below 5 min from data acquisition to product delivery, including data handling, processing and transmission to ground, can be achieved and global EO product latencies below 1 min are feasible in realistic scenarios.

Acknowledgments

This project has been supported by the European Union and the H2020 Research and Innovation program. We thank EUMETSAT for providing MSG-SEVIRI data, EUMETNET for OPERA radar data [8], and the EUMETSAT

References

[1] M. Kerr , S. Cornara , A. Latorre , S. Tonetti , A. Fiengo , S. Aguero , J. I. Bravo , D. Velotto , M. Eineder , S. Jacobsen , H. Breit , O. Koudelka , F. Teschl , E. Magli, T. Bianchi , R. Freddi , M. Benetti , R. Fabrizi , S. Fraile , C. Marcos, EO-ALERT: A Novel Flight Segment Architecture for EO Satellites Providing Very Low Latency Data Products , in: Earth Observation Φ -week, 2019 .

[2] S. Tonetti , S. Cornara , G. V. D. Miguel , L. Carzana , M. Kerr , R. Fabrizi , S. Fraile , C. M. Martín , D. Velotto , EO-ALERT: Next Generation Satellite Processing Chain for Security-Driven Early Warning Capacity in Maritime Surveillance and Extreme Weather Events , in: Living Planet Symposium, Milan, 2019 .

[3] N. Dotzek , P. Groenemeijer , B. Feuerstein , A. M. Holzer , Overview of ESSL's severe convective storms research using the European Severe Weather Database ESWD , Atmospheric Research 93 ( 2009 ) 575 - 586 . URL: http://dx.doi.org/10.1016/j. atmosres. 2008 . 10 .020. doi: 10 .1016/j.atmosres. 2008 . 10 .020.

[4] K. M. Bedka , Overshooting cloud top detections using MSG SEVIRI Infrared brightness temperatures and their relationship to severe weather over Europe , Atmospheric Research 99 ( 2011 ) 175 - 189 . URL: http://dx.doi.org/10.1016/j.atmosres. 2010 . 10 . 001. doi: 10 .1016/j.atmosres. 2010 . 10 .001.

[5] F. Autonès , J.-M. Moisselin , Algorithm Theoretical Basis Document for the Convection Product Processors of the NWC/GEO, 2019 . URL: https://www. nwcsaf.org/web/guest/scientificdocumentation.

[6] K. LaBel, Commercial Of The Shelf (COTS): Radiation Efects Considerations and Approaches, in: NASA Electronic Parts and Packaging Program (NEPP) Electronics Technology Workshop (ETW), 2012 . URL: https://nepp.nasa.gov/workshops/etw2012/talks/ Tuesday/T14_LaBel_COTS_Radiation_Efects.pdf .

[7] D. Sinclair , J. Dyer , Radiation Efects and COTS Parts in SmallSats, 27th Annual AIAA/USU Conference on Small Satellites ( 2013 ) 1 - 12 . URL: http://digitalcommons.usu.edu/smallsat/2013/ all2013/69/.

[8] E. Saltikof , G. Haase , L. Delobbe , N. Gaussiat , M. Martet , D. Idziorek , H. Leijnse , P. Novák , M. Lukach , K. Stephan , OPERA the Radar Project, Atmosphere 10 ( 2019 ) 320 . doi: 10 .3390/ atmos10060320.

[9] M. Steiner , R. A. Houze Jr. , S. E. Yuter , Climatological Characterization of Three-Dimensional Storm Structure from Operational Radar and Rain Gauge Data , Journal of Applied Meteorology 34 ( 1995 ) 1978 - 2007 .

[10] F. Autonès , M. Claudon , Validation report of the Convection Product Processors of the NWC/GEO , 2019 . URL: https://www.nwcsaf.org/web/guest/ scientificdocumentation.

[11] F. Autonès , J.-M. Moisselin , Validation report of the Convection Product Processors of the NWC/GEO , 2016 . URL: https://www.nwcsaf.org/ web/guest/scientificdocumentation.

[12] M. Setvák , M. Radová , J. Kaňák , M. Valachová , K. M. Bedka , J. Šťástka , P. Novák , H. Kyznarová , Comparison of the MSG 2.5-minute Rapid Scan Data and Products derived from these, with Radar and Lightning Observations , EUMETSAT Proceedings ( 2014 ) 22 - 26 .

[13] M. Kim , J. Im , H. Park , S. Park, M. I. Lee , M. H. Ahn , Detection of tropical overshooting cloud tops using himawari-8 imagery, Remote Sensing 9 ( 2017 ) 1 - 19 . doi: 10 .3390/rs9070685.