=Paper=
{{Paper
|id=Vol-3052/short1
|storemode=property
|title=Towards Very-Low Latency Storm Nowcasting through AI-Based On-Board Satellite Data Processing
|pdfUrl=https://ceur-ws.org/Vol-3052/short1.pdf
|volume=Vol-3052
|authors=Robert Hinz,,Álvaro Morón,,Juan Ignacio Bravo,,Murray Kerr,,Cecilia Marcos,,Antonio Latorre,,Francisco Membibre
|dblpUrl=https://dblp.org/rec/conf/cikm/HinzMBKMLM21
}}
==Towards Very-Low Latency Storm Nowcasting through AI-Based On-Board Satellite Data Processing==
https://ceur-ws.org/Vol-3052/short1.pdf
Towards Very-Low Latency Storm Nowcasting through
AI-Based On-Board Satellite Data Processing
Robert Hinz1 , Álvaro Morón1 , Juan Ignacio Bravo1 , Murray Kerr1 , Cecilia Marcos2 ,
Antonio Latorre1 and Francisco Membibre1
1
DEIMOS Space S.L.U., 28760, Tres Cantos – Madrid, Spain
2
Agencia Estatal de Meteorología, Spain
Abstract
Satellite-based Earth Observation (EO) is a key technology for applications like emergency management, civilian security,
and environment and resource monitoring. Demands on amount, type and quality of remote-sensing satellite data and
efficient methods for data analysis have increased sharply in recent years. However, the use of satellite-based image products
for scenarios which require very low-latencies, such as rapid meteorological and civil security applications, is still limited
by the bottleneck created by the classical EO data chain, which involves the acquisition, compression, and storage of sensor
data onboard the satellite, and its transfer to ground for further processing. Onboard processing offers a promising solution
to reduce the latencies between data acquisition and product delivery to the end user. The H2020 EU project EO-ALERT
(http://eo-alert-h2020.eu) implements this approach through the development of a next-generation EO data processing chain
that moves optimised key elements from the ground segment to onboard the satellite. In this article, the feasibility of the
concept is demonstrated using EO-ALERT’s extreme weather nowcasting product as an example. The system is able to detect
and track convective storms and Overshooting Tops, and to send the processed information to ground, within 5 minutes of
the observation.
Keywords
Earth Observation, On-Board Processing, AI, Low latency, Nowcasting, Satellite Architecture
1. Introduction generated directly onboard the spacecraft and transmit-
ted to ground and to the End User with very low latency.
In many EO scenarios including environment and re- While the EO-ALERT concept and architecture enables
source monitoring, emergency management and civilian a wide range of low-latency earth observation products,
security, EO products are only useful if available in a two use-case scenarios are developed to proof the feasibil-
very short time period. However, the use of satellite EO- ity of the approach: Ship detection and extreme weather
based image products for rapid meteorological and civil nowcasting. The ship detection scenario is motivated
security applications is still limited by the bottleneck cre- by the European Maritime Safety Agency’s (EMSA) ves-
ated by the classical EO data chain, which involves the sel detection service and offers possible applications for
acquisition, compression, and storage of sensor data on- monitorization of illegal fishing, illegal immigration, and
board the satellite, and its transfer to ground for further in search and rescue missions.
processing. This introduces long latencies until product In this work, the second application, meteorological
delivery to the end user. nowcasting for early warnings of convective storms, is
The H2020 EU project EO-ALERT [1, 2] (http://eo-alert- used to demonstrate the capabilities of the EO-ALERT
h2020.eu) a collaboration of several partner organizations product and its novel data processing chain in a realistic
(DEIMOS Space (Leader), DLR, OHB Italy, Politecnico scenario. The article is organized as follows: Section 2
di Torino, TU-GRAZ), addresses this problem through gives an overview of the state of convective storm now-
the development of a next-generation EO data process- casting. In Section 3, the EO-ALERT processing chain is
ing chain that moves optimised key elements from the presented as a solution for low latency storm nowcast-
ground segment to onboard the satellite. Applying opti- ing. Sections 4 and 5 present the design and results of the
mised Machine Learning (ML) methods, EO products are detection algorithms for convective storm and Overshoot-
ing Top (OT) detection. Section 6 briefly summarizes the
CDCEO 2021: 1st Workshop on Complex Data Challenges in Earth conclusion of this work.
Observation, virtual, Gold Coast, Australia, November 1, 2021
" robert.hinz@deimos-space.com (R. Hinz);
alvaro.moron@deimos-space.com ( Morón); cmarcosm@aemet.es 2. Background
(C. Marcos)
0000-0001-7333-2477 (R. Hinz); 0000-0002-8861-7376
(C. Marcos)
Deep moist convection processes cause damaging effects
© 2021 Copyright for this paper by its authors. Use permitted under Creative like heavy rainfall and large hail, strong wind gusts, wind
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) shear, lightning, tornadoes, etc. Those, in turn, produce
negative effects like flash floods, power cuts, damaged to move EO data processing elements from the ground
crops, etc. which can negatively affect human lives as segment to the satellite and execute image processing
well as produce high economic losses [3]. Overshooting and machine learning algorithms onboard (O/B). Relying
tops (OTs) are deep convective storm updrafts able to rise solely on image data acquired O/B, processing can be
above the storms’ equilibrium level in the tropopause started as soon as a cloud cell is visible in the acquired
region. OTs are directly related to hazardous weather image (Figure 1). By applying AI-based image process-
at the Earth’s surface such as heavy rainfall, damaging ing, storms can thus be detected before they are seen in
winds, large hail, and tornadoes [4]. An early detection radar data on ground. Alerts for detected convective cells
of these kinds of phenomena is beneficial in many senses. are then sent to ground on-the-fly before the actual raw
Meteorological forecasters use non-hydrostatic nu- data is transmitted. The alert information can trigger and
merical prediction models (NWP) to forecast convective complement further analysis done on ground by other
storm occurrence some days in advance, but the accu- existing solutions.
racy these NWP provide does not allow to know their
exact location, time and intensity level. For this reason,
nowcasting tasks are crucial in these cases. Nowcast-
ing systems of the national meteorological services are
mainly fed by remote sensing data like radar and satellite
images, and derived products.
The international meteorological community is con-
tinuously putting a great deal of effort into the under-
standing of convective phenomena and the improvement Figure 1: Data availability for forecasting and nowcasting at
of nowcasting tools. To that purpose, EUMETSAT has a different stages of storm evolution.
dedicated Satellite Application Facility (SAF) to provide
nowcasting tools (NWCSAF). In particular, the NWCSAF
Rapid Developing Thunderstorms – Convection Warning The proposed novel satellite processing chain opti-
(RDT-CW) product is devoted to the detection, tracking mises the classical EO processing chain in a number
and forecasting of intense convective systems and rapidly of critical aspects and has implications on several tech-
developing convective cells [5]. nological areas, including high-speed avionics, Flight
The RDT-CW is a mainly satellite-based product that Segment/Ground Segment (FS/GS) communications, O/B
provides information on the storm location, size, speed, compression and data handling and O/B image genera-
followed and future trajectory, cloud top cooling rate and tion and processing (Figure 2). In contrast to the classical
severity information, among many other useful descrip- EO data processing chain this approach does not rely
tive parameters. RDT-CW also performs OT detection. on the transfer of raw data to ground and thus greatly
To compute all this information, RDT-CW uses, besides reduces the amount of data transmitted. Together with
satellite imagery at different times, NWP fields, light- the EO-ALERT onboard data compression and high data
ning information and other NWCSAF product outputs. rate communication links, this allows for very low la-
RDT-CW has been both calibrated and validated against tency product delivery. EO-ALERT has a goal latency of
lightning information, and needs between three and nine less than 1 min and requires a maximum latency below 5
minutes to be processed (depending on the region pro- min for both Synthetic Aperture Radar (SAR) and opti-
cessing size and the number of NWCSAF products used cal image products, including those from LEO and GEO
as input), once raw data has been downloaded to ground, satellites.
image generation has reached Level 1.5 and its radiances The Hardware (HW) design is implemented as a hy-
are available. brid solution that uses both Commercial Off-The-Shelf
(COTS) and space-qualified components [6, 7]. COTS
are used in conjunction with mitigation techniques to
3. EO-ALERT Processing chain for increase robustness of the design against radiation ef-
fects, whereas space-qualified components are used for
rapid civil alerts the critical functions. This choice allows keeping weight,
The low latencies required for early warnings of convec- volume and cost of the Payload Data Processing Unit
tive storms obtained from meteorological nowcasting and (PDPU) low with respect to an all space-grade design and
very short-range forecasting are limited by the classical it takes advantage of the state-of-the-art technology and
EO data chain, which involves the acquisition, compres- processing power of the latest COTS components. Pro-
sion, and storage of sensor data onboard the satellite, cessing boards are based on the Xilinx Zynq US+ ZU19EG
and its transfer to ground for further processing by prod- MPSoC featuring a quad core ARM processor and a large
ucts like the RDT-CW. The EO-ALERT project proposes Field-Programmable Gate Array (FPGA).
channels, 5 are used for the generation of the data set:
Ch05 (WV 6.2µm), Ch06 (WV 7.3µm), Ch07 (IR Window
channel 8.7µm), Ch09 (IR10.8µm), Ch10 (IR12.0µm).
Figure 3: Ground Truth data generation from OPERA radar
data.
Ground truth data for training and testing of the ML
algorithm is generated from OPERA weather radar net-
Figure 2: EO-ALERT’s next generation satellite processing work maximum reflectivity data [8]. First, radar images
chain for rapid civil alerts modifies the classical data chain corresponding to cloud-free days are used for the cre-
(black, top) based on raw data compression and transfer, by ation of a clutter map. Radar echoes with an intensity
new innovative key elements and data flows (red, bottom).
less than that locally defined by the clutter map are clas-
sified as spurious echoes caused by EM interferences and
removed from radar images. The algorithm described in
The proposed processing chain is verified and evalu- Steiner et al. [9] is then applied to detect mature con-
ated for the Ship detection and Extreme Weather scenario, vective cells in each radar image. After re-projecting
using relevant EO sensor data. The Extreme Weather sce- OPERA images to the MSG grid, convective labels are
nario will be described in detail in the following. assigned based on the spatial overlap between OPERA
convective cells and EO-ALERT candidate cells detected
in SEVIRI IR10.8µm images (see 4.2). Only cells within
4. Convective storm detection the OPERA radar network’s coverage (Figure 3 top left)
and a latitude below 55∘ 0′ N are included. The ground
4.1. Data truth Phase of Life (PoL; Convective Initiation, Mature,
A key prerequisite for the development of AI/ML algo- Decaying) is assigned to each candidate cell based on the
rithms for the O/B processing chain is the availability of temporal evolution of radar reflectivity, as illustrated in
data sets representative of the data to be used onboard Figure 4: A cell which is convective in the radar image
(generally uncompressed L0, L1). Optical image genera- stays ‘Mature’ until radar reflectivity decreases below
tion from raw data is performed by the O/B processing a threshold (𝑡ℎ𝑑𝑒𝑐 = 35𝑑𝐵𝑍), after which it is consid-
chain and tested for the ship scenario. The EO-ALERT ered ‘Decaying’. Analogously, going backward in time,
dataset for the extreme weather (EW) scenario has been the PoL changes from ‘Mature’ to ‘Convection Initiation’
created from MSG High Rate SEVIRI Level 1.5 data, which when passing below a fixed threshold (𝑡ℎ𝑝𝑟𝑒 = 35𝑑𝐵𝑍).
is obtained from the EUMETSAT Data Centre and cor- Cells which are not discriminated as convective in the
responds to 164 days in 62 periods of one or more con- radar image at any step of their evolution are considered
secutive days between 2016 and 2018. Images have size non-convective, while cells at any PoL are considered
1192pxl x 639pxl with a ground sampling distance of 3 convective.
km/px, covering a total area of 6.855.192 km² contain- The data set is split into train, validation and test set.
ing the European continent. Of the 12 available SEVIRI A summary of the number of dates, images, detected
• Intra-channel differences between different
acquisition times (e.g., min(𝐵𝑇0𝑚𝑖𝑛 𝐼𝑅10.8
) −
𝐼𝑅10.8
min(𝐵𝑇−15𝑚𝑖𝑛 ))
• Inter-channel differences for same acquisition
times (e.g., min(𝐵𝑇0𝑚𝑖𝑛
𝐼𝑅10.8 𝑊 𝑉 6.2
) − min(𝐵𝑇0𝑚𝑖𝑛 )).
Depending on the historical information available for a
Figure 4: Classification schemes: Labels assigned for Phase cell, this results in a total number of 47, 108, 184, 275 or
of Life (CI=1, Mature=2, Decaying=3) and Convective Dis- 381 features.
crimination (Convective=1). Non-convective=0 in both cases. Gradient Boosting Decision Tree (GBDT) ensemble
classifiers are used for discrimination of convective/non-
convective cells, Phase-of-Life-classification and Over-
Table 1
EO-ALERT EW data set information
shooting Top detection. For convective discrimination
and PoL, separate GBDT models were trained for each
Set Days Images Cells % Conv configuration of available historical information. PoL-
Total 164 8358 2289822 26 classification is performed using a one-vs-rest scheme
only on those cells which have previously been discrimi-
Train 123 6369 1719448 25
Val. 17 910 275859 30 nated as ‘convective’.
Test 24 1079 294515 27 4) Alert Generation: Finally, alerts are created for
cells which have been classified as convective. These
alert messages, which contain the comprehensive charac-
terization details of the detected storms, are transferred
candidate cells and the ratio of convective cells is shown
to ground where they can then be evaluated by the end-
in Table 1.
user.
4.2. Image processing algorithm
Image processing and cell discrimination follows a multi-
step algorithmic approach (Figure 5) inspired by the RDT-
CW product. The processing steps are:
1) Candidate Cell Extraction: Radiances from SE-
VIRI IR10.8µm images are converted to brightness tem-
perature images. Temperature minima with a temper-
ature difference between cell top and cell base greater
than 6∘ 𝐶 are detected, the cell boundaries corresponding
to each minimum are found, and the candidate mask is
created.
2) Candidate Cell Tracking: In order to gather in-
formation on the evolution and movement of cells, their
trajectory is followed over subsequent acquisition times.
Cells are matched to those found in the previous image
based on spatial overlap in subsequent candidate maps.
Ambiguities (splitting, merging of cells) as well as the
disappearance and formation of cells are handled. Figure 5: Extreme weather processing steps: Top left: SEVIRI
3) Candidate Cell Discrimination: Each cell is char- IR10.8µm image. Top right: Candidate mask. Bottom left:
Cell tracking. Bottom right: Cell discrimination.
acterized by its corresponding brightness temperatures
in 5 infrared channels in SEVIRI imagery and their re-
spective evolution over time, and cell features for ML
classification are created from historical information
from up to 4 past and the present acquisition times
4.3. Results
(𝑇𝐻 = −60, −45, −30, −15, 0 min), combining: For performance and latency evaluation the extreme
weather algorithm was executed on the EO-ALERT EW
• Cell area (in the candidate mask)
test set, setting a 366pxl x 366pxl region of interest
• Statistics on the Brightness Temperatures (BT) in (ROI) covering approximately 1, 2 x 106 𝑘𝑚2 centered
5 Channels (Minimum, Maximum, Mean) on the Iberian Peninsula. Results are presented for
convective/non-convective discrimination.
Table 2 Table 4
Extreme weather convective discrimination results on SEVIRI- Extreme weather Overshooting Top detection
OPERA test set.
POD FAR F1
History (min) POD FAR F1
0.501 0.448 0.525
0, -15, -30, -45, -60 0.82 0.14 0.84
0, -15, -30, -45 0.70 0.21 0.74
0, -15, -30 0.66 0.23 0.71
0, -15 0.59 0.25 0.66 5. Overshooting Top Detection
0 0.47 0.29 0.57
5.1. Data
Combined 0.68 0.20 0.73
RDT v2018 (OPERA) 0.43 0.28 0.54 The dataset used for overshooting top (OT) classification
RDT v2011 0.74 0.34 - is the one described in [12]. This dataset consists of two
days, 20th June 2013 and 29th July 2013. The first date
contains 1365, the second 446 OTs.
Table 3 The dataset has been divided into 3 sets for training,
Elapsed processing time for optical IP on the target hardware validation and testing. Training is performed over the
Time (s) data from the entire June 20th, validation over the data
from July 29th before 16:10 and testing over the data from
Preprocessing 1.9s
July 29th after 16:10.
Candidate Extraction 1.1s
Tracking 0.4s
Discrimination 0.9s 5.2. Algorithm
Total Elapsed Time 4.3s Overshooting Top detection is based on the work by
Kim et al. [13]. Feature extraction is performed over the
IR Ch09 (10.8µm) from the candidate regions (extracted
Detection performance. Results in terms of Proba- from step 3 of the convective storm detection algorithm
bility of Detection (POD), False Alarm Ratio (FAR) and presented in Section 4.2) where within a region of pixels
F1-score are shown in Table 2. Performance improves a standard deviation filter and a center pixel difference
(i.e., POD increases, FAR decreases) with each additional filter are applied. With these two filters and a subtraction
time step available, reaching POD=0.82 and FAR=0.14 for between Ch05 (6.2µm) and Ch09 (10.8µm), Ch07 (8.7µm)
cells with fully available history. When combining the re- and Ch09 (10.8µm), and Ch10 (12.0µm) and Ch09 (10.8µm),
sults for all history configurations (POD=0.68, FAR=0.20), classification is carried out using a GBDT classifier. A
performance is still compatible with the operational RDT non-maximum suppression algorithm is applied to group
product. The result for “RDT v2018 (OPERA)” is ob- close regions where the same OT has been detected.
tained by validating RDT convective cells versus the
OPERA-derived ground truth. This is not the data RDT-
5.3. Results
CW is originally calibrated on. Due to these differences
in the ground truth data and the classification strategy OT detection is illustrated in Figure 6. In the left picture
[5, 10, 11] the comparison should be considered as qualita- the ground truth of [12] can be found, and in the right
tive. Results shown for the RDT v2011 correspond to the one, the prediction from the EO-ALERT OT detection
official RDT validation campaign for the verification set- algorithm. True positives are represented in green, false
ting “Moderate Lightning Hypothesis, Statistical element positives in red and false negatives in orange.
trajectory” [11]. This setting is similar but not identical
to the EO-ALERT “Discrimination” scheme, and values
are also reported for purpose of qualitative comparison.
The results suggest that for convective discrimination,
the EO-ALERT EW prototype product is compatible and
can compete with the RDT-CW operational product.
Latencies. Processing is performed in a dual-board
scheme on only one processing board. Table 3 shows the Figure 6: OT detection algorithm results.
time elapsed for candidate cell extraction, tracking and
discrimination. Assuming additional transfer delays and
management tasks, it is possible to have the products Detection performance results from areas containing
ready to be sent to ground in 6 seconds. an OT which have been obtained over all detected candi-
date cells are shown in Table 4. The complete discrimi-
nation result, including convection and OT detection, is Convection Working Group (cwg.eumetsat.int) for shar-
illustrated in Figure 7. ing their Overshooting Top database.
References
[1] M. Kerr, S. Cornara, A. Latorre, S. Tonetti, A. Fiengo,
S. Aguero, J. I. Bravo, D. Velotto, M. Eineder, S. Ja-
cobsen, H. Breit, O. Koudelka, F. Teschl, E. Magli,
T. Bianchi, R. Freddi, M. Benetti, R. Fabrizi, S. Fraile,
C. Marcos, EO-ALERT: A Novel Flight Segment
Architecture for EO Satellites Providing Very Low
Latency Data Products, in: Earth Observation
Φ–week, 2019.
[2] S. Tonetti, S. Cornara, G. V. D. Miguel, L. Carzana,
M. Kerr, R. Fabrizi, S. Fraile, C. M. Martín, D. Velotto,
EO-ALERT: Next Generation Satellite Processing
Chain for Security-Driven Early Warning Capac-
ity in Maritime Surveillance and Extreme Weather
Events, in: Living Planet Symposium, Milan, 2019.
[3] N. Dotzek, P. Groenemeijer, B. Feuerstein, A. M.
Figure 7: Illustration of final detection result. Green: Non- Holzer, Overview of ESSL’s severe convec-
convective cells. Red: Convective cells. Green boxes: Over- tive storms research using the European Severe
shooting Top; green lines link the OT to the center of the cell. Weather Database ESWD, Atmospheric Research
Arrows: Direction of cell movement.
93 (2009) 575–586. URL: http://dx.doi.org/10.1016/j.
atmosres.2008.10.020. doi:10.1016/j.atmosres.
2008.10.020.
[4] K. M. Bedka, Overshooting cloud top detections
6. Conclusions using MSG SEVIRI Infrared brightness tempera-
tures and their relationship to severe weather over
This paper provides a detailed overview of the EO-ALERT
Europe, Atmospheric Research 99 (2011) 175–189.
EW Scenario as a realistic application example of the
URL: http://dx.doi.org/10.1016/j.atmosres.2010.10.
EO-ALERT data processing and communication pipeline,
001. doi:10.1016/j.atmosres.2010.10.001.
which provides low-latency nowcasting of convective
[5] F. Autonès, J.-M. Moisselin, Algorithm Theoretical
storms by performing machine learning-based EO satel-
Basis Document for the Convection Product Pro-
lite image analysis directly O/B the satellite. The modu-
cessors of the NWC/GEO, 2019. URL: https://www.
lar storm detection system consisting of candidate con-
nwcsaf.org/web/guest/scientificdocumentation.
vective cell extraction, tracking and ML-based discrim-
[6] K. LaBel, Commercial Off The Shelf (COTS):
ination of convective storms and overshooting tops ob-
Radiation Effects Considerations and Ap-
tains promising qualitative (comparison with the RDT-
proaches, in: NASA Electronic Parts and
CW product) and quantitative (validation against OPERA
Packaging Program (NEPP) Electronics
radar data and CWG OT database) results. Results from
Technology Workshop (ETW), 2012. URL:
hardware testing show that the demanding objective of
https://nepp.nasa.gov/workshops/etw2012/talks/
providing EO products with a latency below 5 min from
Tuesday/T14_LaBel_COTS_Radiation_Effects.pdf.
data acquisition to product delivery, including data han-
[7] D. Sinclair, J. Dyer, Radiation Effects and COTS
dling, processing and transmission to ground, can be
Parts in SmallSats, 27th Annual AIAA/USU
achieved and global EO product latencies below 1 min
Conference on Small Satellites (2013) 1–12. URL:
are feasible in realistic scenarios.
http://digitalcommons.usu.edu/smallsat/2013/
all2013/69/.
Acknowledgments [8] E. Saltikoff, G. Haase, L. Delobbe, N. Gaussiat,
M. Martet, D. Idziorek, H. Leijnse, P. Novák,
This project has been supported by the European Union M. Lukach, K. Stephan, OPERA the Radar
and the H2020 Research and Innovation program. We Project, Atmosphere 10 (2019) 320. doi:10.3390/
thank EUMETSAT for providing MSG-SEVIRI data, EU- atmos10060320.
METNET for OPERA radar data [8], and the EUMETSAT [9] M. Steiner, R. A. Houze Jr., S. E. Yuter, Climatologi-
cal Characterization of Three-Dimensional Storm
Structure from Operational Radar and Rain Gauge
Data, Journal of Applied Meteorology 34 (1995)
1978–2007.
[10] F. Autonès, M. Claudon, Validation report of the
Convection Product Processors of the NWC/GEO,
2019. URL: https://www.nwcsaf.org/web/guest/
scientificdocumentation.
[11] F. Autonès, J.-M. Moisselin, Validation report
of the Convection Product Processors of the
NWC/GEO, 2016. URL: https://www.nwcsaf.org/
web/guest/scientificdocumentation.
[12] M. Setvák, M. Radová, J. Kaňák, M. Valachová, K. M.
Bedka, J. Šťástka, P. Novák, H. Kyznarová, Com-
parison of the MSG 2.5-minute Rapid Scan Data
and Products derived from these, with Radar and
Lightning Observations, EUMETSAT Proceedings
(2014) 22–26.
[13] M. Kim, J. Im, H. Park, S. Park, M. I. Lee, M. H. Ahn,
Detection of tropical overshooting cloud tops using
himawari-8 imagery, Remote Sensing 9 (2017) 1–19.
doi:10.3390/rs9070685.