The Sentinel-1 satellites equipped with synthetic aperture radars (SAR) provide near global coverage of the world's oceans every six days. We curate a data set of co-locations between SAR and altimeter satellites, and investigate the use of deep learning to predict significant wave height from SAR. While previous models for predicting geophysical quantities from SAR rely heavily on feature-engineering, our approach learns directly from low-level image cross-spectra. Training on co-locations from 2015-2017, we demonstrate on test data from 2018 that deep learning reduces the state-of-the-art root mean squared error by 50%, from 0.6 meters to 0.3 meters.
Synthetic aperture radar (SAR) is an important remote
sensing technology able to achieve high spatial resolution (< 10
meter). From SAR satellite data, geophysical properties can
be predicted using statistical models, enabling researchers to
monitor global sea states with unprecedented coverage,
precision, frequency, and without the use of complicated SAR
modulation transfer functions
SARs capture sea surface roughness and many other
geophysical phenomena
In this work, we attempt to extract additional
information from SAR images using deep learning with artificial
neural networks. Deep learning has proven to be an
extremely effective approach to representation learning,
leading to rapid advances in diverse fields such as computer
vision
In this work, we first curate a data set of over 750,000 co-locations of SAR and altimeter satellites, which provides SAR in conjunction with direct measurements of ocean wave heights. The data is used to train deep neural networks to predict significant wave height, Hs, defined as the mean of the top third of a wave height distribution. We compare training on SAR image spectra vs. high-level CWAVE features, and analyze the effect of training data set size.
We curate a training set of historical data from two types
of polar-orbiting satellites: Sentinel-1 SAR satellites and
altimeter satellites. Because the satellites are on different
trajectories, their paths frequently intersect, providing
measurements from roughly the same location at the same time.
Specifically, the data set is constructed using measurements
that are less than 3 hours apart and with spatial differences
less than 300 km, resulting in 753,777 co-location events
from 2015 through 2018 that are geographically well
distributed (Figure 1). These events have both SAR imaging
from Sentinel-1 and significant wave height from an
altimeter, and provide a high-fidelity reference data set
The data set is split into training, validation, and test sets based on year of data collection. Co-location events from 2015 and 2016 were used as the training set, events from 2017 was used as a validation set, and events from 2018 was used as held-out test set. The result was 303,574 training examples, 265,052 validation examples, and 185,151 test examples. The validation set was used for learning rate annealing, early stopping, and hyper-parameter selection, while the test set was only used for the final evaluation of the model.
The Sentinel-1 SAR data set consists of the real and
imaginary components computed from SAR modulation cross
spectra. Each data point within the cross spectra was
created by taking the Level 1 SAR image with 5 5 m pixel
resolution covering a 20 20 km area and applying a 2D
Fourier transformation to different ”looks” within the dwell
time
In addition to the SAR image spectra, we include a number of high-level features in our model. First, we include the time and distance between the satellite co-location measurements, normalized to have zero mean and unit variance — while this information is only available during training. The time (or distance) between satellite observations provides a rough estimate of how much we can trust the altimeter measurement to provide an accurate target because sea states can change faster than our original time and space constraints. These features are simply set to zero at prediction time. Second, the time-of-day was encoded as a value between -1 and 1 using the function f (t) = 2sin( 248t ) 1; this normalization helps stabilize the neural network optimization. Third, latitude and longitude were encoded by representing each as an angle in the range [0; 2 ) then taking the sine and cosine, resulting in four features total. Fourth, a binary label was created to specify the SAR satellite; S1-A or S1-B are calibrated to produce comparable data, but there could be small differences. Finally, we also include the 20 non-dimensional CWAVE parameters that are derived from the image spectra, each normalized using standard scaling to have zero mean and unit variance over the training examples. t u p n I
Dense Dense 256 128 Dense 256
Deep Neural Network Architecture We propose a deep neural network architecture that predicts significant wave height by using the input data from SAR image spectra. The model starts as two branches with separate inputs: one which processes the spectral input and another which extracts information from the high-level features (Figure 3). The spectral input branch takes an input tensor of the shape (72, 60, 2) where the real and imaginary values of the Fourier transform are stacked along the third axis, analogous to the ‘colors’ of an RGB image. This input tensor is then fed sequentially into three convolutional layers containing 64, 128, and 256 filters respectively. A filter size of 3 3 is maintained at each convolutional layer with a rectified linear unit (ReLU) activation. In addition, each layer is followed by a max pooling layer with a 2 2 window. The final convolutional layer is fed into a global max-pooling layer which produces a flattened array of size 256. This is then fed into two additional dense layers with 256 hidden units each with ReLU activation. The non-spectral data branch consists an input layer of the following 32 features: 20 CWAVE features
1 incidence angle mode feature (binary flag representing WV1 or WV2) 1 satellite source feature (binary flag representing Sentinel-1A and Sentinel-1B) 1 time difference between altimeter and sentinel measurements feature 1 spatial difference between altimeter and sentinel measurements feature 1 normalized radar cross section 0 feature 1 normalized variance of radar cross section feature. These 32 high-level features are fed into 11 dense layers with 256 ReLU hidden units each. Both branches yield a flattened array of size 256 which are then concatenated to form a single vector with 512 features. Two hidden dense layers of 256 and 128 hidden ReLU units then integrate the image spectra branch with the second branch. Finally, an output layer with a dropout of 0.337 and softplus activation makes the final prediction.
This model is trained to minimize the mean squared
error (MSE) using the Adam optimizer
Weights are initialized using the scaling suggested by
To compare the three types of models after hyper-parameter tuning, we trained each on data from 2015-2016, and tested on events from 2018, enabling us to explore the relative benefits of deep learning and the use of image spectra features. Table 2 shows that the deep neural network trained on image spectra achieves a significantly lower root mean squared error (RMSE) of 0.33 meters compared to the other methods that rely only on the high-level features: 0.64 m for the linear model and 0.43 m for the deep neural network trained on CWAVE alone. Furthermore, this performance improvement is consistent across small, medium, and large waves.
A feature importance study (Table 2) shows the dependence on each set of features. Two additional models were trained with an identical DNN architecture and hyperparameters, but with specific high-level features removed: the 20 CWAVE parameters removed or the latitude and longitude features. In both models, time of day, incidence angle, incidence angle mode, satellite type, time and distance difference between altimeter and sentinel data, normalized radar cross section 0 and normalized variance of radar cross section are still included. The results show that the high-level CWAVE features are still used by the model, but only slightly — despite containing no additional information, these features add implicit bias to the model. The location features do contain additional information — they essentially allow the model to learn a prior over the wave heights at different locations — but these too only have a small effect on performance.
<1m 1m - 3m 3m - 8m >8m
Finally, we explore the impact of increasing the size of
the training set on the discrepancy between including and
not including CWAVE parameters in our final model. In
this experiment, we fix the hyper-parameters, train on data
from 2015-2017 (568,626 examples), then test on 2018. The
models are trained for a fixed 30 epochs where the
learning rate is annealed by a factor of 0.4 every 10 epochs.
Initial learning rate and dropout are identical to that of our
optimal deep neural network architecture. Figure 4 shows
the mean performance of six randomly-initialized networks
trained with different fractions of the data set. An
ensemble (arithmetic mean) of the 6 models using all features and
the complete training set gives a test RMSE of 0.307 — a
50% reduction in RMSE from the previous state-of-the-art
of 0.6 m
Our results demonstrate that a deep convolutional neural network can extract useful representations from SAR image spectra that is not captured by engineered CWAVE features. In a direct comparison between two hyper-parameter optimized deep neural networks, the network with the image spectra information obtained a 29% reduction in RMSE (0.33 m vs. 0.43 m). This is in keeping with the success of deep learning in other fields, where the expertly engineered features are discarded in favor of learned features.
Our results show that there is still some advantage to including the CWAVE features in the model, even with a training set of over 500,000 examples. However, we also show that this advantage diminishes as the number of training examples increases (Figure 4). The CWAVE features, being derived from the image data, provide no additional information, but they help bias the model towards a good solution. As the training data set increases, the model is slower to overfit, and the benefit of including the CWAVE features disappears.
Latitude and longitude information is useful for predicting significant wave heights because there are regional characteristics of the wave climate (Stopa et al. 2013). However, the feature importance study shows that our model only makes minimal use of this information. This is encouraging, because it implies that the model is relying almost entirely on the direct measurements rather than geographical information.
Our results demonstrate that deep learning provides a 50% decrease in RMSE compared to the previous state-of-theart in predicting significant wave height from SAR. Instead of relying on the set of engineered CWAVE features that capture most of the discriminative information, our deep learning approach learns directly from the low-level, highdimensional image spectra. Furthermore, our results indicate that there is still room for improvement with additional training data. Thus, we should we should expect the performance of our model to increase as more co-location events are collected over the next couple years.
This work was made possible thanks to SAR data access granted by ESA projects: Sentinel-1 A Mission Performance Center (4000107360/12/I-LG) and Sentinel-1 Ocean Study (S1-4SCI-16-0002). The altimetry data was sourced from the Integrated Marine Observing System (IMOS). All Sentinel-1 L2 data used in this study can be obtained from the Copernicus Data Hub (cophub.copernicus.eu). The authors would like to thank NVIDIA for a hardware grant to PS. The technical support and advanced computing resources from the University of Hawai‘i Information Technology Services Cyberinfrastructure are gratefully acknowledged.