Not only can discovering patterns and insights from atmospheric data enable more accurate weather predictions, but it may also provide valuable information to help tackle climate change. Weather4cast is an open competition that aims to evaluate machine learning algorithms' capability to predict future atmospheric states. Here, we describe our third-place solution to Weather4cast. We present a novel Variational U-Net that combines a Variational Autoencoder's ability to consider the probabilistic nature of data with a U-Net's ability to recover fine-grained details. This solution is an evolution from our fourth-place solution to Trafic4cast 2020 with many commonalities, suggesting its applicability to vastly diferent domains, such as weather and trafic. The code for this solution is available at https://github.com/qiq208/weather4cast2021_Stage1 R1-3 correspond to the core challenge in which training, validation and test data are provided, while regions R4-6 Meteorological satellites around the globe are constantly correspond to the transfer learning challenge in which gathering a trove of data about the atmosphere. How- only the test data are provided. In addition, static inforever, the high-dimensionality nature of atmospheric data mation, such as altitude, latitude and longitude, are also makes it challenging to analyse, hindering the discovery given for all regions. of valuable insights. With the advent of machine learning Weather4cast demands an algorithm that can return methods, it is believed these methods can help better un- the atmospheric states over the defined regions for the derstand atmospheric data. To evaluate the applicability next 8 hours (32-of 15-minute intervals) given an hour of such techniques to atmospheric data, Weather4cast (4-of 15-minute intervals) worth of data. While only 4 [1] by the Institute of Advanced Research in Artificial target variables are required, namely (a Intelligence is an open competition that challenges its channel of CTTH), _ (a channel of CRR), participants to develop algorithms to predict the future ___ (a channel of ASII-TF) and (a states of the atmosphere over specific regions. channel of CMA), any channels of the weather products The Weather4cast dataset [2] is obtained from Me- or static information of the regions can be used as input teosat geostationary meteorological satellites operated variables. by EUMETSAT for the period from February 2019 to This work describes a novel Variational U-Net soluFebruary 2021. The Meteosat images are processed by tion which achieved third place in both the core and NWC SAF software into weather products. The weather transfer learning challenges of Weather4cast. This Variaproducts of interest are: Cloud Top Temperature and tional U-Net can be viewed as a U-Net with a Variational Height (CTTH), Convective Rainfall Rate (CRR), Auto- Autoencoder (VAE) style bottleneck, or as a VAE with matic Satellite Image Interpretation - Tropopause Folding U-Net style skip connections. The intuition behind this detection (ASII-TF), Cloud Mask (CMA), and Cloud Type architecture is to combined VAE's ability to consider the (CT). Each of these weather products is recorded in 15- probabilistic nature of data with U-Net's ability to recover minute intervals and consists of multiple channels. Each ifne-grained details. channel is in the format of an image of shape 256x256 pixels, with each pixel covering an area of about 4x4 km. The regions of interest are illustrated in Figure 1; regions 2. Related work
problem, in which the inputs are the first 4 frames of a
video, and the outputs are the subsequent 32 frames. This
format of the problem is identical to that of Trafic4cast
[
While the formats of Weather4cast and Trafic4cast
are equivalent, the diferences in the underlying domains
cannot be ignored. Specifically, weather is considered
more random than trafic. Multiple scenarios are possible
given a set of observations, and this inherent randomness
needs particular attention, as it is not compatible with
the deterministic nature of a typical U-Net. Segmentation
of medical images also sufers from intrinsic ambiguities.
To handle these ambiguities, Kohl et al. [
ifc4cast, the main structure of the proposed Variational
U-Net largely resembles the authors’ fourth-place
solution to last year’s Trafic4cast [
Inspired by the works of Kohl et al. [
The architecture of the Variational U-Net is shown in Figure 2.
Similar to the authors’ Trafic4cast solution [
A series of experiments were performed to find the most efective set of input features, and the validation set was used to evaluate the performance of each feature set. The resulting input feature set is listed in Table 1, and those rejected are summarised in Table 2.
= 2 + 80 *
(1) 2 is a modified mean squared error, it takes into account missing values and the diference in scale of the 2 = 1 ∑3︁2 ∑︁ ∑︁,(,, − ˆ,,)2 (2) tFrroolmlinignoitviaelrfitetxinpgeroifmthenetms,oidtebletcoamthee atrpapinairnegntdtahtaatwcaosn32 × 4 =1 ∈ , =1 a key to success in both the core and transfer learning challenges. Hence, several regularisation strategies were where = {, _, , employed. Within the model itself, the move to the Varia___}, , is the total number of non- tional U-Net from a traditional U-Net, combined with the missing pixels for a given target variable at a given introduction of dropout layers throughout the encoder time and is the target variable weighting: and decoder, both aimed to improve the generalisation of the model. To expose the model to as much variation in ⎪⎧31.610, = input as possible, a single model was used for all regions ⎪⎪⎨4139.4, = _ in the competition and trained on all available training = ⎪5.2191, = data. Furthermore, for the final leaderboard submission, ⎪ the model was further trained for another cycle on all ⎪⎩142.17, = ___ the validation data available.
is the KL divergence between the estimated Gaussian distribution (, 2) and a prior distribution (0, 1):
= 21 ∑5=1︁21 2 + 2 − log − 1 (3) ttahonedafinlmlaoolwdmefolodarercqlhuwiitacekscetturrarfeieneweddabsoacncokdnaadtnuadcftreleodamornnailnsligrne.gglHeioornewsg,eisovoners,
The factor of 80 in Equation 1 was determined there is a risk that some of the decisions made might
empirically to balance the relative importance of the two not be optimum for a model trained on data from all
terms in the loss function. regions. Results from the main experiments can be found
in Appendix A.
3.4. Optimisation Final experiments on all three regions were conducted,
and models were evaluated based on either the test
The Variational U-Net is trained using the Adam op- learderboard or the final leaderboard. It is worth noting
timiser with Cyclic Cosine Annealing described by that the test leaderboard allowed multiple submissions
Loshchilov et al. [
Using a batch size of 12, the final model was first training phase, as well as the additional cycle training on trained for 6 cycles (12 epochs) on the training data, then the validation data. it was further trained for an additional cycle (2 epochs) on both the training and validation data.
whereas others (e.g. cloud top altitude) did not. It was
found that linearly interpolating temperature provided
Although various U-Net architectures were explored, it an improvement to the validation score, however, this
was interesting to observe that the final architecture was did not read across to the final leaderboard score. The
very similar to the architecture used for Trafic4cast [
In terms of feature engineering, the experiments test leaderboard score (~2.3%) over individually trained showed that the inclusion of some extra features (e.g. models. This finding suggests that that the model may cloud top pressure) improved predictive capability, continue to improve its general predictive ability for any region with the addition of more training data. This hypothesis was further supported as training on the validation data further improved the final leaderboard score for both core and transfer learning challenges.
Weather4cast provided the opportunity to explore the use of machine learning techniques to the age-old problem of weather forecasting. Furthermore, the similarity of format to Trafic4cast also provided the chance to investigate how transferable machine learning models can be across vastly diferent domains. After experimenting with various U-Net architectures, the final model was very similar to the authors’ Trafic4cast model. The main diferences being changes to suppress overfitting, i.e. moving to the Variational U-Net model and inclusion of dropout layers throughout. The authors also found that training on data from all regions in one model outperformed training individual models on each region for both the core and transfer learning challenges. This suggests that the model prediction for all regions can be improved by training on more data.