Earth Observatory is a growing research area that is using AI for short time forecasting, a Now-casting scenario. In this work, we tackle the challenge of weather forecasting using the video transformer network. In recent times, many variants of the vision transformer were explored, with major constraints being the computational complexity of Attention and the data hungry training. We explore the use of Video Swin-Transformer together with a carefully crafted augmentation scheme to tackle the data hungry transformer network. In addition, we use a gradual spatial reduction on the encoder side and cross-attention on the decoder. The proposed network is tested on the Weather4Cast2021 weather forecasting challenge data, which requires the prediction of 8 hours ahead future frames (4 per hour) from an hour weather product sequence. The model results in a highly competitive performance on both the validation and test datasets. The code is available online at https://github.com/bojesomo/Weather4cast2021-SwinEncoderDecoder.
tention networks, external attention, and pyramid vision transformer among others. Dense prediction, including Weather forecasting is an important requirement in au- semantic segmentation, has been also investigated using tonomous vehicles and food production [1], owing to vision transformer as backbone [12, 11]. the relationship between successful implementation of Swin Transformer is among the promising Vision these applications and accurate weather prediction. For transformer architectures, which explore a carefully instance, knowledge of the weather is an important as- crafted patch information mixing using the shift window pect of the location context, when designing autonomous attention technique (hence the name Swin) [13]. The navigation and collision avoidance systems. In food pro- method resulted in high performing image recognition duction, weather forecasting has already proven to be models using the vision transformer. In order to leveran important factor for crop yield management and ad- age its success in classification, researchers explored the equate soil nutrient replenishment. Deep learning has Swin transformer as backbone for dense prediction. Cao been widely used in short time weather forcasting includ- et. al. used Swin transformer blocks in the encoder, boting long short-time memory (LSTM) [2, 3], Autoencoders tleneck and decoder branch of a UNet structure [14]. The [4, 5], CNN [6, 7, 3] and deeply connected neural net- work also introduced the patch expanding layer, i.e., the works [8]. opposite of the patch merging layer, used in the pyramid
Due to the success of attention based networks in nat- vision transformer [14]. The 3-dimensional (3D) variant ural language processing (NLP), many researchers have of the Swin transformer (Video Swin transformer) was recently shifted their attention in exploring their use in proposed by Liu et. al. [15]. This paper used 3D patch computer vision, including image classification, object embedding, 3D shifted window multi-head self attention detection and semantic segmentation [9, 10, 11, 12]. A pi- as well as patch merging, and the proposed approach oneering work in this area, which uses patch based image led to a parameter eficient network with strong perforencoding, is the vision transformer [10]. Dosovitskiy et. mance in a variety of datasets (Kinetics-400, kinetic-600, al. [10], which carefully laid down the techniques for en- and Something-Something v2) [15]. coding image patches similar to word embedding in NLP, In this research, we propose a number of improvefollowed by one-to-one use of a transformer network ments in the video Swin transformer [15], including 3D as proposed by Vaswani et. al. [9]. Many researchers patch expanding, and using cross attention block in the explored this direction by proposing highly eficient at- decoder, as well as a carefully designed data augmentation process, which removes the need for pre-training the network on a large dataset. The proposed architecture and layers are given in Section 2, while the experimental results of the proposed solution in weather4cast2021 stage-1 challenge is presented in Section 3.
CIKM 2021: 1st Workshop on Complex Data Challenges in Earth Observation, Nov 01, 2021 " 100046384@ku.ac.ae (A. Bojesomo); hasan.almarzouqi@ku.ac.ae (H. Al-Marzouqi); panos.liatsis@ku.ac.ae (P. Liatsis)
© 2021 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmmUoRns LWiceonsrekAstthribouptionP4r.0oIncteerenadtiionnagl s(CC(CBYE4U.0)R.-WS.org)
sequence task. To this end, we employed transformer models, which were used in NLP with very promising results. The proposed model uses multiple stages for gradual spatial dimensional reduction in the encoder.
This dimensional reduction is important so as to capture salient representations of global features. While the encoder uses self attention, the decoder uses self attention for its main input and merges the skip connected input from the encoder using mixed attention [9]. The attention layer used in this research is the shifted window attention proposed in [15].
As shown in Fig 1, the input goes through a 3D patch 2.3. Patch merging layer embedding layer, which forms the token provided to the transformer architecture. Output tokens are expanded and projected back to the original format. The model includes three encoder-decoder blocks, each having four 3D transformer layers (encoder/decoder). We limited the number of blocks to three to better handle the data demanding nature of transformers as we do not pre-train our model on any other dataset [13, 15, 10, 12]. Likewise, 2.4. Patch expanding layer we used equal number of transformer layers per block for simplicity (four in our case).
The building blocks of our model shown in fig. (1a) are detailed below.
MLP() = (1)2 (2) where ∈ ℜ... x is the input, 1 ∈ ℜ x 4 is the weight matrix of the first (hidden) fully-connected layer and 2 ∈ ℜ4 x is the weight matrix of the second (output) fully-connected layer.
× 2 neighboring patches, and applies a fully-connected layer on the 4C-dimensional concatenated features to get 2C-dimensional output [13, 15]. This results in a learned down-sampling operation.
prior to computing local attention within the windowed blocks. Also, relative position bias is used in both W-MSA and SW-MSA [13, 15]. Following the work of Li et. al., we used a window size of (1, 7, 7), shift size of 2 in our implementation of the 3D Swin transformer block [15]. For the MLP, we used two fully-connected layers with a ratio of four for the hidden features (eqn. 2). (1)
NLP includes standard multi-head self attention (MSA), followed by a feed-forward network (MLP). Each of these layers is preceded by Layer Normalization (LN) in Vision transformer [10], as opposed to post normalization used in NLP [9]. In this research, 3D shifted window MSA is employed, owing to the spatiotemporal nature of the input. The Swin transformer uses an interchange of slid- 2.6. Encoder ing windows, as shown in Fig (1b and c) with a window (local) attention, followed by another local but shifted window attention. With such a setup, any two layers of attention follow (1): ¯ = W-MSA(LN(− 1)) + − 1 = MLP(LN(¯)) + ¯ ¯+1 = SW-MSA(LN()) + +1 = MLP(LN(¯+1)) + ¯+1 where LN, MLP, W-MSA and SW-MSA represent layer normalization [16], multilayer perceptron, windowed multi-head self-attention and shifted window multi-head self-attention, respectively. The main diference between
attention block to merge the skip-connection of the encoder to the decoding input. The skip-connected input is used as the key and value parameters, while the decoding input form the query.
a multi-stage Video Swin transformer. Specifically, we use three stages, each having four 3D transformer blocks, followed by a patch merging layer [13, 15]. The attention layer used here is the multi-head self attention as explained in section (2.2) [15].
a cross attention layer for feature mixing. The skip connection from the encoder serves as the key parameter K, and value parameter V , while the continuing input from the patch expanding layer serves as the query parameter Q in the attention block (fig 1a). 2.8. Neck data presented except flipping. This is important as we train a single model with data from the three provided Although not considered in Fig (1a), our proposed model regions (R1, R2, R3), while we tested the model on all considered the possibility of linking the encoder to the available regions (R1-R6) to account for the transfer learndecoder via a neck block. When considered, We use two ing challenge.
Transformer blocks without a preceding patch merging As shown in Table (2), the proposed model resulted layer as the neck [14]. Table (1) shows some models in an MSE of 0.5337 and 0.4959, for the core and transfer where we either use a neck or not in our experiments. challenges, respectively, with only 688,080 parameters. Moreover, this is the first instance of using vision trans2.9. Prediction Head formers in spatiotemporal forecasting. As shown in Table This is used to project the output of the last decoder block (1) , we equally trained another model with an embed(final output) to the expected dimensions and format. We ding dimension of 32 but this could not be tested on the employed a patch expanding layer for possible recovering leaderboard due to restriction on number of model that of the spatial dimension that may have been updated by can be submitted. We considered the use of a neck in our the multi-stage encoder-decoder network. Finally, a fully UNet-like model resulting in a conclusion that this does connected layer is used to project the features into the not help our model to improve (Table 1). In Table (2) , ifnal dimension. we compared our model with the baseline models (persistence and Unet) provided on the leaderboard which shows that our model performs better with less than 700,000 3. Experimental Results parameters. In Tables (1), it is important to note that the validation MSE is computed on the validation set (Region Data Description : We used the datasets in the R1-R6 [17]) while the core and transfer MSE in Table (2) Weather4Cast2021 "stage-1" challenge for evaluation pur- are computed using a specially crafted MSE computation poses [17]. This competition has two challenges: which puts a segmentation mask into consideration [17]. • Core Challenge: data contain the training, validation, and test sets for three regions {R1 – Nile region (covering Cairo), R2 – Eastern Europe (covering Moscow), and R3 – South West Europe (covering Madrid and Barcelona)} • Transfer Challenge: data contain only the test set for three additional regions {R4 – Central Maghreb (Timimoun), R5 – South Mediterranean (covering Tripoli and Tunis), and R6 – Central
Europe (covering Berlin),} Weather parameters including temperature (on accessible surfaces: top cloud or earth), convective rainfall rate, probability of occurrence of tropopause folding, and cloud mask are selected as target variables for the competition. Each weather image contains 256 x 256 pixels, with each pixel corresponding to an area of about 4 km x 4 km. The images were recorded at 15 minute intervals throughout the year. Model Training : The model described in Fig. 1 was implemented in Pytorch. The mean squared error (MSE) was used as the loss function, with the Adam optimizer 4. Conclusions and Future Work [18]. The learning rate was initially set to 1e-4 and was manually reduced to 1e-7, when performance plateaued We presented the first ever use of 3D Swin-Transformer on the validation set. Our model was trained with care- in a UNet architecture for short time spatiotemporal forefully considered data augmentation for segmentation casting, which resulted in competitive results, i.e., an purposes. Specifically, we use RandomHorizontalFlip and MSE of 0.5337 and 0.4959, for the core and transfer chalRandomVerticalFlip, which ensures that we can leverage lenges (Weather4Cast2021 [17]), respectively, with only data augmentation without making any change in the 688,080 parameters. The model having only three blocks of four Swin-transformers in both encoder and decoder
was implemented in PyTorch and trained using PytorchLigthning [19]. As transformer architecture is still relatively new to vision domain, we plan to explore other variants of attention layers in the future. Likewise, we equally plan to explore token mixing using hypercomplex networks like sedenion [20]. The code with the implementation of the proposed approach is available online at https://github.com/bojesomo/Weather4cast2021SwinEncoderDecoder.