In this paper we use a novel backpropagation technique, Direct Backpropagation (DBP), to train a neural network and use it to detect flooding in Twitter posts. We use the textual information from the tweets and the visual features from the associated images to classify the posts into two categories, flood (1) and no-flood (0). We also fuse these two modes using fusion methods for the classification. For the classification task we employ a neural network that we train using our proposed method instead of typical backpropagation method. This work has been done in the context of the MediaEval 2020 Flood-Related Multimedia Task.
Satellite images have been used for flood detection for decades
[
The results show that among all of our runs, feature-level fusion of textual information performs best and achieves the highest FScore of 0.41, which was around 6% higher than the average of all participants. Our visual information only run also achieves better than average F-Score. However, the fusion of visual and textual information, both feature-level and decision fusion performs poorly compared to the average.
For the textual approach we adapt the Bag-of-Words (BoW) method for text feature extraction. The textual data were provided as in full text and hashtags. However, the full texts, called the description, also contained the hashtags in them. Therefore, we removed the keywords from them and collected the keywords separately. We also remove the URLs, any punctuation, usernames and alphanumeric symbols from the full text. The keywords of a tweet, called tags, 2.2
For the visual features we employed Xception[
We used the described textual and visual features for the
classiifcation task, where we employed a neural network. This neural
network uses direct backpropagation (DBP) technique instead of
typical backporpagation (BP) method. This technique sends the
cost function calculated in the last layer back to every layer to
calculate their gradients which removes the dependency on previous
layer in the learning process and hence reduces the processing time
and cost. The direct backpropagation method is described more
elaborately in [
Equation (1) and (2) show the hidden layer update direction for BP, and DBP respectively. Here, considering a three layered network, δa2 and δa1 are the second and first layer gradients respectively, f ′() is the derivative of the non-linearity, ⊙ is an elementwise multiplication operator, B are random feedbacks, e is the gradient at the last layer, known as the cost function, W are the forward weights and W T are the symmetric weights. Our proposed learning method can be presented by equation (2).
δa2 = (W3T e) ⊙ f ′(a2), δa1 = (W2T δa2) ⊙ f ′(a1) δa2 = (W3T e) ⊙ f ′(a2), δa1 = (W2T e) ⊙ f ′(a1) (1) (2) 2.4
Here, we used two types of fusion approaches, feature-level fusion and decision fusion. In feature-level fusion, the features from different modes, such as visual or textual are concatenated together and used as input feature of the neural network. In decision fusion, the probability values of relevancy to flooding generated by each mode separately are averaged together to calculate the relevancy of a post.
As instructed in the task, run 1 uses fusion of textual and visual data. We used a feature-level fusion of description, tags and Xception for this run.
Run 2 uses textual information only. However, we also implemented a fusion method by adding a feature-level fusion of description and tags.
This is a visual information only run. We used visual features extracted by Xception for this run.
In this run we took textual information only similar to run 2. However, here the description and the tags were fused in a decision fusion manner.
Run 5 is also a decision fusion run. Here we used description, tags, Xception features in decision fusion manner.
The neural network was trained with a 10-fold cross validation for every experiment. 3.6
Table 1 shows the F-score results generated by the organizers on our submitted runs and the average of all participants. It shows that our textual only, visual only and fusion runs perform better than average. However, the visual features perform poorly compared to the textual information, both in our submission and in average. This is because of the nature of the dataset. As described in the task description, the tweets were retrieved by using the keywords as 3 3.1
3.3
Run 3 3.4
Run 4 3.5
Run 5 search criteria and then the tweets were annotated, not the associated images. Manual inspection of the associated images showed that many of the flood relevant tweets did not contain any flooding evidence in the images. Our results and the average scores also show that fusion of the visual information with textual information degraded the result compared to textual only, which also identifies the visual information as noise. We have also inspected the weights generated by the neural network for each features. To evaluate each mode’s credibility we inspected the weight values generated by the neural network for each feature and presented in Figure 2. Here, we used four modes namely, description, tags, Xception and InceptionV3. It shows that the visual features have average of negative weights, that means they have a negative impact in the classification process. InceptionV3 had the lowest weight values with an average of -0.02 and therefore, performed very poorly when it was used for the classification. That is why we did not include it in any of the submitted runs. 4
We illustrated our approaches for the MediaEval 2020 Flood-Related Multimedia Task. Our approaches contained five runs, where we used the textual information and the visual information separately and also in fused manner. The average results of all participants were not promising (highest average F-score was 0.3571). The highest F-score we achieved was 0.4158, using textual information only, where we fused the tweet text and the keywords. The visual features performed poorly and degraded the overall performance when fused with textual information.