In the MediaEval 2017 Multimedia Satellite Task, we propose an approach based on regression random forest which can extract valuable information from a few images and their corresponding metadata. The experimental results show that when processing social media images, the proposed method can be high-performance in circumstances where the images features are low-level and the training samples are relatively small of number. Additionally, when the low-level color features of satellite images are too ambiguous to analyze, random forest is also a e ective way to detect ooding area.
The outburst of social media provides us with an opportunity to deal with speci c tasks, e.g., disaster prediction and speci c scene identi cation. Such problems can be crucial in agriculture, urbanization and environment monitoring. The MediaEval 2017 Multimedia Satellite Task consists of two subtasks: Disaster Image Retrieval from Social Media (DIRSM) task and Flood-Detection in Satellite Images(FDSI) task. The former one requires the prediction system to identify ooding circumstances in social media pictures, while the latter task aims to judge that which district in a certain area of a satellite image is su ering from ooding.
As to the theoretical basis of the task, existing work such as [
In this paper, we propose to employ regression random forests
to rank the relevance of ooding in both social media images taken
by cameras and satellite images describing the overall situation of
a certain district. It is shown that our method can well balance the
e ciency and e ectivity. In other words, our method achieves
compatible performance with extraordinary short time-consuming. More
speci cally, we design two prediction systems based on
regression random forest method, which has been proven e ective in
handling high-dimensional data and preventing over tting when
training set is comparatively small [
The goal of DIRSM subtask is to retrieve all images which show
direct evidence of a ooding event from social media streams. The
details of this subtask are described in [
2.1.1 Feature Extraction. In recent years, Convolutional Neural
Networks (CNNs) has been dominating in the eld of computer
vision, such as recognition and detection. Therefore, except for the
baseline features provided by the organizers, we also extract robust
CNN feature to improve the performance of our system. Speci
cally, we apply ResNet-152 [
2.1.2 Models Definition. In order to improve the performance of our system, the learning algorithm we choose must satisfy several requirements: (a) it should remain high-performance under circumstances where data is restricted; (b) it should excel in accuracy among current algorithms; (c) it can handle thousands of input variables without variable deletion; (d) its speed should be high enough. Due to the consideration above, we use regression random forest for the ranking of the 5 runs.
As an important application in ensemble methods, random
forest is a high-performance method both for classi cation and
regression. When the number of training set images is not large enough
to utilize other learning methods (e.g., deep learning), using
random forest can prevent over tting and unbalance of features in
datasets [
Theoretically, the results will improve with the number of trees in a random forest increasing. However, since the computation cost increases as well as the advancement decreases when new trees are added to a larger forest. This phenomenon indicates that the number of trees should be limited.
As for the details of the parameters setting, we set the bagging
percent of the forest as 0.9 [
The aim of the FDSI subtask is to develop a model that is able to
identify regions in satellite imagery which are a ected by a
ooding. Same as the DIRSM subtask, the details can be nd in [
The same as before, we use ResNet and ca e to extract the features of the satellite images. We still use the 2048-dim vectors of bottom conv layer in ResNet. Meanwhile, we utilize random forest to process the images in the development set due to the same reason, which is that the number of features to learn in the satellite images is small and the development set is even smaller than the DIRSM subtask. We set the bagging percent of the forest at 0.8, the minimal leaf size as 20, and the number of trees at 400. During training, we use the rst four development set folders as training set, the other two as validation set. 3 3.1
Because of di erent requirements in the 5 runs, we train features with slightly di erent setting. In run 1, 4 and 5, we only utilize the images features in training. To augment the dataset, we randomly crop and ip horizontally the original images in the development set, and obtain 5320 more images for run 4 and 5 additionally. In run 2, we utilize the text given in associated metadata of the development set. We process each word in every image description as GloVe vector, the dimension of each vector 300, and constrict each sentence to its maximum length to generate a matrix including all the sentences. In run 3, we use both the text and the development set images to train the random forest.
The mean average precision (mAP) scores we get from the 5 runs is shown in Table 1. The mAP scores listed are the mean of average precision at the top 50, 100, 200, 300, 400 and 500 rankings of each run.
Experimental results of DIRSM are shown in Table 1. As we can see, metadata only (run 2) perform much worse than visual information only model (run 1), which indicates that associated descriptions are much noisy than visual information for ooding prediction. Intuitively, more feature bring more information, and gain better performance. However, run 3 (combination of visual and textual feature) performs a little worse than run 1. This also demonstrates that textual description introduces much noise, even induce to decrease the performance of original images. 3.2
For the run 1, 2 and 3, we only utilize the original satellite images and their ground truth masks in training. For run 4 and 5, we use the cropped satellite images and horizontally ipped images as well.
Table 2 exhibits the intersection of union (IoU) of experimental results. the best performances lie in the location 05 and the new location provided by the task organizers. Possible reasons may be that the number of the images in these location is relatively small, reducing the possibility of over tting. Besides, the mean performance of the last 2 runs is better than the rst 3 ones, claiming that the general run cast the better results. It is possible that the performance in the rst 3 ones su er from the variance of more images, but generally the results are at large satisfying. 4
In this paper, we illustrated our approach for the MediaEval 2017 Multimedia Satellite Task. In both subtasks, combining random forests and CNN features enhanced the performance of the detection. In the DIRSM subtask, combining the features learnt from text and images improved the regression performance of labeling, but our methods still su er from noise. As to FDSI subtask, the performance of the proposed method could be better when the number of test images fewer. The best result remained in the location 05 and the new location. Overall, the proposed method got promising performance in processing both social media stream and satellite images.