In our working notes we discuss di erent machine learning approaches for the prediction of species for di erent geolocations based on environmental variables. We used convolutional neural networks for image classi cation as well as data extraction with tree boosting as models. Furthermore, we analyzed di erent approaches for data extraction, as well as data augmentation and group forming to improve our models performance.
2.1
Since the dataset was given in two versions - images and a .csv- le containing single values - we planned to develop approaches for both types. After some analysis of the .csv- le, we noticed, a large amount of cells without values. These cells render their respective rows unusable. To ll out the missing values, we rst tried to calculate them by averaging over the 4 centered pixels of the respective images, since the single values are supposed to be the EV values at the exact location of the occurrence. We also did this calculation for the images, for which values existed in the .csv- le, but realized that it was not possible to reproduce these values. Because of this discrepancy, we decided very early to not use the given single values from the .csv- le, but rather to create our own .csv- le by extracting the single values from the images like mentioned above.
We also used another variation of this extracting algorithm, were we averaged the single values over the whole image instead of only using the 4 centered pixels. Both variations were used in di erent submissions we made. We did not transfer the values of each EV into their respective ranges, but rather kept them at the range from 0 to 255, like in the images. 2.2
After this pre-processing step, we encountered a lot of raw numbers associated with di erent classes. They were not intuitively understandable, since the number of classes was also very high. For this reason we tried to create a ngerprint for each class. We did this by counting the single values for each class and creating a diagram, which shows how often each value occurs in every class. This way we gained some insight which values are discriminative for a speci c class. 2.3
The relatively high number of 33 di erent channels indicated that not all channels are equally useful. We decided it would be worth to cut some of them, because they are not su ciently discriminative. To achieve this, we searched for combinations of channels based on the extracted single values, that were sufcient to fully separate the training set as good as possible. We came across multiple combinations, one of which was used later when training a multi-model CNN. 2.4
Since the TIFF images of the dataset are no photographic images, it was hard to grasp the features that identify a speci c class. In order to get a better understanding of what characterizes di erent classes, we created an heatmap for each class, which contains the average values of all images for that classes. Furthermore, all values were normalized to the range between 0 and 1.
However, the heatmaps for the most common classes looked very similar, which further proved our assumption that the given data is not distinctive enough to predict classes properly. 2.5
We used 10% of the training set, to create a separate validation set. We used this split to estimate, how well our models and di erent approaches performed and to generate intermediate checkpoints for some of our models. We have ensured, that the validation set contained no species which were not included in the remaining train set. 3
In this section, we will comment on our submissions made during the challenge. For each submission, we will describe the approach we used and we will discuss why some approaches produce better results then others. 3.1
The most basic model we used was based on random guessing. We extracted all 3336 possible species and assigned 100 of them randomly to each occurrence of the test set so that every occurrence has a set of 100 di erent species. The score of this submission (ST 2) was 0:0016. 3.2
For the probability model, we rst counted the occurrences of each species in the training set. Then we assigned the species with descending probabilities to each test occurrence (so every occurrence had the same prediction). That means that the most common species was at rank one, the second most common species at rank two and so on. The score of this submission (ST 7) was 0:0134. 3.3
The vector model calculated the di erences between the occurrences of the training set and the test set and predicted each test occurrence with the species of the training occurrence with the smallest di erence. Therefore we took all 33 features and shaped a 33-dimensional vector V out of each occurrence. We calculated the di erence D as followed: # Vn = Vector of n-th training occurrence # Vm = Vector of m-th test occurrence # # # Vnm = Vn Vm
#
Dnm = jjVnmjj2 Then we sorted all Dnm values in descending order and looked which species Snm were assigned. As the species occurred multiple times for each m we always kept the rst occurrence in Snm. The result was a sorted list of 3336 species for each occurrence m. At the end we took the rst 100 species of the lists and ranked them ascending for the nal submission. The score of this submission (ST 5) was 0:0271. 3.4
When using the TIFF-images directly for training, we decided to use
convolutional neural networks (CNN). In this section we will discuss the di erent
approaches we tried during the challenge. We used the Keras framework [
VGG-like-models The rst approach was a simple feed forward CNN, similar
to the popular VGG model [
The exact architecture we used consisted of ve convolutional layers followed by a global average pooling layer and the classi cation layer. We implemented max pooling after the rst two and every second convolutional layer, which divides the spatial dimensions of the image by half. Additionally, we doubled the number of lters in the convolutional layers after each pooling layer .
During training we also recognized, that our model was su ering from severe over tting. We decided to use regularization in form of L2-normalization in the convolutional layers and dropout after each global pooling layer.
We also used two approaches for this architecture. One model that is trained with all 33 channels of the image and one ensemble, which consists of 6 individually trained models, each one using one of 6 channels we assumed to be the most discriminative ones (1, 5, 13, 14, 21 and 22). These channels were found by using the method we described earlier (2.3).
To further improve our models, we tested classic a ne transformations as data augmentation on the images that were used for training. First, we only used random ipping (vertical or horizontal) and random rotations. We later added random crops to the range of possible augmentations, to see if they would lead to further improvements. When doing the crops we selected a random area inside the image. Since the crops are random the all have di erent sizes, so we resized them to the original 64x64 pixels.
DenseNet We also included the more complex DenseNet [
We noticed that the result had very high uctuations and that it was not clearly visible which augmentation methods did provide an improvement, when looking at the test set scores. For the single model, only ips and rotations seemed to provide an improvement and the ensemble seemed to work best without any augmentations. The crops did not lead to any improvement and the DenseNet is about as good as the VGG-like single model.
However, these scores are the result of the o cial test set evaluation. When validating on our separate validation set, we generally found all augmentations to be helpful. 3.5
One other approach besides neuronal networks was the usage of Extreme
Gradient Boosting (XGBoost [
Single Model The rst strategy was to build one single model for the prediction. In this case the target variable mentioned earlier was a list of species probabilities for each dataset occurrence. The evaluation method we choose during the training was the multi-class classi cation error rate (merror-metric in XGBoost). We submitted the variations at Table 2. Multi-Model The second strategy was to predict each species separately. Therefore we created 3336 di erent models. In this case the target variable mentioned earlier was a list of probabilities for the occurrence of the current selected single species (one out of 3336) for a dataset occurrence. Each model was evaluated during the training with the log-loss metric because the resulting probabilities were decimal numbers. We tried out di erent parameter settings, which can be seen at Table 3. Groups Model To get better scores, we created an advanced model which built several groups with species that are similar. The idea was to group species, because we argued that it is likely that more than one species occur at one location. In order to nd similar species, we looked at the occurrences for each species in the train set and calculated a representative value per species. One way of doing so was to look at each feature column and estimate which value occurred most. The second method was to calculate the mean value for each feature column. This way, we calculated a simple ngerprint for each species.
In the next step we calculate the di erence between all species with the same method like in the Vector Model. The result was a 3336x3336 matrix in which the distance between each species was held. We had to round the feature values to 2 digits and we ignored species which occurred less then a count of o.
After that, we de ned the parameter t for threshold which indicates how great the distance between two species could be to consider them in the same group. After this, we created a network in which all species relations were contained. The following gure illustrates some of the problems that arise with that approach.
Therefore, we de ned a additional parameter min edges which de nes how much neighbours a species needs to be in a group. As result we got less groups but instead ones with more similar species.
At both runs we used all 64x64 pixels. For the test submission we rst predicted the groups for each test occurrence and then we mapped them back to the species. Therefor we took all species of the speci c groups and sorted them descending with regard of their probability in the train set.
At variation 1 we have got 2148 groups in which were 266 species in groups with an size greater than one. We then trained each group with 2148 models. At variation 2 we got 3301 groups and 39 species in groups with a size greater one.
We realized that it was not an improvement to build groups, because the test scores were not as good as the scores without groups. The reason could be that the machine learning algorithm detected by itself which species are similar and it therefore was more accurate then building the groups manually. 4
We published our source code at GitHub3. The code is written in Python and uses di erent third party libraries. The project page provides instructions on how to execute the code. We used an i7 INTEL CPU and a NVIDIA P6000 for model training. 3 https://github.com/stefantaubert/lifeclef-geo-2018
As the submissions of other participants and our own results show, predicting plant species based on environmental variables can be a challenging task. Furthermore, it is especially challenging when it comes to predicting only a single species, since there are many species that may occur under the same environmental conditions. However, we evaluated various machine learning techniques to get an idea of which work best for this problem.
Our approaches were divided into two main groups, one using the 33-channel images and CNN, the other one using single values, that were extracted from these images. For the CNN part, we mainly used a traditional feed forward network in two variations: One using all 33 channels of the images, the second was an ensemble of 6 individually trained models, each using one of 6 selected channels. After using the unprocessed versions of the images, we also added di erent a ne transformations as data augmentation. On the test score, augmentations did not lead to an improvement, but when evaluating them on our local validation set they had proven to be useful. In general, at their best, our CNN were merely better then a probability based approach, which implies that more optimization would be needed to make this technique work properly.
Our single value approach used boosted trees with the XGBoost framework. The rst parameter we tried to optimize with this approach was nding the right depth for our trees. We started with a very high depth, but we found, that less depth and therefore less complex models yield better results. After that, we also changed the way in which we extract the single values from the images. We were able to show that averaging only over the 4 centered pixels leads to better results than averaging over the whole image. Finally, we also tried out forming groups of similar species and predicting them instead of single species. We de ned predicting a group as the prediction of all species within this group ordered along their frequency in the train set. This data processing step did not lead to an improvement of our score.
In general we have to state, that our local validation score constantly di ered from the challenges test score, which led us to the conclusion that the species of the tasks dataset are not completely distinguishable using the given EV.