In this paper, we propose an automatic approach for plant image identification. We enhanced the well-known VGG 16-layers Convolutional Neural Network model [1] by replacing the last pooling layer with a Spatial Pyramid Pooling layer [2]. Rectified Linear Units (ReLU) are also replaced with Parametric ReLUs [3]. The enhanced model is trained without external dataset. A post processing method is also proposed to reject irrelevant samples. We further improved identification performance using observation identity (ObservationId) provided in the dataset. Our methods showed outstanding performance in official evaluation results of the LifeCLEF 2016 Plant Identification Task.
Nowadays, conservation of biodiversity is becoming an important duty for us. To
achieve this, accurate knowledge is essential. However, even for professionals,
identifying a species can be a very difficult task. Convolutional Neural Networks (CNNs)
are leading the best performance in various image retrieval tasks such as the
ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [
In recent years, CNN has gained popularity in various image retrieval tasks
including the LifeCLEF Plant Identification Task (PlantCLEF). Over the past years,
participants of PlantCLEF have adopted to use CNN in their works [
Considering the new setting, we propose a post processing method to reject these samples. In the next section we propose to enhance the VGG 16-layers model with Spatial Pyramid Pooling and Parametric Rectified Linear Units. We detail our model training strategy in Section 3, followed by irrelevant sample rejection algorithm in Section 4. Section 5 shows evaluation results and Section 6 concludes this paper. 2
In the following sections, a few enhancements to the VGG 16-layers model will be described. 2.1
Spatial Pyramid Pooling The original VGG 16-layers model requires input image of spatial size 224 × 224. On the other hand, the images in the PlantCLEF 2016 dataset are arbitrarily sized, as shown in Figure 1. Following the spatial size restriction in VGG 16-layers model, input images has to be either cropped or warped. Both of these methods may lead to information loss.
vertical image square image
horizontal image
0.50
0.56
0.63
0.71
0.83
1.00
1.20
1.40
1.60
1.80
2.00
To circumvent such restriction, we have replaced the last pooling layer (pool5) with a
Spatial Pyramid Pooling (SPP) layer [
The following strategies are used to train the enhanced model. 3.1
Data preparation The PlantCLEF 2016 dataset consists of 113204 images. 2048 images are used for validation purpose, while the remaining 111156 images are used to train the model.
We have augmented the training images as follows: while preserving aspect ratio, images are resized such that the shorter side becomes 224 and 336. Random cropping to 224 × 224 and random flipping by y-axis are also applied to the resized images. For evaluation process, images are resized such that the shorter side becomes 224, and no cropping or flipping is applied. Cropping is not required during evaluation as the enhanced model accepts image of any size. 3.2
Image Mean As the enhanced model is trained from scratch i.e. without external resources, image mean is computed from the training set. Excluding LeafScan images (with high RGB values due to bright background), the computed mean values of red, green and blue channels are 105, 111, and 79 respectively. Images augmented in Section 3.1 are subtracted with these mean values before being used for training. 3.3
Training
We use the Caffe framework to train the enhanced model. To train such a deep model,
we use Xavier’s method [
F(α)
F(β) 1 2 3 class 4
F(γ) 1 2 3 class 4 1 2 3 class 4 Based on the scenario above, we realized that there are cases that a sample should be rejected by all of the binary classifiers. To identify such irrelevant sample, an algorithm is elaborated in the next section. 4.2
Irrelevant Sample Rejection Algorithm After training the enhanced model as mentioned in Section 3, a matrix of raw (i.e. before softmax normalization) scores × of classes and training samples are extracted from the classifier layer (fc8). Incorrect predictions are omitted. With only correctly predicted class scores ′× ( ′ ≤ ), rejection threshold = [ 1, … , ] of each class is computed from the class wise minima, as shown in (2). = ′ min , (2)
Fig. 5. Rejection threshold of the training set, sorted in ascending order 30 25 20 15 10 5 0 threshold class (sorted) During evaluation, any sample with score lower than for all of the classes will be rejected as irrelevant. With the evaluation set, 195 out of 8000 images are rejected. Some of the images are shown in Figure 6. ing training set thresholds are computed, and then averaged into . As detailed in (3), the denominators are the thresholds of the training set, while the numerators are of the validation set. The values in (3) are based on Figure 7.
= is then multiplied to the threshold of the training set, as shown in (4), before using it to reject samples during evaluation.
′ = (4) Applying ′ to the evaluation set, only 69 samples are rejected as it is lower compared to the original . Some of the images are shown in Figure 8. Identification based on a single image may be insufficient. In the PlantCLEF2016 dataset, images based on the same observation share a unique ObservationId. One ObservationId may be assigned with multiple images of different organs. Specifically images of flower or fruit which have much characteristic features, their existence in an observation often improves identification performance.
To further improve the identification performance, after rejecting samples as explained in Section 4.2 and 4.3, we sum the raw (i.e. before softmax normalization) class scores of images with the same ObservationId. The summed scores are then softmax normalized. As the result, images with the same ObservationId share the same normalized scores. 5
In our evaluation experiments, the enhanced CNN model is used to extract class scores. Four different post processing methods are then applied to the extracted class scores, detailed as follows. As mentioned in Section 4.2 and 4.3, is the rejection threshold obtained from training set, while ′ is the rejection threshold obtained by considering both training and validation sets.
─ Run 1: Sample rejection with , identification based on single sample ─ Run 2: Sample rejection with ′, identification based on single sample ─ Run 3: Sample rejection with , observation based identification ─ Run 4: Sample rejection with ′, observation based identification In our run files (Bluefield), we provide scores up to top 30 classes, and rejected samples are entirely excluded from our run files. Evaluation results compared with other participants are summarized in Table 1 and Figure 9. Bluefield Run 4 yields the highest official score among all of the participants. Usage of ObservationId shows significant improvement. On the other hand, rejection threshold ′ that takes validation set into account shows a slight improvement in official MAP. 6
In this paper, we described our approach to PlantCLEF 2016, focusing on model enhancements, data augmentations, and an irrelevant sample rejection strategy. We still leave some rooms for improvements, which are itemized as follows: As mentioned in Section 3.1, images for training process are resized to two scales.
We should consider applying random scaling instead of a constant number of (two) scales. Other than scaling, random rotation should be applied as well. In Section 4.2 and 4.3, instead of taking minima as rejection threshold, we should consider using mean and standard deviation to obtain a more stable threshold. The PlantCLEF 2016 dataset includes rich metadata information such as Genus, Family, Date, Longitude, Latitude, Location and Content (organ type). However, in our work, only ClassId (class label) and ObservationId are used. More metadata information should be utilized to obtain more accurate identification performance. We would like to thank MEXT KAKENHI, Grant-in-Aid for Challenging Exploratory Research, Grant Number 15K12027 for partial support of our work.