Automatic plant identification addresses the identification of the plant species in a given photograph. Plant identification challenge within the Conference and Labs of the Evaluation Forum (CLEF) [1,2,3,4,5,6,7] is the most well-known annual event that benchmarks content-based image retrieval of plants. The campaign has been run since 2011, with plant species and number of training images almost doubling every year, reaching to 10,000 classes in the 2017 evaluation. Considering very high similarities between species and a large variety of imaging and plant conditions, the problem is rather challenging.
Our team participated in the PlantCLEF 2017 campaign under the name of SabanciU-GebzeTU. In all of our runs, we used an ensemble of 4-9 convolutional networks, with different classifier combination criteria. The base networks were pre-trained deep convolutional neural networks of GoogLeNet [8] and VGGNet [9] that were fine-tuned with plant images. The campaign organizers provided two separate data sets: the main training set consisted of 256,203 images with clean labels (collected from the Encyclopedia of Life (EOL)) and the web crawled data consisted of around 1.6 million of images with noisy labels. The test set was sequestered until a few weeks before results submission. Details of the campaign can be found in [1].
The rest of this paper is organized as follows. Section 2 describes our approach based on: fine-tuning GoogLeNet and VGGNet models for plant identification and applying score-level classifier fusion. Section 3 describes the data sets and experimental results. The paper concludes in Section 4 with the summary and discussion of the utilized methods and obtained results.
Our approach was fine-tuning and fusing of two successful deep learning models, i.e. GoogLeNet [8] and VGGNet [9], using the implementations provided in the Caffe deep learning framework [10]. These models are, respectively, the first-ranked and second-ranked architectures of the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) 2014-both trained on the ILSVRC 2012 dataset with 1.2 million labeled images of 1,000 object classes.
In this work, we fine-tuned the GoogLeNet and VGGNet models starting from the learned weights of our PlantCLEF2016 system [11]. In the first network, we used only the training portion of EOL with internal augmentation (during training at each iteration a random crop of the image is used and randomly mirrored horizontally), to get some quick results. This network was the VGGNet architecture with all but the last layer of weights being fixed. In fact, in all of the experiments, we could only fine-tune the last 1-2 layers, as learning was very slow otherwise. This network achieved 41% accuracy.
After getting the base system running, we started using 8-fold external augmentation for training and later we started to incorporate images from the noisy data set into the training data: as the web crawled data is not reliable, we tested 200,000 images from the noisy data set using the best networks we had thus far and took only those images for which prediction matched the groundtruth.
We also tried VGGNET using Batch Normalization and GoogleNet architecture, with roughly similar performance. In both of these networks, all of the layers were fixed except for the last one due to scarce computing resources. Another network concentrated on the most common 1000 species and while we found that this network only achieved a 27% accuracy, it helped improve the performance of the ensemble like all other networks. In this fashion, each successive network (for a total of 9 different ones) was trained for either more iterations, or with new data added, or with different network architecture. At last, we trained one of the previous networks with all available training data, merging the validation set to the training set. This was done for only one network given the limited time.
Score-level averaging is applied to combine the prediction scores assigned to each of the augmented patches within a single network. As for the final systems, the obtained scores from all networks are combined using Borda count [12] or based on the maximum score of different classifiers. Our main problem was computational resources, faced with a very large number of classes and large amount of data. Only 60,000 images from the noisy data set were verified (to check for prediction and label match) and added to the training set. All trains and tests were run on a Linux system with a Tesla K40c and 12GB of video memory and in most cases training a network took 2-3 days.
For training and validating our system, we used the EOL data consisting of 256,203 images of different plant organs, belonging to 10,000 species. Specifically, we randomly divided the training portion of the dataset into two subsets for training and validation, with 174,280 and 81,923 images respectively. The test portion of the dataset consists of a separate set of 25,170 images that was sequestered by the organizers, until the last weeks of the campaign. We will call these three subsets train, validation and test subsets respectively in the remainder of this paper.
The base accuracy of the networks trained with all of the 10,000 classes ranged from 41% to 48.4% and the combined accuracy was 61.03%, on the validation subset. The combination was helpful even with highly correlated networks and taking less successful networks from the ensemble always reduced the performance The most successful individual network, based on the accuracy of the validation set, was the VGGNet using the largest training set (the train subset and around 60,000 samples from noisy data) and with a large batch size (60).
The submitted runs are described below and the results (mean inverse rank) released by the campaign organizers are given in [1] and shown in Figure 1. Detailed scores and ranking of the best runs from the top teams are shown in Table 1. -Run 3. This system was the same as System 4 except for using a combination based on maximum confidence. -Run 4. This system was the same as System 1 except for classifier combination weights.
The main objective was to preserve the high scores we obtained in 2016, despite the 10-fold increase in the number of classes [11]. Unfortunately, the large number of classes and limited computational power made it impossible to successfully fine-tune the networks or use most of the images from the noisy data set. While our results were significantly below the best performing system this year, our results are not too far from our results last year, despite 10-fold increase in the number of classes. It was overall a challenging exercise to deal with a large real life problem.