Open-set Plant Identi cation Using an Ensemble of Deep Convolutional Neural Networks

Abstract

Open-set recognition, a challenging problem in computer vision, is concerned with identi cation or veri cation tasks where queries may belong to unknown classes. This work describes a ne-grained plant identi cation system consisting of an ensemble of deep convolutional neural networks within an open-set identi cation framework. Two wellknown deep learning architectures of VGGNet and GoogLeNet, pretrained on the object recognition dataset of ILSVRC 2012, are netuned using the plant dataset of LifeCLEF 2015. Moreover, GoogLeNet is ne-tuned using plant and non-plant images for rejecting samples from non-plant classes. Our systems have been evaluated on the test dataset of PlantCLEF 2016 by the campaign organizers and our best proposed model has achieved an o cial score of 0.738 in terms of the mean average precision, while the best o cial score is 0.742.

Keywords

• open-set recognition
• plant identi cation
• deep learning
• convolutional neural networks

-

Automated plant identi cation is a ne-grained image classi cation problem concerned with small inter-class and large intra-class variations. As with many other problems, recent research in this area has concentrated on deep learning schemes in which signi cant improvements have been obtained compared to traditional methods [ 1,2,3,4 ]. Raw data is fed into these networks in multiple levels, allowing the system to automatically discover high-level features for plant species recognition. However, due to their high computational complexity for training from scratch, the existing pre-trained deep networks are ne-tuned for plant identi cation purposes [ 2,3,5 ].

The plant identi cation challenge of the Conference and Labs of the Evaluation Forum (CLEF) [ 6,7,8,9,10 ] is one of the most well-known annual events that benchmark content-based image retrieval from structured plant databases including photographs of leaves, branches, stems, owers, and fruits. The latest annotated plant dataset is provided by the LifeCLEF 2015 campaign with over 100,000 pictures of herbs, trees, and ferns belonging to 1,000 species collected from Western Europe.

Open-set recognition is a challenging task in computer vision which deals with identi cation or veri cation problems where samples from unknown classes could be presented to the system as well [ 11,12 ]. To create a similar scenario, the PlantCLEF 2016 campaign has provided a test image query which is very di erent from the previous years' CLEF datasets in nature [ 13,14 ]. This dataset is composed of unknown plant species and non-plant objects to create an openworld plant dataset. Therefore, the task challenges are not limited to automatically recognizing the known plant species, but also rejecting unknown plants and objects as well.

Our team participated in the PlantCLEF 2016 challenge under the name of SabanciUGebzeTU achieved the second rank with a very small di erence from the rst-ranked team. In our proposed systems, we ne-tune the pre-trained deep convolutional neural networks of GoogLeNet [ 15 ] and VGGNet [ 16 ] for plant identi cation using the LifeCLEF 2015 plant task datasets. We augment this data using di erent image transforms such as rotation, translation, re ection, and scaling to overcome over tting while training and to improve the performance during testing the system on highly noisy test data. The overall system is then composed of these two networks using score-level averaging. To enable rejections, we train another deep learning system to separate plants from nonplants. Speci cally, we ne-tuned GoogLeNet using the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) 2012 dataset as negative examples{after removing the potted plant category.

The rest of this paper is organized as follows. Section 2 describes the proposed methods based on ne-tuning of the GoogLeNet and VGGNet models for plant identi cation and open-set rejection, data augmentation, and classi er fusion. Section 3 is dedicated to the description of the utilized dataset and presentation of designed experiments and their results. The paper concludes in Section 4 with the summary and discussion of the utilized methods and obtained results. 2

Proposed Method

Our proposed method for automated plant identi cation is based on ne-tuning and fusing of two successful deep learning models, i.e. GoogLeNet [ 15 ] and VGGNet[ 16 ]. These models are, respectively, the rst-ranked and second-ranked architectures of the ILSVRC 2014{both trained on the ILSVRC 2012 dataset with 1.2 million labeled images of 1,000 object classes.

GoogLeNet [ 15 ] contains 57 convolutional layers, 14 pooling layers, and one fully-connected layer while VGGNet [ 16 ] involves 16 convolutional layers, ve pooling layers, and three fully-connected layers. Both networks take color image patches of size 224 224 pixels as the input and connect a linear layer with Softmax activation in the output.

We used the plant task datasets of LifeCLEF 2015 for ne-tuning the pretrained models. The training parameters consist of a weight decay of 0.0002, a base learning rate of 0.001, and a batch size of 20. Data augmentation is applied to decrease the chance of over tting during training and to improve performance while testing. For this purpose, we randomly extract K square patches from each original image around its center. The original image is also rotated by R degrees and the largest square image is cropped from the center of each rotated image. All the extracted patches as well as the original image (K + 3 images) are then scaled to 256 256 pixels and the mean image is subtracted from them. Finally, ve patches of size 224 224 pixels are extracted from four corners and the center of each image. These patches are then also re ected horizontally, resulting in 10 (K + 3) patches per input image in total.

Score-level averaging is applied to combine the prediction scores assigned to each class for all the augmented patches within a single network. Finally, the obtained scores from every deep network classi er are combined using the same averaging rule. The observation id that indicates di erent images of the same unique plant is not used in our systems; hence the recognition results are based only on single images.

For the unknown-class rejection task within the unseen data, we separately ne-tuned the GoogLeNet model for a binary classi cation problem, i.e. plants vs. non-plants. The training samples for this purpose consist of plant images from LifeCLEF 2015 and non-plant images from ILSVRC 2012. Using the obtained results from the binary classi er, we reject those images that are classi ed as non-plant and receive a low con dence score in the main plant identi cation system. 3

Experiments and Results

To train and validate our systems, we used the plant dataset of LifeCLEF 2015 [ 17 ] consisting of 113,204 images of di erent plant organs belonging to 1,000 species of trees, herbs, and ferns. We randomly divided the training portion of the dataset into two subsets for training and validation, with 70,904 and 20,854 images, respectively. The test portion of the dataset consists of a separate set of 21,446 images; however the ground-truth for the test dataset was only recently released and thus used in a limited fashion in our system. In the remainder of this paper, we will call these three subsets train, validation, and test subsets, respectively.

The PlantCLEF 2016 test dataset includes 8,000 samples submitted by the users of the mobile application Pl@ntNet [ 13 ]. This dataset is highly noisy and essentially di erent from the plant task datasets of LifeCLEF 2015 since it contains pictures of unknown plants and objects. We use the data augmentation approach explained in Section 2 and set K = 5 and R = 10 to train and test 80 patches per input image. In some experiments, we set K = 9 and R = f10; 20g to train and test 140 patches per image.

To ne-tune GoogLeNet for the open-set unknown-class rejection task, K is set to zero and no rotation is applied so as to augment data with only 10 patches per image. The training data for this problem is obtained by combining the earlier training subset of LifeCLEF 2015 for the plant class samples and an equal number of non-plant object samples from ILSVRC 2012, giving about 140,000 samples in total. In addition, the validation set is obtained from combining the earlier validation subset of LifeCLEF 2015 and an equal number of non-plant object samples of ILSVRC 2012, resulting in about 40,000 samples in total. Images used in the training and validation subsets contained distinct samples.

We implemented the GoogLeNet and VGGNet deep models using the Ca e deep learning framework [ 18 ] with pre-trained weights obtained from Ca e Model Zoo provided by the Berkeley Vision and Learning Center (BVLC). In the rest of this section, we explain our conducted experiments, their validation results, and the prepared runs.

It is worth mentioning that we have not used additional plant images for training any of the systems. Furthermore, all systems are fully automatic except for Run 3 where we manually remove 90 images in order to measure our performance in the case of a closed set problem (no query from unknown classes). Run 1. In this run, we rst ne-tuned the pre-trained models of GoogLeNet and VGGNet using the train subset of LifeCLEF 2015 that we augmented with 140 and 80 patches per image till 600,000 and 500,000 iterations, respectively. Next, we ne-tuned the pre-trained GoogLeNet with almost all of the data (train+test+validation/2) augmented with 80 patches per image until 200,000 iterations. We fused all the obtained scores for augmented image patches from di erent networks' classi ers. With this system, we achieved an accuracy of 79.80% on the given validation set (validation/2).

To reject test samples of unknown classes, we ne-tuned the pre-trained GoogLeNet with the obtained plant/non-plant dataset until 100,000 iterations and achieved an accuracy of almost 100% on the given validation set. Next, we tested the combined deep model on the non-plant validation set; from the obtained plant identi cation prediction scores it was inferred that these scores have a uniform-like distribution. We set the rejection threshold to T = 0:4 based on the top-1 score for maximal rejection of non-plant samples and minimum rejection of plant data. In the test stage, we utilized our combined system for score prediction as well as rejection of samples whose top-1 score was less than 0.4. We rejected 480 images from the PlantCLEF 2016 test dataset in this manner and obtained our rst run.

Run 2. All the implemented steps for achieving this run were the same as those performed for Run 1. The only di erence was that no image rejection was performed in obtaining this run.

Run 3. All the steps utilized in preparing this run were common with Run 1 except for the rejection method. In this experiment, we manually reviewed the test images and set aside 90 non-plant images. This was done to measure the system performance with only known classes; however, as this score is later provided by campaign organizers, this system is not very relevant. Test Results. We submitted the classi cation results of the aforementioned systems on the o cial test set of the PlantCLEF 2016. The utilized metric for evaluation was the mean average precision. In other words, all prediction scores were extracted for each class and sorted in the descending order to compute the average precision for that class. Finally, the mean was computed across all classes. The released results by the challenge organizers [ 14 ] are shown in Figure 1. As the o cials results indicate, we obtained the second place in the open-set plant identi cation competition of PlantCLEF 2016. Moreover, the uno cial results indicate that we achieved the top score in the image-based plant identi cation.

From the o cial results, we can conclude that the automatic rejection method (Run 1) was the top performer among our three systems followed by the manual rejection scheme (Run 3) and the method with no rejection (Run 2).

Comparing automatic rejection (Run 1) and no rejection (Run 2) systems, we see that the error in automatic rejection is very small when considering only known classes (0.806 vs 0.807), while the gains in the open-set and invasive species problems are slightly more signi cant (0.738 vs 0.736 and 0.704 vs 0.683, respectively). 4

Conclusions

In this paper, we reviewed the details of our proposed systems used in the openset plant identi cation problem of PlantCLEF 2016 campaign. We fused two powerful deep learning methods of VGGNet and GoogLeNet after ne-tuning them with the augmented plant task datasets of LifeCLEF 2015. Meanwhile, we used plant and non-plant images to ne-tune GoogLeNet for a binary classi cation to perform unknown-class rejection. Our systems were o cially evaluated on the test dataset of PlantCLEF 2016 and our best proposed model achieved a mean average precision of 0.738.

Our main focus in PlantCLEF 2016 was to see how we could improve the competitive results obtained in LifeCLEF 2015 [ 10 ]. To this end, we studied the best ways to ne-tune deep learning models; in particular, we experimented with the iteration size, batch size, and data augmentation. We observed that the accuracy steadily improved with the increased number of iterations and with data augmentation.

Acknowledgments. This work is supported by the Scienti c and Technological Research Council of Turkey (TUBITAK) under the grant number 113E499. Dr. Aptoula was a liated with Okan University when this work was conducted.

Fig. 1. The o cial released results of PlantCLEF 2016

References

1. Chen , Q. , Abedini , M. , Garnavi , R. , Liang , X. : IBM research Australia at LifeCLEF 2014: Plant identi cation task . In: CLEF (Working Notes) . ( 2014 )

2. Choi , S. : Plant identi cation with deep convolutional neural network: SNUMedinfo at LifeCLEF plant identi cation task 2015 . In: CLEF (Working Notes). ( 2015 )

3. Champ , J. , Lorieul , T. , Servajean , M. , Joly , A. : A comparative study of negrained classi cation methods in the context of the LifeCLEF plant identi cation challenge 2015 . In: CLEF (Working Notes). ( 2015 )

4. Ghazi , M.M. , Yanikoglu , B. , Atoula , E. , Muslu , O. , Ozdemir , M.C. : Sabanci-Okan system in LifeCLEF 2015 plant identi cation competition . In: CLEF (Working Notes) . ( 2015 )

5. Ge , Z. , McCool , C. , Sanderson , C. , Corke , P. : Content speci c feature learning for ne-grained plant classi cation . In: CLEF (Working Notes) . ( 2015 )

6. Goeau, H., Bonnet , P. , Joly , A. , Boujemaa , N. , Barthelemy , D. , Molino , J.F. , Birnbaum , P. , Mouysset , E. , Picard , M.: The CLEF 2011 plant images classi cation task . In: CLEF (Notebook Papers/Labs/Workshop). ( 2011 )

7. Goeau, H., Bonnet , P. , Joly , A. , Yahiaoui , I. , Barthelemy , D. , Boujemaa , N. , Molino , J.F. : The ImageCLEF 2012 plant identi cation task . In: CLEF (Online Working Notes/Labs/Workshop). ( 2012 )

8. Goeau, H., Bonnet , P. , Joly , A. , Bakic , V. , Barthelemy , D. , Boujemaa , N. , Molino , J.F. : The ImageCLEF 2013 plant identi cation task . In: CLEF (Working Notes) . ( 2013 )

9. Goeau, H., Joly , A. , Bonnet , P. , Selmi , S. , Molino , J.F. , Barthelemy , D. , Boujemaa , N.: LifeCLEF plant identi cation task 2014 . In: CLEF (Working Notes). ( 2014 )

10. Goeau, H., Bonnet , P. , Joly , A. : LifeCLEF plant identi cation task 2015 . In: CLEF (Working Notes). ( 2015 )

11. Scheirer , W.J. , Jain , L.P. , Boult , T.E.: Probability models for open set recognition . IEEE Transactions on Pattern Analysis and Machine Intelligence 36 ( 11 ) ( 2014 ) 2317 { 2324

12. Bendale , A. , Boult , T. : Towards open world recognition . In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . ( 2015 ) 1893 { 1902

13. Joly , A. , Goeau, H., Glotin , H. , Spampinato , C. , Bonnet , P. , Vellinga , W.P. , Champ , J. , Planque , R. , Palazzo , S. , Muller, H.: LifeCLEF 2016: multimedia life species identi cation challenges . In: Proceedings of CLEF 2016 . ( 2016 )

14. Goeau, H., Bonnet , P. , Joly , A. : Plant identi cation in an open-world (LifeCLEF 2016) . In: CLEF (Working Notes) . ( 2016 )

15. Szegedy , C. , Liu , W. , Jia , Y. , Sermanet , P. , Reed , S. , Anguelov , D. , Erhan , D. , Vanhoucke , V. , Rabinovich , A. : Going deeper with convolutions . In: IEEE Conference on Computer Vision and Pattern Recognition . ( 2015 )

16. Simonyan , K. , Zisserman , A. : Very deep convolutional networks for large-scale image recognition . Computing Research Repository (CoRR) ( 2014 ) arXiv: 1409 . 1556 .

17. Joly , A. , Goeau, H., Spampinato , C. , Bonnet , P. , Vellinga , W.P. , Planque , R. , Rauber , A. , Palazzo , S. , Fisher, B. , Muller, H.: LifeCLEF 2015: multimedia life species identi cation challenges . In: Experimental IR Meets Multilinguality , Multimodality, and Interaction. ( 2015 ) 462 { 483

18. Jia , Y. , Shelhamer , E. , Donahue , J. , Karayev , S. , Long , J. , Girshick , R. , Guadarrama , S. , Darrell , T.: Ca e: Convolutional architecture for fast feature embedding . In: Proceedings of the 22nd ACM International Conference on Multimedia. ( 2014 ) 675 { 678