MICA at LifeCLEF 2015: Multi-organ Plant Identi cation

Abstract

In this paper, we describe the system including image preprocessing, feature descriptor extraction, classi cation method and fusion techniques that we applied for LifeCLEF 2015 - multi-organ plant identi cation task. In the preprocessing step, we apply relevant preprocessing techniques for each type of plants' organs based on the characteristic of the organs. For the feature descriptor, we propose to use kernel descriptor (KDES) with di erent types of kernel for all organs types. For ower and entire images, we combine KDES with HSV histogram. At the image level, we apply Support Vector Machine (SVM) as a classi cation method. Finally, we investigate di erent late fusion techniques in order to build the retrieved observation list.

-

Plant identi cation is a process that aims at matching a given specimen plant to a known taxon. This is a di cult and time consuming task even for the botanist experts. Recently, with the advanced research in computer vision community, a number of works have been proposed for plant identi cation based on images. However, most of these methods have been dedicated to one sole organ of the plants. The most widely used organs are leaf and ower. Since 2014, with the availability of dataset for multi-organ plant identi cation, the plant identi cation moves from image-centered to observation-centered [ 1 ]. The plant identi cation can be de ned as a image retrieval problem with the input is a set of organ images of a query plant and the output is the ranked list of retrieved plants. In this paper, we present our work dedicated to LifeCLEF 2015 - multi-organ plant identi cation task [ 2 ]. Our previous work for leaf-based plant identi cation has shown that the KDES is a robust descriptor for leaf representation [ 3 ]. Therefore, in this work, in order to analyze the performance of KDES for others organs, we apply KDES as descriptor at image level for all types of organs. For the fusion techniques, we investigates three fusion techniques: BC (Borda Count), IRP (Inverse Rank Position) and WP (Weighted Probability). Finally, we propose to use IRP and WP in our runs. The remaining of the paper is organized as follows. In Section 2, we present in detail the methods applied in each step of our system including preprocessing, feature extraction and fusion techniques. The experimental results on both validation and testing sets are shown in Section 3. Section 4 gives some conclusions and future works. 2 2.1

Proposed Approach

Overview

The overview of our system is illustrated in Fig. 1. The system consists of four main modules that are image preprocessing, feature extraction, classi cation and fusion. { Feature extraction: There are two main features extracted from organ images: KDES and HSV histogram. Since HSV histogram is well-known feature, in this paper, we describe only KDES. { Classi cation: We apply SVM as classi cation method at image level for all types of organs. { Fusion: we investigates three fusion techniques: BC (Borda Count), IRP (Inverse Rank Position) and WP (Weighted Probability).

Based on this sytem, we have created 3 runs to LifeCLEF 2015. { Run 1: In this run, we employ KDES for all types of organs and IRP (Inverse

Rank Position) for result fusion at observation level. { Run 2: The di erence between Run 1 and Run 2 is that for ower and entire images, we combine HSV histogram with KDES. We also apply IRP for result fusion. { Run 3: This run is similar to Run 3. However, instead of using IRP, we employ WP (Weighted Probability). 2.2

Preprocessing Techniques

The preprocessing techniques aim to separate the regions-of-interest (ROI) from an image. Except the leaf scan images, most of images of the observations are captured from natural scenes. In ImageClef 2014 [ 1 ], teams utilized a simple threshold method (such as Otsu threshold) in order to separate leaf regions for leaf-scan images. IBM-AU team deployed more complicated techniques (e.g., active contour-based, pre-determined size of ROI) to make a boundary box of ower, fruit, or leaf on complicated background, so on [ 1 ]. In this work, we deploy appropriate preprocessing techniques for each type of images as described below: { Leaf on the complicated background: An Interactive segmentation method, which is based on Watershed algorithms as described in [ 8 ], is applied to segment leaf from background regions; Moreover, size and main direction of the leave are normalized in a normalization procedure based on moment calculations [ 8 ]. Fig. 2 shows some results of the preprocessing techniques applied on the leaf images with the complex background. Because this is an interactive technique, it requires an user's manipulation. It is time consuming when working with a large number of images. { Leaf-scan images: We adapt a saliency extraction method as described in [ 4 ] and a common segmentation technique (e.g., mean-shift algorithm). An segmented region is selected based on a condition that its corresponding saliency value is large enough. The connected-region techniques then are applied to connect the selected regions. The main ow works are expressed in Fig. 3. Because leaf-scan images contain only simple background, we obtain stable results with leaf-scan images. The main reasons are that saliency values of leaf regions are more signi cant from background ones. { Flower and fruit images: we apply same saliency-segmentation procedure as

Leaf-scan image for selecting the ROIs on ower and fruit images; However, main di erence from leaf-scan images is that we do not immediately use the results after connecting selected-regions. Because ower and fruit images are captured in natural scene; It is di cult to obtain stable and correct segmented results; Instead of that, a boundary box is obtained based on top-left and bottom-right points on boundary of the connected-regions. The results of the selected ower and fruit regions are shown in Fig. 4 { Stem images: We observe that stem covers almost regions on the captured image. Moreover, we take into account texture of the stem regions, then a simple procedure to select ROIs on the stem image is based on a lter technique. We apply a Hanning window on the stem image. The size and sigma of the Hanning window is pre-determined. We then crop stem regions using the ltered image. The crop procedure utilizes a pre-determined pad which is 15 pixels from image border for both dimensions. Fig. 5 shows results of the ROI extracted on a stem image. 2.3

Feature Extraction

Kernel descriptor (KDES) has been proposed rstly by Liefeng Bo et al. [ 5 ]. In our previous works [ 8 ], [ 3 ], KDES has been proved to be robust for leaf representation. In this work, we propose to use KDES for all types of organ images. KDES is extracted from images of the organs after preprocessing through 3 steps: pixel-level feature extraction, patch-level feature extraction and image-level feature extraction. We employ the same process to compute KDES as proposed by Liefeng Bo et al. [ 6 ], [ 5 ]. However, we make the choice of pixel-level feature for each types of organs. We extract gradient as pixel-level feature for leaf, leaf scan, ower, fruit, branch and entire images while we use LBP (Local Binary Pattern) for stem images. The gradient vector at a pixel z is de ned by its magnitude m(z) and orientation (z). In [ 5 ], the orientation (z) is de ned as follows: (z) = [sin( (z) cos( (z))] (1)

Local binary patterns (LBP) is computed in the manner shown in Fig. 7. Each pixel is compared to each of its 8 neighbors. Where the neighborhood pixel's value is greater than the center pixel's value, the resulting LBP value is 1. Otherwise, it is 0. The result is an 8-digit binary number. Let denote the resulting binary 8-dimensional vector at pixel z by b(z), and denote the standard deviation of pixel values in the 3 3 neighborhood around z by s(z). b(z) and s(z) are treated as texture pixel level features. From the pixel-level feature, we extract the patch-level feature by generating patches from images and computing the approximate map based on match kernel. Corresponding to each type of pixellevel, we have a match kernel. After extracting patch-level feature, based on a pyramid structure, we can get the nal feature vector for organ image. For organ-based plant classi cation, we apply multi-class SVM. At the end of this step, from one organ image as query, we have a list of ranked plants. In our work, we investigate three di erent fusion techniques that are BC (Borda Count), IRP (Inverse Rank Position) and WP (Weighted Probability) [ 7 ]. These techniques are explained in the Eq. 2, Eq. 3 and Eq. 4.

BC(i) = IRP (i) = n X rank(i; j) j=1

1 Pn 1

j=1 rank(i;j) W P (i) = n X w(j)rank(i; j) j=1 (2) (3) (4) where n is the number of retrieved lists, rank(i; j) is rank of species i in list jth and w(j) is the weight of list jth 3 3.1

Experimental and Results

Data Preparation

The training data of this year nally results in 27,907 plant-observations illustrated by 91,759 images while the test data results in 13,887 plant-observationqueries illustrated by 21,446 images [ 2 ]. The number of images for each organ in training and testing set is shown in Tab.1. We apply the proposed preprocessing techniques as described in Sec. 2.2 in organ images. In order to evaluate our system, we divide the training set into training and validation sets by taking randomly 1/5 observation for validation and the remaining for training. For the leaf image (with complex background), since we use interactive segmentation and this is time consuming, therefore, in order to reduce the work, if the leaf scan set contains images of the leaf of one plant, instead of using images from leaf training set, we use images from leaf scan training set for this plant. The number of images for each organ in our training and validation set is shown in Tab. 2. We implement our proposed system in C++ and Matlab and use two libraries: OpenCV and KDES (http://www.cs.washington.edu/robotics/projects/kdes/). In order to evaluate the performance of our system on validation sets, we implement the score image and score observation as described in [ 2 ]. 3.3

Obtained Results

This section presents the results on both datasets: validation and testing set. For the testing set, we use the results released by the organizers of the task. Results on Validation Set We performs two experiments. In the rst experiment, we use KDES for all organs (KDES with LBP kernel for stem and KDES with gradient kernel for the others). In the second experiment, for ower and entire images, we combine HSV histogram with KDES by using IRP as fusion technique as organ level.

The results at image level and observation level of two experiments are shown in Tab. 3, Tab. 4 and Tab. 5 and Tab. 6 respectively.

Besides the score image, we compute the accuracy at rank k.

Accuracy = where T is the true recognition and N is the number of queries. One image or observation is correctly recognized if the relevant plant is in the k rst plants of the retrieved list. In our experiments, we compute accuracy at rank 1 and rank 10.

From the results obtained with two experiments, we extract three observation. Firstly, the obtained results have shown that KDES is a good descriptor for Leaf and Leaf Scan images. The score at image level for Leaf Scan is 62.88 % and the accuracy at the rank 1 is 78.28%. The performance of KDES is reduced when applying for Leaf images. The reason is that in Leaf image set, there are a number of compound leaf and we donot apply any speci c technique for compound leaf. The KDES is not a good choice for the others types of organ. This shows that KDES is relatively good and distinctive feature for classify the classes with high intra similarity such as leaf. Secondly, the combination of HSV histogram and KDES improves slightly the performance for Flower and Entire images (from 20.63% to 22.55% for Flower and from 10.36% to 11.3% for Entire). This shows that the global feature such as histogram can not help to identify the plants by using Flower and Entire images. More robust and local features need to be investigated. Finally, we can see the performance of di erent fusion techniques. It shows that IRP and WP obtain better results than BC in both experiments. Based on the results of two experiments in validation dataset, we decide to submit three runs to LifeClef 2015 - multi-organ plant identi cation task. The characteristic of each run is described as follows: { Run 1: In this run, we employ KDES for all types of organs and IRP (Inverse

Rank Position) for result fusion at observation level. { Run 2: The di erence between Run 1 and Run 2 is that for ower and entire images, we combine HSV histogram with KDES by using IRP. We also apply IRP for observation result fusion. { Run 3: This run is similar to Run 3. However, instead of using IRP, we employ WP (Weighted Probability).

Results on Test Set The score image and score observation of our three runs on test set is shown in Fig. 8 while the score for each type of organs is illustrated in Fig. 9. Our team is ranked at 5th place. We can see that Run 2 is slightly better than Run 1 and Run 3. This is consistent with the results obtained in the validation set. From the detailed score for each type of organs, we can see that KDES is relatively good in comparison with other descriptors used by others labs/teams. Our method for Leaf Scan obtains the second place. The score obtained for Leaf Scan is 0.737 while the score of the rst place team is 0.766. 4

Conclusions and Future works

In this paper, we have presented our proposed system for multi-organ plant identi cation. We have described in detail the proposed system and analyzed the results obtained on both validation and testing set. The obtained results with KDES for Leaf Scan are promising. However, the results are still limited for the others types of organs and multi-organ plant identi cation. In the future, we focus on compound leaf and descriptors for the other types of organs.

References

1. Goeau, H., Joly , A. , Bonnet , P. , Selmi , S. Molino , J-F . et al.: LifeCLEF Plant Identi cation Task 2014 . L. Cappellato; N. Ferro; M. Halvey ; W. Kraaij. CLEF2014 Working Notes. Working Notes for CLEF 2014 Conference, She eld, UK, September 15-18 , 2014 , CEUR-WS, 598 { 615 , ( 2014 )

2. Goeau, H., Joly , A. , Pierre , B. : LifeCLEF Plant Identi cation Task 2015 . CLEF working notes 2015 , ( 2015 )

3. Le , T. L. , Duong , N. D. , Nguyen , V. T. , Vu , H. , Hoang , V. N. , Nguyen , T. N. : Complex background leaf-based plant identi cation method based on interactive segmentation and kernel descriptor . In Proceeding of the 2nd International Workshop on Environmental Multimedia Retrieval 2015 ( EMR 2015), in Conjunction with ACM Conference on Multimedia Retrieval (ICMR ), Shanghai, China, ( 2015 ).

4. Achanta , R. , Hemami , S. , Estrada , F. , Susstrunk , F. : Frequency-tuned salient region detection . In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 1597 - 1604 , ( 2009 )

5. Bo , L. , Ren , X. , Fox , D. : Kernel Descriptors for Visual Recognition . In the Annual Conference on Neural Information Processing Systems (NIPS) , 1 - 9 ( 2010 )

6. Bo , L. , Sminchisescu , C. : E cient Match Kernel between Sets of Features for Visual Recognition . In the Annual Conference on Neural Information Processing Systems (NIPS) , 1 - 9 , ( 2009 )

7. Jovi , M. : Image Retrieval Based on Similarity Score Introduction : Image Similarity Computation . Pattern Recognition , 461 - 470 , ( 2006 )

8. Le , T. L. , Tran , D. T. , Hoang , V. N. : Fully automatic leaf-based plant identi - cation, application for Vietnamese medicinal plant search . In the fth symposium on Information and Communication Technology (SoICT 2014 ), Hanoi, VietNam , ( 2014 )