=Paper=
{{Paper
|id=Vol-1391/21-CR
|storemode=property
|title=MICA at LifeCLEF 2015: Multi-organ Plant Identification
|pdfUrl=https://ceur-ws.org/Vol-1391/21-CR.pdf
|volume=Vol-1391
|dblpUrl=https://dblp.org/rec/conf/clef/LeNVN15
}}
==MICA at LifeCLEF 2015: Multi-organ Plant Identification==
https://ceur-ws.org/Vol-1391/21-CR.pdf
MICA at LifeCLEF 2015: Multi-organ Plant
Identification
Thi-Lan Le, Nam-Duong Duong, Hai Vu, and Thanh-Nhan Nguyen
International Research Institute MICA
HUST - CNRS/UMI-2954 - GRENOBLE INP
Hanoi University of Science and Technology
Hanoi, Vietnam
{Thi-Lan.Le, Hai.Vu, Thanh-Nhan.Nguyen}@mica.edu.vn;
duongksclck55@gmail.com;
Abstract. In this paper, we describe the system including image prepro-
cessing, feature descriptor extraction, classification method and fusion
techniques that we applied for LifeCLEF 2015 - multi-organ plant iden-
tification task. In the preprocessing step, we apply relevant preprocessing
techniques for each type of plants’ organs based on the characteristic of
the organs. For the feature descriptor, we propose to use kernel descrip-
tor (KDES) with different types of kernel for all organs types. For flower
and entire images, we combine KDES with HSV histogram. At the im-
age level, we apply Support Vector Machine (SVM) as a classification
method. Finally, we investigate different late fusion techniques in order
to build the retrieved observation list.
1 Introduction
Plant identification is a process that aims at matching a given specimen plant to
a known taxon. This is a difficult and time consuming task even for the botanist
experts. Recently, with the advanced research in computer vision community, a
number of works have been proposed for plant identification based on images.
However, most of these methods have been dedicated to one sole organ of the
plants. The most widely used organs are leaf and flower. Since 2014, with the
availability of dataset for multi-organ plant identification, the plant identification
moves from image-centered to observation-centered [1]. The plant identification
can be defined as a image retrieval problem with the input is a set of organ
images of a query plant and the output is the ranked list of retrieved plants. In
this paper, we present our work dedicated to LifeCLEF 2015 - multi-organ plant
identification task [2]. Our previous work for leaf-based plant identification has
shown that the KDES is a robust descriptor for leaf representation [3]. Therefore,
in this work, in order to analyze the performance of KDES for others organs, we
apply KDES as descriptor at image level for all types of organs. For the fusion
techniques, we investigates three fusion techniques: BC (Borda Count), IRP
(Inverse Rank Position) and WP (Weighted Probability). Finally, we propose
to use IRP and WP in our runs. The remaining of the paper is organized as
follows. In Section 2, we present in detail the methods applied in each step
of our system including preprocessing, feature extraction and fusion techniques.
The experimental results on both validation and testing sets are shown in Section
3. Section 4 gives some conclusions and future works.
2 Proposed Approach
2.1 Overview
Fig. 1. Overview of our three runs for LifeCLEF 2015.
The overview of our system is illustrated in Fig. 1. The system consists of four
main modules that are image preprocessing, feature extraction, classification and
fusion.
– Preprocessing: We propose different preprocessing techniques based on the
characteristic of the organ images. Concretely:
• Leaf images: we employ our interactive image segmentation as presented
in [3].
• Leaf scan images: we apply salient region segmentation method and per-
form petiole removal and leaf normalization.
• Flower and fruit images: we propose an algorithm to detect the ROI
(Region Of Interest) based on salient characteristic.
• Stem images: in order to emphasize the stem region, we use the Hanning
window.
• Branch and entire images: we do not apply any preprocessing techniques
on the images of the branch and the entire.
– Feature extraction: There are two main features extracted from organ im-
ages: KDES and HSV histogram. Since HSV histogram is well-known feature,
in this paper, we describe only KDES.
– Classification: We apply SVM as classification method at image level for all
types of organs.
– Fusion: we investigates three fusion techniques: BC (Borda Count), IRP
(Inverse Rank Position) and WP (Weighted Probability).
Based on this sytem, we have created 3 runs to LifeCLEF 2015.
– Run 1: In this run, we employ KDES for all types of organs and IRP (Inverse
Rank Position) for result fusion at observation level.
– Run 2: The difference between Run 1 and Run 2 is that for flower and entire
images, we combine HSV histogram with KDES. We also apply IRP for
result fusion.
– Run 3: This run is similar to Run 3. However, instead of using IRP, we
employ WP (Weighted Probability).
2.2 Preprocessing Techniques
The preprocessing techniques aim to separate the regions-of-interest (ROI) from
an image. Except the leaf scan images, most of images of the observations are
captured from natural scenes. In ImageClef 2014 [1], teams utilized a simple
threshold method (such as Otsu threshold) in order to separate leaf regions for
leaf-scan images. IBM-AU team deployed more complicated techniques (e.g.,
active contour-based, pre-determined size of ROI) to make a boundary box of
flower, fruit, or leaf on complicated background, so on [1]. In this work, we deploy
appropriate preprocessing techniques for each type of images as described below:
– Leaf on the complicated background: An Interactive segmentation method,
which is based on Watershed algorithms as described in [8], is applied to
segment leaf from background regions; Moreover, size and main direction
of the leave are normalized in a normalization procedure based on moment
calculations [8]. Fig. 2 shows some results of the preprocessing techniques
applied on the leaf images with the complex background. Because this is an
interactive technique, it requires an user’s manipulation. It is time consuming
when working with a large number of images.
– Leaf-scan images: We adapt a saliency extraction method as described in [4]
and a common segmentation technique (e.g., mean-shift algorithm). An seg-
mented region is selected based on a condition that its corresponding saliency
value is large enough. The connected-region techniques then are applied to
connect the selected regions. The main flow works are expressed in Fig. 3.
Because leaf-scan images contain only simple background, we obtain stable
results with leaf-scan images. The main reasons are that saliency values of
leaf regions are more significant from background ones.
– Flower and fruit images: we apply same saliency-segmentation procedure as
Leaf-scan image for selecting the ROIs on flower and fruit images; However,
Fig. 2. Results of the segmented leaf from complex background. Upper row: original
images; Lower row: corresponding the segmented leaf results using proposed techniques
Saliency map Saliency value on
results whole image SK
Input Select the SM if Connected Interested-
image S M > α SK regions Regions
Saliency value on
Mean-shift
each segmented
segmentation
regions SM
Fig. 3. The preprocessing techniques for selecting the regions of interest for leaf-scan;
fruit; flower images
main difference from leaf-scan images is that we do not immediately use the
results after connecting selected-regions. Because flower and fruit images
are captured in natural scene; It is difficult to obtain stable and correct
segmented results; Instead of that, a boundary box is obtained based on
top-left and bottom-right points on boundary of the connected-regions. The
results of the selected flower and fruit regions are shown in Fig. 4
Fig. 4. The results of selected ROIs on flower and fruit images
– Stem images: We observe that stem covers almost regions on the captured
image. Moreover, we take into account texture of the stem regions, then
a simple procedure to select ROIs on the stem image is based on a filter
technique. We apply a Hanning window on the stem image. The size and
sigma of the Hanning window is pre-determined. We then crop stem regions
using the filtered image. The crop procedure utilizes a pre-determined pad
which is 15 pixels from image border for both dimensions. Fig. 5 shows
results of the ROI extracted on a stem image.
2.3 Feature Extraction
Kernel descriptor (KDES) has been proposed firstly by Liefeng Bo et al. [5]. In
our previous works [8], [3], KDES has been proved to be robust for leaf repre-
sentation. In this work, we propose to use KDES for all types of organ images.
KDES is extracted from images of the organs after preprocessing through 3 steps:
pixel-level feature extraction, patch-level feature extraction and image-level fea-
ture extraction. We employ the same process to compute KDES as proposed by
Liefeng Bo et al. [6], [5]. However, we make the choice of pixel-level feature for
Fig. 5. The result of selected ROIs on stem image. Left: original stem image; Right:
Filtered image using a Hanning window. ROI is marked in yellow box on the filtered
image
each types of organs. We extract gradient as pixel-level feature for leaf, leaf scan,
flower, fruit, branch and entire images while we use LBP (Local Binary Pattern)
for stem images. The gradient vector at a pixel z is defined by its magnitude
Fig. 6. KDES extraction and organ-based plant identification.
m(z) and orientation θ(z). In [5], the orientation θ(z) is defined as follows:
θ(z) = [sin(θ(z) cos(θ(z))] (1)
Local binary patterns (LBP) is computed in the manner shown in Fig. 7.
Each pixel is compared to each of its 8 neighbors. Where the neighborhood
pixel’s value is greater than the center pixel’s value, the resulting LBP value
is 1. Otherwise, it is 0. The result is an 8-digit binary number. Let denote the
resulting binary 8-dimensional vector at pixel z by b(z), and denote the standard
deviation of pixel values in the 3×3 neighborhood around z by s(z). b(z) and s(z)
are treated as texture pixel level features. From the pixel-level feature, we extract
the patch-level feature by generating patches from images and computing the
approximate map based on match kernel. Corresponding to each type of pixel-
level, we have a match kernel. After extracting patch-level feature, based on
Fig. 7. Local binary patterns (LBP)
a pyramid structure, we can get the final feature vector for organ image. For
organ-based plant classification, we apply multi-class SVM. At the end of this
step, from one organ image as query, we have a list of ranked plants.
2.4 Multi-organ Plant Identification
In our work, we investigate three different fusion techniques that are BC (Borda
Count), IRP (Inverse Rank Position) and WP (Weighted Probability) [7]. These
techniques are explained in the Eq. 2, Eq. 3 and Eq. 4.
n
X
BC(i) = rank(i, j) (2)
j=1
1
IRP (i) = Pn 1 (3)
j=1 rank(i,j)
n
X
W P (i) = w(j)rank(i, j) (4)
j=1
where n is the number of retrieved lists, rank(i, j) is rank of species i in list
j th and w(j) is the weight of list j th
3 Experimental and Results
3.1 Data Preparation
The training data of this year finally results in 27,907 plant-observations illus-
trated by 91,759 images while the test data results in 13,887 plant-observation-
queries illustrated by 21,446 images [2]. The number of images for each organ in
training and testing set is shown in Tab.1. We apply the proposed preprocessing
techniques as described in Sec. 2.2 in organ images. In order to evaluate our
system, we divide the training set into training and validation sets by taking
randomly 1/5 observation for validation and the remaining for training. For the
leaf image (with complex background), since we use interactive segmentation
and this is time consuming, therefore, in order to reduce the work, if the leaf
scan set contains images of the leaf of one plant, instead of using images from
leaf training set, we use images from leaf scan training set for this plant. The
number of images for each organ in our training and validation set is shown in
Tab. 2.
Table 1. Training and testing sets provided by LifeCLEF2015
Leaf Leaf scan Flower Fruit Stem Branch Entire
Training Set 13,367 12,605 28,225 7,720 5,476 8,130 16,235
Testing Set 2,690 221 8,327 1,423 584 2,088 6,113
Table 2. Training and validation sets used in our experiments
Leaf Leaf scan Flower Fruit Stem Branch Entire
Training Set 15,220 9,787 22,945 6,356 4,485 6,542 13,031
Validation Set 1,814 2,610 5,280 1,364 994 1,588 3,204
3.2 Working Environment
We implement our proposed system in C++ and Matlab and use two libraries:
OpenCV and KDES (http://www.cs.washington.edu/robotics/projects/kdes/).
In order to evaluate the performance of our system on validation sets, we imple-
ment the score image and score observation as described in [2].
3.3 Obtained Results
This section presents the results on both datasets: validation and testing set.
For the testing set, we use the results released by the organizers of the task.
Results on Validation Set We performs two experiments. In the first experi-
ment, we use KDES for all organs (KDES with LBP kernel for stem and KDES
with gradient kernel for the others). In the second experiment, for flower and
entire images, we combine HSV histogram with KDES by using IRP as fusion
technique as organ level.
The results at image level and observation level of two experiments are shown
in Tab. 3, Tab. 4 and Tab. 5 and Tab. 6 respectively.
Besides the score image, we compute the accuracy at rank k.
T
Accuracy = (5)
N
where T is the true recognition and N is the number of queries. One image or
observation is correctly recognized if the relevant plant is in the k first plants of
the retrieved list. In our experiments, we compute accuracy at rank 1 and rank
10.
Table 3. Results at image level of the first experiment on validation set
Score (Image) Accuracy(%)
Rank 1 Rank 10
Leaf 32.90 24.26 46.36
Leaf Scan 62.88 78.28 92.76
Flower 20.63 10.95 24.62
Fruit 13.96 6.52 20.46
Stem 13.16 16.60 34.20
Branch 7.18 3.53 9.70
Entire 10.36 6.40 14.61
Table 4. Results at observation level of the first experiment on validation set
BC IRP WP
Score (observation) 21.86 23.31 22.22
Rank 1 22.49 24.22 22.96
Rank 10 37.80 39.28 38.84
Table 5. Results at image level of the second experiment on validation set
Score (Image) Accuracy(%)
Rank 1 Rank 10
Leaf 32.90 24.26 46.36
Leaf Scan 62.88 78.28 92.76
Flower 22.55 11.38 38.05
Fruit 13.96 6.52 20.46
Stem 13.16 16.60 34.20
Branch 7.18 3.53 9.70
Entire 11.30 6.62 17.51
From the results obtained with two experiments, we extract three observa-
tion. Firstly, the obtained results have shown that KDES is a good descriptor
for Leaf and Leaf Scan images. The score at image level for Leaf Scan is 62.88
% and the accuracy at the rank 1 is 78.28%. The performance of KDES is re-
duced when applying for Leaf images. The reason is that in Leaf image set, there
Table 6. Results at observation level of the second experiment on validation set
BC IRP WP
Score (observation) 21.75 23.27 22.36
Rank 1 22.31 23.29 22.83
Rank 10 38.75 39.51 39.95
Fig. 8. Score image and score observation of our three runs on test set [2].
are a number of compound leaf and we donot apply any specific technique for
compound leaf. The KDES is not a good choice for the others types of organ.
This shows that KDES is relatively good and distinctive feature for classify the
classes with high intra similarity such as leaf. Secondly, the combination of HSV
histogram and KDES improves slightly the performance for Flower and Entire
images (from 20.63% to 22.55% for Flower and from 10.36% to 11.3% for Entire).
This shows that the global feature such as histogram can not help to identify
the plants by using Flower and Entire images. More robust and local features
need to be investigated. Finally, we can see the performance of different fusion
techniques. It shows that IRP and WP obtain better results than BC in both
experiments. Based on the results of two experiments in validation dataset, we
decide to submit three runs to LifeClef 2015 - multi-organ plant identification
task. The characteristic of each run is described as follows:
– Run 1: In this run, we employ KDES for all types of organs and IRP (Inverse
Rank Position) for result fusion at observation level.
Fig. 9. Detailed scores obtained for each type of plant organs of our three runs on test
set. [2]
– Run 2: The difference between Run 1 and Run 2 is that for flower and entire
images, we combine HSV histogram with KDES by using IRP. We also apply
IRP for observation result fusion.
– Run 3: This run is similar to Run 3. However, instead of using IRP, we
employ WP (Weighted Probability).
Results on Test Set The score image and score observation of our three
runs on test set is shown in Fig. 8 while the score for each type of organs is
illustrated in Fig. 9. Our team is ranked at 5th place. We can see that Run
2 is slightly better than Run 1 and Run 3. This is consistent with the results
obtained in the validation set. From the detailed score for each type of organs,
we can see that KDES is relatively good in comparison with other descriptors
used by others labs/teams. Our method for Leaf Scan obtains the second place.
The score obtained for Leaf Scan is 0.737 while the score of the first place team
is 0.766.
4 Conclusions and Future works
In this paper, we have presented our proposed system for multi-organ plant
identification. We have described in detail the proposed system and analyzed
the results obtained on both validation and testing set. The obtained results
with KDES for Leaf Scan are promising. However, the results are still limited
for the others types of organs and multi-organ plant identification. In the future,
we focus on compound leaf and descriptors for the other types of organs.
References
1. Goëau, H., Joly, A., Bonnet, P., Selmi, S. Molino, J-F. et al.: LifeCLEF Plant
Identification Task 2014. L. Cappellato; N. Ferro; M. Halvey; W. Kraaij. CLEF2014
Working Notes. Working Notes for CLEF 2014 Conference, Sheffield, UK, September
15-18, 2014, CEUR-WS, 598–615, (2014)
2. Goëau, H., Joly, A., Pierre, B.: LifeCLEF Plant Identification Task 2015. CLEF
working notes 2015, (2015)
3. Le, T. L., Duong, N. D., Nguyen, V. T., Vu, H., Hoang, V. N., Nguyen, T. N.:
Complex background leaf-based plant identification method based on interactive
segmentation and kernel descriptor. In Proceeding of the 2nd International Work-
shop on Environmental Multimedia Retrieval 2015 (EMR 2015), in Conjunction
with ACM Conference on Multimedia Retrieval (ICMR), Shanghai, China, (2015).
4. Achanta, R., Hemami, S., Estrada, F., Susstrunk, F.: Frequency-tuned salient re-
gion detection. In IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), 1597-1604, (2009)
5. Bo, L., Ren, X., Fox, D.: Kernel Descriptors for Visual Recognition. In the Annual
Conference on Neural Information Processing Systems (NIPS), 1-9 (2010)
6. Bo, L., Sminchisescu, C.: Efficient Match Kernel between Sets of Features for Visual
Recognition. In the Annual Conference on Neural Information Processing Systems
(NIPS), 1-9, (2009)
7. Jovi, M.: Image Retrieval Based on Similarity Score Introduction : Image Similarity
Computation. Pattern Recognition, 461-470, (2006)
8. Le, T. L., Tran, D. T., Hoang, V. N.: Fully automatic leaf-based plant identifi-
cation, application for Vietnamese medicinal plant search. In the fifth symposium
on Information and Communication Technology (SoICT 2014), Hanoi, VietNam,
(2014)