We describe our participation in the plant identi cation task of ImageClef 2012. We submitted two runs, one fully automatic and another one where human assistance was provided for the images in the photo category. We have not used the meta-data in either one of the systems, for exploring the extent of image analysis for the plant identi cation problem. Our approach in both runs employs a variety of shape, texture and color descriptors (117 in total). We have found shape to be very discriminative for isolated leaves (scan and pseudoscan categories), followed by texture. While we have experimented with color, we could not make use of the color information. We have employed the watershed algorithm for segmentation, in slightly di erent forms for automatic and human assisted systems. Our systems have obtained the best overall results in both automatic and manual categories, with 43% and 45% identi cation accuracies respectively. We have also obtained the best results on the scanned image category with 58% accuracy.
A content-based image retrieval (CBIR) system for plants would be very useful
for plant enthusiasts or botanists who would like to learn more about a plant they
encounter. Until the rst ImageCLEF Plant Identi cation Competition
organized in 2011 [
As with the rst competition, the plant identi cation task in ImageCLEF
2012 consisted of identifying images of plants that were captured by di erent
means: scans, scan-like photos (called pseudo-scans) and unrestricted photos, as
shown in Fig. 1. In this way, the competition aimed to benchmark state-of-the-art
in both isolated leaf shape and unrestricted plant image recognition problems.
The details of this competition are described in [
Content-based plant identi cation problem faces many challenges such as
color, illumination, size variations that are also common in other CBIR problems,
as well as some speci c problems, such as variations in the composition of the
leaves that change the plant shape. In addition, one can see that color is less
identifying in the plant retrieval problem compared to many other retrieval
problems, since most plants have green tones as their main color with subtle
di erences. In the rare cases that color is discriminative for a certain plant,
then it may still be the case that the leaves of that plant may change colors
due to seasonal variations. Shape features are useful in identifying isolated leafs,
but not really useful in identifying full or partial plant images [
(a) (b) (c) (d) (e) (f) As a collaboration between Sabanc and Okan Universities, we submitted two runs, one fully automatic and another one where human assistance was provided for the images in the photo category. The only distinction between our two runs is that the images in the photo category undergo a human-assisted segmentation process, as explained in Section 3.2. Hence, in the remainder of this paper, we will talk about one system, since everything besides the segmentation of photos are the same in the two systems corresponding to the two submitted runs.
The system is designed as two separate sub-systems, one for scan and
scan-like images and another one for photos. Since the meta-data included the
acquisition type, an input image is automatically sent to the correct subsystem,
but that was the extent of the use of the meta-data. Our system shares many
common parts with the system we sent to ImageCLEF2011, described in [
In ImageCLEF2011, we had concentrated our e orts almost exclusively to scan and pseudo-scan images, while photos were neglected due to lack of time. This year, we developed automatic and human-assisted segmentation algorithms for the photo category. In both segmentation algorithms, we aimed to segment the image to leave only a single leaf, so as to use our isolated leaf recognition engine. The segmentation was reasonably successful, in that our system had obtained 5.3% accuracy in photos last year, while the automatic photo identi cation accuracy was 16% this year. The segmentation steps are explained in Section 3.
For recognizing photographs, we found that many of our other feature descriptors, especially global shape descriptors, would not be useful if the photo was one of a partial plant. On the other hand for scan and scan-like categories containing isolated leaves, all three main feature categories are expected to be useful: shape, texture and color. After experimenting with a large number of descriptors, we selected a 117-dimensional feature vector for the scan/scan-like sub-system and a subset of these for the photo category. The features used in our system are explained in Section 5.
Another major problem we encountered last year was the over- tting problem in classi er training: our results obtained in o cial tests were seriously lower compared to the results obtained with cross-validation experiments done on the training set (around 40% di erence in accuracy). This year our feature selection and classi er optimization approaches were run on a separate test set which was partitioned from the original training data such that the two did not to include images of the same individual plant (e.g. the same exact tree). For the same reason, we have excluded our powerful new morphological texture features, in order to study them further. This issue is explained in 6. 3
The ultimate goal of segmentation is the separation of the leaf from its background. If no a priori knowledge is available about the image, the segmentation problem becomes equivalent to unconstrained color image segmentation, where the various leaves can be located within an equally immense variety of backgrounds, such as those shown in Fig. 2. One might argue at this stage that the background can provide information facilitating the recognition of the leaf/plant in the foreground. For instance if the background consists of forest ground versus the sky, we can eliminate some alternatives as implausible. However, this would additionally require the description and recognition of the background, which given its limitless diversity potential, increases the complexity of an already challenging problem.
The accurate separation of the background from the plant/leaf under
consideration is of crucial importance for the subsequent stage of feature
extraction, since a poor result would a ect many of the features. Given the
ill-posed nature of segmentation, in addition to the lack of any useful a priori
knowledge, it becomes evident that some form of human intervention or feedback
is necessary for an accurate and reliable segmentation. However, we also have
to take into account practical considerations, since no user/expert would like to
spend a prolonged amount of time on this stage, especially when dealing with
voluminous amounts of data. That is why we have explored both automatic and
human assisted segmentation strategies.
Although the ImageClef dataset is classi ed into three categories, namely scan,
pseudo-scan and photos, from a segmentation viewpoint scan and pseudo-scan
type images are of similar quality, in other words they both possess a mostly
noise-free, spectrally homogeneous background, occasionally containing some
amount of shadow in the pseudo-scans. Consequently, as far as scan and
pseudoscan type images are concerned, automatic segmentation has been trivially
resolved through Otsu's method [
Photos on the other hand consist of unconstrained acquisitions of leaves, as shown in Fig. 2, and thus their automatic segmentation is a far greater challenge. In order to counter this problem we adopted a combination of spectral and spatial techniques.
More precisely the only assumption concerning the input has been that
the object of interest, i.e. the leaf, is located roughly at the center of the
image and possesses a single dominant color. The image was rst simpli ed
by means of marginal color quasi- at zones [
At this point our initial assumption about the object of interest's central
location was used, as we employed the central 2/3 area of the image in order
to determine its dominant color, which was obtained by means of histogram
clustering in the LSH color space. Assuming that the mean color of the most
signi cant cluster (i.e. reference color) belongs to the leaf/plant, we then
switched to spectral techniques, so as to determine its watershed basins. Since
camera re ections can be problematic due to their low saturation, we computed
both the achromatic, i.e. grayscale distance image from the reference gray
(Fig. 4d) and the angular hue distance image (Fig. 4f) from the reference hue
href [
if href j if jh jh href j < href j (1) We then applied Otsu's method on both distance images, that provided us with two masks (Figs. 4e & 4g), representing spectrally interesting areas w.r.t. the reference color. The intersection of the two masks was used as the nal object mask (Figs. 4h). As shown in Fig. 4i, spectral and spatial techniques indeed complement each other well, while the use of both chromatic and achromatic distances increases the method's robustness. However the main di culty is the accurate determination of the reference or dominant color; this can easily corrupt the entire process if computed incorrectly or if the leaf under consideration has more than one dominant colors. 3.2
As the automatic segmentation module relies strongly on its initial assumption, it does not work very well, resulting in over or wrong segmentation. We investigated human assisted segmentation for the semi-manual category of the competition.
Ideally, given an input image we would like the user/expert to spend
at most a few tens of seconds in order to provide some amount of high level
knowledge. Assuming that the provided knowledge is valid, we chose to use the
marker based watershed transform [
To accomplish this we need a suitable topographic relief as input, where
object borders are denoted as peaks and at zones as valleys; which is why we
employ the morphological color gradient of the input image [
Grayscale (e) Grayscale mask (f) Hue distance (g) Hue mask (h) Mask intersection (i) Mask superposition on the original since if they are too small they can lead to partial leaf detection and conversely if they are excessively large, the leaf will be confounded with its background. 4
After segmentation, we have a single leaf which is segmented from its background, regardless of the type of the original image (scans or photos). For the case of photos, this isolated image is well segmented in the case of humanassisted segmentation, except possibly for some noise at the contour; in the case of automatic segmentation, the leaf may be over or under-segmented. Note that in all photos, the segmented image may also be rotated in an arbitrary direction, while most of the scanned leaves are oriented with their axes aligned with the vertical, stem part down.
Our preprocessing consists of orientation normalization for photos, followed by height normalization (keeping the height/width ratio unchanged). We have also experimented with stem location and orientation normalization for scans and pseudo-scans, however these steps are not yet mature, causing errors in as many images as the ones they correct. (a) Original image (b) Morphologi- (c) The markers (d) Result cal gradient
Fig. 5: Stages of human assisted photo segmentation on image #4835. 5
In a di cult problem such as this, it was clear that we needed to use all main feature categories (shape, texture and color), even though some of them were expected to be more useful than others. We have kept most of the features that were used in last years's competition, and we added some new ones that we thought would be useful.
Considering the intra-class color variations levels, it became evident at
an early stage that our approach would have to be mainly texture and shape
based, even though the discriminatory potential of color has not been ignored
completely. Moreover, it was chosen not to employ any further complicated
descriptors, such as scale invariant feature transforms (SIFT) or maximally
stable extremal regions (MSER), for two reasons. First, our past experience
with plant retrieval [
In total, we evaluated about 20 di erent shape, texture and color
features. The evaluated features consist of region-based (e.g. regional moments)
and contour-based (e.g. Fourier descriptors, border covariance) shape features;
texture features (morphological, Gabor, orientation histograms); and color (color
moments and di erent length and quantization of the color histogram in LSH
space). Here we summarize the new features, while others are described in detail
in [
Feature Selection We evaluated individual features using the global classi er described in Section 6.2, to observe their relative merits, but more importantly, to see the errors made in the system and to add more features as a remedy. Some of the above mentioned features (e.g. lobe descriptor) were added as a result of this process. In this process, we have also excluded the morphological texture features that we used in ImageCLEF2011.
The full set of our nal features (117 dimensional), along with their e ectiveness on cross-validation and test data, can be found in the Results section, in Table 1 and Table 2 for shape and texture features respectively. Color features were evaluated but we did not nd them useful, so they do not appear in the nal feature list. 6 6.1
The ImageClef2012 plant database contains 126 tree species and the images for
each species are contributed by various people, some of the images being taken
from the same individual plant [
This year, we took a di erent approach and divided the database into training and testing, rather than using cross-validation that may result in a random split of very similar images. In short, we put all of the images of an individual plant either in training or testing. Speci cally, for each of the species, if the species images contained two or more individual plants, one of them (the one with the least number of pictures) is separated for testing while all others are selected for training. In total 5163 scan and scan-like images are used for training and 1526 scan and pseudo-scan images and all photos (1733 images) are used for testing during our experiments.
Note that this approach for dividing the training set has the disadvantage that few of the species having images from a single individual plant did not have any representative images in test data. However, in terms of feature selection, we thought that this should not have a signi cant e ect, as features can be said to be species-independent.
As seen in Tables 1 and 2, cross-validation accuracies are almost always higher than test set accuracies, and especially so for texture features. This supports the association between cross-validation and the over- tting that applies in this particular problem. 6.2
We trained three classi ers in total for the automatic run: one Support Vector Machine (SVM) classi er for scans, pseudo-scans and manually segmented photos using all the features for isolated leaf recognition; one SVM classi er for automatically segmented photos using a subset of the features, and nally a single local classi er. The details of these classi ers are explained below. Finally, a score level combination of the two classi ers is performed for each category of images.
This year, the ambiguity resolution step, where we had used a third
classi er trained to distinguish between the top-5 choices [
Global Classi er Using the selected features described in Section 5, we trained two SVM classi ers, one for scans (scan and pseudo-scan categories) and manually segmented photos, and another for automatically segmented photos. When the photos are manually segmented, the results are similar to scanned images, hence we used the same classi er as for scans. For automatically segmented photos, the noise on the contours are more apparent, so we discarded the Fourier descriptors obtained via the Fast Fourier Transform (FFT) as they may be a ected from contour noise. For these classi ers, we used a SVM using the radial basis function kernel whose parameters were optimized using crossvalidation and grid search on the training data.
The training samples for these two classi ers were obtained from scan and pseudo-scan images only, so that the feature extraction was not a ected from segmentation noise.
Local Classi er Another classi er was built to compare local features obtained
from the stable points on the boundary of the leaves. The local features used here
were shape context [
Finally, using all the features (117-long feature vector), we obtained a 85.9% accuracy on cross-validation tests and 66.1% accuracy over the test data, while the results reaches almost 71% with classi er combination. Note that the test accuracy with all features show a slight decrease compared to using all shape features, but there is a signi cant increase in cross-validation accuracy when using all features. We believe that these issues will be less pronounced in the future when training and test data sizes increase.
Also note that our accuracies are measured as average accuracy over the test images, while the o cial scoring function computes an average across users (the people who have collected the images being seen as users of such a plant identi cation system).eing seen as users of such a plant identi cation system). Classi er Feature Length Cross Val Acc.(%) Global classi er (SVM) 117 85.94 Local classi er N/A N/A Classi er Comb. N/A N/A
Table 3: Classi er combination accuracies.
Test Acc.(%) 66.12 63.17 70.97 Hence, a signi cant amount of the di erence between the test results reported here and the o cial results may be due to this.
Accuracies for automatically processed photos from the test set are given
in Table 4. The photos are categorized into 4 categories by the organizers: i)
picked leaf ii) leaf iii) branch iv) leafage, containing roughly a picked single leaf
on a variety of backgrounds, a leaf which may be hanging from the branch, and
plant foliage with or without branches [
This year the organizers categorized automatic and manual or human-assisted runs separately, while this information was not clear in last year's results. According to the o cial results given in Table ??, our runs achieved 1st place in overall classi cation, for both automatic and human-assisted categories. We also obtained signi cantly better results on the scan category, compared to the next best system. Note that we only give a partial results table, including only the best automatic systems in addition to our two systems.
Although our results are satisfactory comparatively to other participants,
we believe there is still much room for improvement. Currently, plant
identication systems cannot be of practical use, maybe only to narrow down the
alternatives. That is why we have established our future work plan in two
directions, with both converging on the common goal of accuracy improvement.
For one, having resolved the issues of over- tting, we now possess a sound feature
selection and optimization setup, that we intend to exploit in order to explore
the latest and especially morphology related content descriptors [
Moreover, we are well aware of our comparatively poor results in the photo
category, as well as of its strong ties to segmentation quality. Hence we will focus
particularly on improving the performance of leaf isolation, in both automatic
and human-assisted approaches. To this end, we plan to further incorporate the
latest results obtained from our ongoing work on color quasi- at zones [
Finally, we will think of ways to bene t from color and other new descriptors.
Run Auto/Manual Scan Sabanci Okan-run2 Manual 0.58 Sabanci Okan-run1 Automatic 0.58 INRIA Imedia plantnet run1 Automatic 0.49 INRIA Imedia plantnet run2 Automatic 0.39 LSYS DYNI run 3 Automatic 0.41 ARTELAB run 1 Automatic 0.40