Automatic plant identi cation tasks have witnessed increased interest from the machine learning community in recent years. This paper describes our (team IITK's) participation in the Plant Identi cation Task, CLEF 2012, organized by the Combined Lab Evaluation Forum (CLEF) where the challenge was to identify plant species based on leaf images. We rst categorize the di erent types of images and then use a variety of novel preprocessing methods such as shadow and background correction, petiole removal and automatic lea et segmentation for identifying the leaf blobs. We next use complex network framework along with novel tooth detection method and morphological operations to compute several useful features. Finally, we use a random forest for classi cation. Based on the proposed approach, we achieved 2nd rank on the overall score in the competition.
Automatic plant identi cation tasks have gained recent popularity due to its
use in quick characterization of plant species without requiring the expertise of
botanists. Leaf-based features are preferred over owers, fruits, etc. due to the
seasonal nature of the later and also the abundance of leaves (except may be
for the winter season). The Combined Lab Evaluation Forum (CLEF) hosts an
annual competition on classifying plant species based on images of leaves. While
there are some other important publicly available leaf image datasets such as
the Flavia Dataset [
This paper describes our (team IITK) approach for the ImageCLEF Plant Identi cation Task, CLEF 2012. Our focus for this endeavor was on two main points: (i) providing good recognition accuracy for natural images and (ii) automating the process for the controlled setup images. We have been able to achieve both our targets satisfactorily as corroborated by the fact that one of our submitted runs achieved the 2nd position overall in the competition.
Our contributions in this paper are as follows: 1. We propose novel pre-processing strategies for shadow removal and background noise correction. 2. We propose a fully automatic lea et extraction approach for compound leaves. 3. We propose the use of tooth features, that provide a second level of discrimination for leaves with similar shape. 4. We also incorporate the use of an e ective feedback based image segmentation interface for natural photographs. 2
In this section, we present our proposed approach in detail. We begin with a description of the dataset followed by the preprocessing techniques. Next, we discuss the image features used and conclude with the classi er. 2.1
The ImageCLEF Pl@ntLeaves II dataset consists of a total of 11572 images from 126 tree species in the French Mediterranean area. The dataset is subdivided into three di erent types based on the acquisition methodology used: scans (57%), scan-like photos, i.e., pseudo-scans (24%) and natural photographs (19%). The entire dataset is divided into a training and a test set as speci ed in Table 1. Type
Scan Pseudo-scan Natural Total
Associated with each image are metadata elds that include acquisition type, GPS coordinates of the observation, name of the author of the picture. The training images also contain the taxon name of the leaf species, and the task is to predict this eld for the test images.
To classify an image, we rst need to segment the leaf from the image. The process of segmentation, however, is not straightforward at all owing to the presence of several bottlenecks such as shadow, occlusion and complex leaves (Fig. 1 highlights some of these). While some of the roadblocks such as shadow removal, petiole removal, etc. are common to most of the images, a quick glance at the dataset suggests that no common segmentation scheme can be applied to all the images. Images having a single leaf have di erent issues than compound leaf images. It is, therefore, useful to group the images with similar issues into a category and address each category separately. Based on this observation, the dataset was each divided into three categories as follows { Category 1: Scan + Pseudo-scan, Single Leaf { Category 2: Scan + Pseudo-scan, Compound Leaf { Category 3: Natural Photographs All natural photographs (type 3) were put in a single category. The remaining images were put in two separate groups depending on whether they contained single or compound leaves.
Fig. 2 shows the overview of our system. Based on the category of the image, we follow di erent paths. We next discuss the image preprocessing techniques for each category. 2.2
The image preprocessing involved steps such as basic segmentation, petiole removal, shadow removal, background noise removal, etc. which collectively aid the extraction of the leaf part from the image. The procedure is fully automatic for category 1 and category 2 images while it is semi-automatic (interactive) for category 3.
This category is composed of scan and pseudo-scan images of single leaf
species. Fig. 3(a) shows one such image. We rst perform OTSU thresholding3
3 OTSU performs binarization by selecting an optimum threshold to separate the
foreground and background regions of the image such that their combined
(intraregion) variance is minimal.
[
This category is composed of scan and pseudo-scan images of compound leaf
species. Fig. 4(a) shows one such example. Such species contains a main stalk
and several lea ets that branch out from the main stalk. Using shape-descriptors
on the entire leaf does not capture the characteristics of the di erent compound
leaf species. Thus, it is necessary to perform all analysis at the lea et level. The
challenge then is to segment a single lea et from the compound leaf image. Our
system undertakes the following steps to achieve the same:
1. Binarization and Shadow and Noise Removal: Since these two steps
do not involve the intricacies of leaf structure, we follow the exact same
procedure as in Category 1 images. Fig. 4(a) shows such an example.
2. Main Stalk Elimination: Since the ultimate aim is to extract a single
lea et, rst the main stalk needs to be identi ed and removed from the
image. A simple erosion operation does not work as the thickness of the
main stalk can vary quite largely. We t a curve of order 4 to approximate
the main stalk (using the 'poly t' operator from Octave [
petiole fragments around the lea et. We resolve these in the next steps.
Fig. 4(d) shows the three extracted candidates. 5. Noise Removal: We use a simple range lter to address the background fragments. We rst compute the average RGB values for the background by traversing around the periphery of the original image. We then account for the background noise pixels by examining each pixel in the periphery and deselecting it (from the binary mask) if it falls within a particular range of the average background color. 6. Stalk Removal: The strategy used for stalk removal for category 1 images cannot be applied to the lea ets of compound leaves. This is because, unlike the single leaf images, the lea ets are not vertically oriented and upright. Hence, an erosion-based scheme is used to eliminate the stalk from the lea et bitmap. The inherent assumption in this scheme is that the erosion operation does not a ect the shape and margin of the lea ets. Although this step did not completely solve the problem, it did manage to reduce the stalk content (if present) in the image. Fig. 4(e) shows the three candidate lea ets after noise and stalk removal. 7. Best Candidate Selection: The output of the last step produces three candidate lea ets of which we need to choose only one. While it is easy for a human being to choose the \best" among the three, we wanted to automate the entire process. Hence, we use the following simple method for the same. For each candidate, we compute a 50-dimensional feature vector using complex network shape descriptors (see Sec. 2.3 for a description of the features). We then compute the Euclidean distances between each pair of candidates and nd the maximum pairwise distance. The candidate that does not feature in this is in some sense in the middle and was, thus, chosen as the \best" candidate.
Category 3 images consist of natural photographs. There is no distinction
between single and compound leaves in this category. Fig. 5 illustrates a few
images in this category. The images su er from multiple issues including occlusion,
out of focus problem, etc. It is di cult to design an automated segmentation
system that is robust to all these issues. We, therefore, resort to a semi-automated,
feedback-based segmentation scheme using the GrabCut algorithm [
Fig. 6 shows the output at various stages for segmentation of leaf from a sample category 3 image. The top row denotes the user interactions and the bottom row denotes the output for the corresponding user actions. 2.3
The preprocessing phase, as described in Sec. 2.2, results in an image containing a single lea et which acts as input to the feature extraction phase. The feature extraction phase results in a 90-dimensional feature vector comprising of geometrical, shape and texture features for each image. On a coarse level of granularity, the feature vector can be divided into three categories: 50 complex network, 28 tooth and 12 morphological features. These are described next.
The motivation behind using complex network features [
The graph thus obtained has connections between every pair of vertices and, therefore, cannot be considered as a complex network. We accomplish the transformation of this regular network into a complex network by thresholding the edge weights. We consider views of the network at multiple resolutions (thresholds), each containing di erent number of edges. An edge is retained if its weight is less than the threshold at that level and discarded otherwise.
The complex network thus constructed provides ve di erent features that can be broadly categorized as degree descriptors and joint-degree descriptors. 1. Degree Descriptors: The degrees of the vertices are rst normalized into [0; 1] by dividing by the total number of vertices in the network. Then, the maximum degree and mean degree are realized as the two degree descriptors:
N M aximum = max di; i=1
M ean =
N X di=N i=1 where di denotes the degree of vertex vi and N is the total number of vertices. 2. Joint-Degree Descriptors: The joint-degree descriptors rely on the correlations between the degrees of vertices connected by an edge. The correlation for a degree di is captured by the probability P (di) that two neighboring vertices have the same degree di. This probability is computed empirically as the ratio of the number of all connected vertices with the same degree di to the total number of edges. Using this, the following three descriptors are realized: Entropy =
D X[P (di) log2 P (di)]; i=0
Energy =
Sum = D X[P (di)]2; i=1
D X[P (di)] i=1 where D is the maximum degree in the network.
We realize the network at 10 di erent resolutions ranging uniformly from 0:05 to 0:50. At each threshold, we calculate 2 degree descriptors and 3 joint-degree descriptors, thus producing a total of 50 complex network features.
We next describe the tooth features. A tooth point is a pixel on the contour that has a high curvature, i.e., it is a peak. To determine whether a point Pi on the contour is a tooth point or not, we examine the angle subtended at Pi by its neighbors Pi k and Pi+k (where k is a threshold). Fig. 7 shows an example. If the angle is within a particular range, then Pi is a tooth; otherwise, it is not. The upper bound of is =2, i.e., the right angle. The lower bound was determined by manual observations. The value that gave the best result was sin 1 0:8, i.e., the sine of the angle was constrained to be 0:8 sin 1. The feature value for a leaf at a particular threshold k is the number of tooth points on its contour.
Fig. 8 shows the output of the tooth point detection method on a leaf for two di erent thresholds. As the threshold, k, is increased from 5 to 45, the number of contour points detected as tooth points decreases from 25 to 11.
Since it is possible for two di erent types of leaves to have nearly the same number of teeth at a particular threshold, we compute the tooth-based features at multiple increasing values of k. While no single threshold may be good enough to distinguish di erent leaf margins, they should be su ciently separated when multiple thresholds are used. We use 28 thresholds from 3 to 30.
For morphological features, we use the same ones as described in [
(a) k = 5
(b) k = 45
Vertical orientation was performed using principal component analysis (PCA). It was observed that one of the two principal components generated by PCA is along the direction of the main vein. This works due to the fact that most of the lea ets are either ellipsoidal or palmately lobed in shape. The ellipsoidal lea ets have the main vein along the major axis of the ellipse that ts the lea et. For palmately lobed lea ets, the two sides of the lea et along the main vein are almost symmetric, and hence, the main vein is along one of the principal components.
To determine whether a lea et is ellipsoidal in nature, we rst compare its area with the minimum enclosing ellipse. If they match well, we consider the lea et to be elliptical in shape. To vertically orient the lea et along the main vein, we then rotate the image along the rst principal component of the ellipse.
For palmately lobed lea ets, on the other hand, we associate a symmetry measure with each principal component. This measure checks the similarity between the two sides of the lea et along the axis represented by that principal component. The one with a higher symmetry measure is considered to be along the main vein and the lea et is rotated along that direction.
This vertical orientation process renders the task of marking endpoints of the
main vein automatic, which in turn, makes computing the set of 12 morphological
features as mentioned in [
Perimeter
These geometric parameters yield the following 12 morphological features: 1. Smooth Factor: This is de ned as the ratio of the area of the image smoothened by a 5 5 averaging lter to that smoothened by a 2 2 averaging lter. 2. Aspect Ratio: This is de ned as the ratio of physiological length to physiological width, i.e., L=W . 3. Form Factor: This measures the \roundness" of the leaf and is computed as 4 A=P 2. 4. Rectangularity: This measures how rectangular the leaf is and is computed as L:W=A. 5. Narrow Factor: This measures the \narrowness" of the leaf and is de ned as D=L. 6. Perimeter Ratio of Diameter: This is de ned as the ratio of the perimeter of the leaf to its diameter of the leaf, i.e., P=D.
This is de ned as the ratio of the perimeter of the leaf to the sum of its physiological length and physiological width, i.e., P=(L + W ). 8. Vein Features: The skeletal structure of a leaf is de ned by the vein patterns which play an important role in distinguishing the leaves that have similar shape. The standard and most widely used procedure for computing the vein features is to perform a morphological opening operation on the grayscale image. A at, disk shaped structuring element of radius r is used and the resultant image is then subtracted from the contour of the leaf. The output resembles the vein structure of a leaf on which we compute the following 5 features: A1=A, A2=A, A3=A, A4=A, A4=A1 where Ar denotes the area of the remaining leaf obtained using a structuring element of radius r, and A is the area of the original leaf as mentioned earlier.
The proposed preprocessing techniques failed to successfully segment some of the images in the dataset (see Fig. 10 for examples). These leaves are, in a way, compound of compound leaves { each branch of a leaf is analogous to
(a) Scan (b) Pseudo
(c) Photograph
a compound leaf. This distinction of \complex leaves" is done by examining
the number of connected components calculated after performing the erosion
operation on category 2 images. The leaves with high number of connected blobs
( 30) are termed as complex, thereby marking the others as normal compound
leaves. Since a successful leaf segmentation is critical to the computation of the
proposed features, we bypass the usual feature extraction process for the complex
leaves, and instead, use a bag of features model [
We rst compute the SURF descriptors [
Given a complex image, we then extract all its SURF points, and denote
each SURF descriptor by the nearest cluster center (equivalently, the code
corresponding to it). This results in a histogram of size 8 for every image where
bin i represents the number of SURF points that correspond to that particular
code. The histogram represents the feature vector of the image.
We tested many classi ers for predicting the type of species, and after extensive
experimentation, chose random forests [
We submitted four di erent runs for the CLEF 2012 plant identi cation task. All of them used the same preprocessing and classi cation methodologies and di ered on the basis of (i) inclusion or exclusion of the author id metadata eld, and (ii) choice between two di erent parameter sets for the random forest classi er. The two di erent parameter sets di er on the basis of number of trees in the random forest and the number of attributes (features) considered at each node of the trees to make a split. Table 2 provides a complete description of the classi er parameters. Run ID
Single Leaf Compound Leaf # Trees # Split Features # Trees # Split Features where U denotes the number of photographers that have submitted at least one test image, Pu denotes the total number of images taken by the uth photographer, Nu;p denotes the number of images of the pth plant taken by the uth photographer, and Su;p;n denotes a value between 0 and 1 explained next.
For each test image, multiple guesses about its correct class were allowed. The term Su;p;n measures how good the guess (for the nth image of the pth plant taken by the uth photographer) is, and is inversely proportional to the rank of the correct guess. The later the rank, the lower is the score, and the function decreases rapidly.
Table 3 shows the details and nal standings of our four submitted runs.
Our best submission, Run 3, was the only one that did not use any metadata
information. For the other runs, we used the photographer id metadata eld
as the 91st feature as we observed that it improved the classi cation accuracy
signi cantly for a 10-fold cross validation evaluation. we utilized SMOTE [
In this paper, we have tackled the problem of plant identi cation using leafbased features. We used the Pl@ntLeaves II dataset by CLEF as the benchmark to support our results. We proposed novel preprocessing strategies for shadow removal and background noise correction. We also proposed an automated lea et segmentation procedure for compound leaves. We introduced the use of tooth features to discriminate leaves with similar shapes but di erent margins. We also implemented an important improvement on the calculation of morphological features by automating the process of detecting the endpoints of the main vein. We nally used a combination of shape, morphological and tooth features with
Run ID Scorescan Scorepseudo Scorenatural Scoreaverage Rank random forests for classi cation. Our approach helped us achieve an overall rank of 2nd in the Plant Identi cation Challenge, CLEF 2012.
In future, we would like to achieve e ective automatic segmentation of natural photographs as well. We would also like to work on optimal fusion of di erent types of features as well as classi er boosting to improve the results.
We would like to thank Aditya Nigam, Tejas Gandhi and Manash Pal for their guidance and feedback during the course of the project. We would also like to thank our department for giving us the resources and the freedom to pursue such non-curriculum academic interests.