Our aim is to implement a plant identification application that can run on smartphones, and this shared task includes it. After the plant identification task of 2013 we concluded that the most frequent trees (e. g. in Hungary) can be identified well by a leaf, when there is a white paper background behind it at the time of photographing. This is why we want to classify more accurately this kind of pictures. Our other important goal is to develop a system which can be trained online while smartphone users take more and more photos. In the previous year there was a separate shared task for the 'Sheet as Background' photos, however this year the task is very complex, and supposes solutions for hard image processing problems, not just feature extraction and classification solutions.
The ImageCLEF Plant identification task [
The training dataset [
Table 1 lists some statistics concerning the photos for each category. As can be seen, in the case of ”Branch”, ”Fruit”, ”Leaf” and ”Stem” the majority of species are strongly underrepresented; very little data is available. In the case of ”Flower”, however, there are many more photos, and for almost the all species (483), from the 500 species. The scan-like pictures have the best example/species ratio, however there are many species for which scan-like leaf data is not available. From this point of view, we should surely focus on the classification using mainly the scan-like leaf and flower pictures, or when using the other categories, then at the score aggregation we should consider the number of examples in the training set for a category and species. This category contains the most examples for a species, and it seems to be the easiest to preprocess, namely to get a mask of a leaf. We tried to find the less complex methods for masking, feature extraction and classication, because our aim is a fast method that can be used on a smartphone without waiting long for the classification or ranking results. Masking: Our masking method is rather simple. After taking the blue channel of the color picture, a 3x3 blur is appiled on it, and Otsu’s thresholding method is applied. The result is obtained from this after the salt-and-pepper noise has been removed.
The masking method will fail in some cases, for example when the background has a similar intensity on the blue channel than the leaf, but in most cases it works well.
Features: In order to classify the species by their leaves, an important task is to extract features suitable for this. Since the leaves’ orientation on the photo is not fixed, the appropriate features have to be scale and rotation invariant. Instead of starting from scratch, we based our features on two sets taken from the literature.
First, we implemented the vein and metric features introduced in [
More complex features are based on the TAR and TSL descriptors defined
in [
The segmentation of interesting parts in the task of the other categories is not simple, and we think that it should be the first step for getting good descriptors. However, for the flower task we can make some assumptions, for example the color of the flowers is very important. This is why we collected color-gradient features for all these tasks.
Features: First, from a test photo, 9 new pictures are created corresponding to different color intervals, and one picture for the grey part. Fig. 1 illustrates this step. After this decomposition, gradient-based features were computed on all the 9 picture channels, namely, an intensity histogram, a gradient magnitude histogram, and a gradient angle histogram. These histograms (containing 10 bins each) were also normalized before being used as features in the classification process.
Classification Method: Since in most cases very little training data is available per species, we used nearest neighbor classifier to generate test results. 3
For the experiments, we used 10-fold cross validation method to evaluate our models. As mentioned earlier, for scan-like leaf photos a random forest was trained, while for all the other problems nearest neighbour classifiers were used. 3.1
On the scan-like leaves task we achieved an accuracy score of 55.84%, which
is competitive with the results of the ImageCLEF task’s performance from the
previous year [
The following prior information was used during the aggregating process: – N(s,c): the number of training data for a species and a category (stem, flower, etc.) – R(s,c): the ROC AUC value got from the training database by applying cross-validation for a species and a category – (S1,f1,S2,f2,...)(s,c): the species with which a specific species is frequently confused, and the frequency of this mistake (for a species and a category). After the official evaluation, it turned out, that our aggregating strategies achieved almost the same results, hence it is enough to present only the first aggregation method of the three similar methods here. The aggregation rule is the following: – When the NN classifier of a category ”c” assigns an S0 class label, then a score of 1 is assigned to S0. – Taking the prior information for S0 → (S1, f 1, S2, f 2, ...) contains the frequent mistakes, S1 gets a rank of 1/4, S2 gets a rank of 1/5, etc. – All the previous ranks are multiplied by a Trust (S0,c) factor, which is computed as: (R(S0, c) − 0.5) ∗ (1/(1 + exp(5 − N (S0, c)))). That means that the trust value is much higher when a ROC AUC value is much greater than 0.5, and the number of learning examples is much higher than 5. – This is done for all the photos taken of a plant, and a weighted voting is applied to get the final scores and ranking. 4
The results fit our expectations. Our main target was the scan-like leaf recognition, and the other categories were handled with a lower priority. Table 3 summarises our results on the different categories, Fig. 2 illustrates our results on the complex recognition task, and Fig. 3 shows our results for the category of LeafScan images, which are the most important for us. In the task of scan-like leaves we achieved a good result using features can be computed very efficiently and are described aerlier.
This work has been supported by the European Union and the European Social Fund through project FuturICT.hu (grant no.: TAMOP-4.2.2.C-11/1/KONV2012-0013). Here, our goal was to define simple, but useful and powerful features for scan-like leaf images, and some features based on color and gradient for the other tasks, by which better results can be achieved than those for a trivial classifier. From the results of our experiments, we think that scan-like leaves can be classified quite well, and our aim is to improve the performance on this task by implementing an on-line learning solution based on a voting mechanism like intelligent crowdsourcing.