The PlantCLEF 2016 challenge focused on tree, herb and fern species identification based on different types of images. The aim of the task was to classify the plants in the images to species and to give a confidence score depicting the probability that a prediction is true. We elaborated different classification methods for this challenge. We applied dense SIFT for feature detection and description; and Gaussian Mixture Model based Fisher vector was calculated to represent an image with high-level descriptor. Fisher vectors were classified by a special SVM, the C-support vector classification algorithm with RBF (Radial Basis Function) kernel. Furthermore, we applied deep learning method to train convolutional neural network (CNN) for feature learning and fullyconnected layers with softmax output for classification. We also combined these classifiers using the weighted average of their outputs. The final results show that the CNN achieved better result than the SVM, and the combined method slightly surpasses the CNN.
Being able to identify the different species of plants growing in agricultural
areas and to automatically detect the presence of invasive species is crucial.
Identifying plants is usually a difficult task, sometimes for professionals (such
as farmers or wood exploiters) as well. Using content-based image retrieval
technologies is a promising possibility in this scenario. In order to solve it a
challenge is announced in the LifeCLEF campaign [
The image-based plant identification task, briefly PlantCLEF 2016 [
The aim of the task was to classify images into the known categories (species), but the classification system had to be robust to unseen categories. It was a more difficult problem, because the test set contained images of species that were not in the training set (these are unseen categories). Besides the images contextual metadata (date, location, author and rating information) were also available. 2
In the last two years there have been a number of successful deep learning,
SVM and combined solutions in the LifeCLEF competition. In 2014 a
combined system of convolutional neural nets and SVM won the challenge [
We elaborated fully automatic methods (one by Fisher vectors and SVM, and another one by deep learning) for the classification of the images, and then they are taken in decreasing order based on reliability of classification decision; the next sections will present the details. 3
The first part of the classification was the representation of each image based
on visual content. Following the general trend, we applied BoW
(Bag-ofWords) model [10⎼12] for this purpose. This consists of three steps: (i)
feature detection, (ii) feature description, (iii) image description as usual phases
in computer vision and we solved these steps similarly to our previous work
[
For feature detection and description we used the SIFT (Scale
Invariant Feature Transform) algorithm [
For the classification subtask we used a variation of SVM (Support
Vector Machine), the C-SVC (C-support vector classification) [
The results of SVM classification was submitted as ‘BME TMIT Run2’. 4
Nowadays state-of-the-art image recognition and classification solutions
generally use deep learning methodology. Deep convolutional neural networks
are able to learn the descriptive features of the image database in many
abstraction levels. Convolutional neural networks raised a lot of interests in
2012, when a team led by Geoffrey Hinton and Alex Krizhevsky won the
ImageNet Large Scale Visual Recognition Competition [
In the data preparation phase we applied cropping, scaling and normalization. Hence we only cropped the center of the images along the shorter dimension and scaled it down to the network’s input dimension, which is 224x224 pixels. Finally, we normalized the red, green and blue color channels individually to zero mean and unit variance. An example of the resulting image is shown in Figure 1.
Batch norm.
Max-pooling 3x3, s:2x2
Batch norm.
Max-pooling 3x3, s:2x2
Batch norm.
Max-pooling
3x3, s:2x2
For training the PlantCLEF 2015 database we used a modified version of
AlexNet [
The hardware we used for training were a NVidia GTX 970 (4 GB) and a NVidia Titan X (12 GB) GPU cards hosted in two i7 servers with 32 GB RAM. Ubuntu 14.04 with Cuda 7.5 and cuDNN 4.0 was used as general software architecture. For data preparation, training and evaluating deep neural networks the Torch7 [26] deep learning framework was used. For calculating mean average precision (MAP) values the sklearn Python package was used.
The results of CNN classification was submitted as ‘BME TMIT Run1’.
We combined the outputs of the classifiers. Based on our preliminary testing the weighted average of the outputs were used for creating the ‘BME TMIT Run4’. Besides two classifiers described above a third one was constructed using metadata. After normalizing and cleaning the data we calculated new metadatas (e.g. the season) in order to get more informative variables. Next we applied Random Forest (RF), and we measured the accuracy. The metadata based classification had the lowest MAP value, SVM and CNN gave much better results. Therefore, we chose the following weight parameters: 0.1 for metadata, 0.3 for SVM and 0.6 for CNN.
Furthermore, we built an aggregated model to detect unseen categories by attempting to filter out the images with unknown classes. We measured the largest distance among fisher vectors of training images; in the case if a test image’s fisher vector is farther from all training fisher vectors then this distance, we reject that particular image (as outlier). As well as, images with very low overall (0.3) decision values were also rejected. As a result, only the remaining test images were included in ‘BME TMIT Run3’. 6
We trained 1 CNN and 7 SVM classifiers (one for each viewpoint) and we conducted a preliminary testing on the PlantCLEF 2015 test data, and measured the MAP (Mean Average Precision) values. The results of the preliminary evaluation can be seen in the first row of Table 2.
Some of the test images had no viewpoint attribute at all, and some of them were marked as ‘Other’. Therefore, an image with known viewpoint was classified with the appropriate SVM classifier (with same viewpoint), and decision values of this classifier were used in the runfile as predictions. For testing an image with unknown viewpoint we constructed a classifier using the weighted average of the decision values coming from all trained classifiers. At the estimation of weight parameters we took the “goodness” of different viewpoint classifiers into consideration: LeafScan: 0.3, Leaf: 0.15, Flower: 0.15, Fruit: 0.15, Stem: 0.15, Branch: 0.05, Entire: 0.05.
In Table 2, we presented multiple MAP (Mean Average Precision) values based on the size of the sorted lists associated with the classes. The results of the post-testing (i.e. test was after the run submission) of SVM and CNN show that the best MAP value is reached when only the 10 most probable predictions are involved in the calculations. Unfortunately, we analyzed the influence of MAP calculations (second and third rows of this table belong to post-testing) only after the official results were released. For the competition we submitted predictions of all the 1000 classes for every test image that caused worse MAP value.
The evaluation was executed on the PlantCLEF 2015 test data, and we calculated the predictions on 2016 test data. It is important to note that these runfiles contained the total results, which means that we gave all decision values for each ClassId for each MediaId. In the official evaluation MAP was used for measurement of goodness of the image classification, considering each class Ci of the training set as a query. In this query evaluation all predictions with ClassId=Ci in the runfile were extracted, and ranked by decreasing probability and the average precision (AP) was computed for that class. The MAP is mean of these AP values. To evaluate more specifically the targeted usage scenario consisting in detecting invasive species, a secondary MAP was computed by considering as queries only a subset of the species that belongs to a blacklist of invasive species. Recognition system was expected to be robust to unseen categories by automatically detecting the numerous false positives classification hits. The official results can be seen in Table 3, where the first two columns contain the unseen categories, while the last column ignores them. We elaborated different classification methods for image-based plant identification task. We applied dense SIFT for feature detection and description; and Gaussian Mixture Model based Fisher vector was calculated to represent an image with high-level descriptor. The chosen classifier was the C-support vector classification algorithm with RBF (Radial Basis Function) kernel, and we optimized two hyperparameters (C from C-SVC and γ from RBF kernel) by a grid search with two-dimensional grid.
We also used convolutional neural networks for the task. The images were normalized to zero mean and unit variance, they were also cropped and scaled down. We used a modified version of the AlexNet model, and we performed batch size optimization. With the winning batch size the deep learning method achieved considerably higher MAP score than SVM.
We constructed a classifier by combining the decisions values of the metadata, SVM and CNN classification methods. The weights of these techniques were determined based on the preliminary tests. Furthermore, a novel approach was used for rejecting the outlier test images (i.e. images with unseen categories), this approach used the information coming from both distance measurement of Fisher vectors and CNN.
It should be noted that our team was formed at the middle of March and we started working on the project in April. According to the investigation of MAP calculations, we must admit that if we would have optimized the length of the sorted lists for the MAP calculations, we might have achieved better results in the official competition.
Bálint Pál Tóth gratefully acknowledges the support of NVIDIA Corporation with the donation of an NVidia Titan X GPU used for his research. 23. He, K., Zhang, X., Ren, S., & Sun, J. Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification. arXiv preprint arXiv:1502.01852 (2015) 24. Ioffe, S. and Szegedy, C. "Batch normalization: Accelerating deep network training by reducing internal covariate shift." arXiv preprint arXiv:1502.03167 (2015) 25. Zeiler, M. D. "ADADELTA: an adaptive learning rate method." arXiv preprint arXiv:1212.5701 (2012) 26. Collobert, R., Kavukcuoglu, K., & Farabet, C. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop (No. EPFL-CONF-192376). (2011)