We present our deep learning based plant identi cation system in the LifeCLEF 2015. The approach is based on a simple deep convolutional network called PCANet and does not require large amounts of data due to using principal component analysis to learn the weights. After learning multistage lter banks, a simple binary hashing is applied to the ltered data, and features are pooled from block histograms. A multiclass linear support vector machine is then trained and the system is evaluated using the plant task datasets of LifeCLEF 2014 and 2015. As announced by the organizers, our submission achieved an overall inverse rank score of 0.153 in the image-based and an inverse rank score of 0.162 in the observation-based task of LifeCLEF 2015, as well as an inverse rank score of 0.51 for the LeafScan dataset of LifeCLEF 2014.
In recent years, research in the area of automatic plant identi cation from photographs has concentrated around annual plant identi cation competitions that are organized within the CLEF campaigns including ImageCLEF [1{3] and LifeCLEF [4{6]. CLEF is devoted to promoting and evaluating multilingual and multimodal information retrieval systems and the main goal of these competitions is to benchmark the challenging task of content-based identi cation and retrieval of plant species which is of immense importance in botany, agriculture, plant taxonomy, pharmacy, and pharmacology. The task is carried out using images of di erent types of plant parts, such as leaves, branches, stems, owers, and fruits.
Since 2011, competitive submissions for the plant identi cation task have
been made to ImageCLEF and LifeCLEF in which the participating systems
have utilized widely di erent approaches; still, the problem is far from being
solved due to several challenges including large variations in color, illumination,
background, size, and shape. Deep learning approaches are new and suitable for
solving such problems with large amounts of intra-class variability [
There are two tasks within the LifeCLEF 2015 campaign: image-based and
observation-based plant identi cation tasks. The image-based task requires
identi cation given a single image while the goal of the observation-based task is to
identify plants based on multi-image query. The latter corresponds to the
scenario in which a photographer uses the same camera to take snapshots from
di erent views of various organs of a plant species under the similar lighting
conditions and on the same day. The campaign started in 2011 with the
imagebased task covering over 70 tree species, and the observation-based task became
the main track later in 2014. By 2015 [
In this work, we have utilized a system di erent from our previous
submissions [8{11] to recognize plant species using a new deep convolutional network
known as PCANet [
PCANet is a recently proposed convolutional network architecture that combines
the strengths of principal component analysis (PCA) and deep learning [
The general structure of PCANet and our proposed architecture for plant identi cation are presented in this section. 2.1
PCANet initializes by applying principal component analysis to overlapping patches of all images. The selected principal components form the rst layer lters and the projections of the patches on to the principal components form the response of units in the rst layer.
We then repeat this methodology to form a cascaded linear map in the next layers of the deep convolutional network architecture. Next, the method uses binary quantization and hashing for the multi-stage ltered image sets to concatenate them in the decimal form. Finally, local histograms are extracted from the blocks of the quantized images and spatial pyramid pooling method is applied to these histograms to extract features.
The algorithm is explained in detail as follows. The training data contains i = 1; 2; :::; N images Ii of size m n. In the rst stage, patches of size k1 k2 pixels are extracted around each pixel in the image Ii. Afterwards, all such overlapping patches are collected, vectorized, and mean subtracted to obtain Xi. Repeating this operation for all images, we obtain a patch collection X, as: Iil1 = Ii where k kF is Frobenius norm, IL1 is the identity matrix of size L1 L1 and the solution simply consists of nding L1 principal eigenvectors of XXT . Therefore, the PCA lters for the rst layer form weights Wl11 for l1 = 1; 2; :::; L1, by converting the eigenvectors to matrices of size k1 k2. Hence, the l1th ltered image is calculated by convolving the l1th lter with the ith patch-mean removed image, Ii, as, (1) (2) (3) (4) (5) (6)
X = [X1; X2; :::; XN ] 2 Rk1k2 Nmn Next, in order to calculate the desired lter banks of orthonormal lters, V , PCA minimizes the reconstruction error to compute their L1 principal components. The constrained optimization is formulated as:
Til1 =
L2 X 2l2 1H(Iil1 l2=1
Wl22 ) In fact, this conversion maps each L2 binary bits acquired from corresponding pixels of the double ltered binary images into a single graylevel image pixel in the range [0; 2L2 1].
We can repeat the same approach to learn L2 PCA lters for the second layer to create double ltered images. For this purpose, all the overlapping patches of each ltered image Iil1 are collected, vectorized, and mean subtracted to obtain Yil1 . Repeating this algorithm for all ltered images, we obtain,
Y = [Y11; :::; YN1 ; Y12; :::; YN2 ; :::; Y1L1 ; :::; YNL1 ] 2 Rk1k2 L1Nmn Similnadrilny,gPLC2Aprilntceirpsafloerigthenevseecctoonrds olafyYerY, WTla22nfdorrel2ar=ra1n;g2in;:g::;tLhe2m,aares ombattariinceeds by of size k1 k2. Therefore, the double ltered image, computed sequentially using the l1th and l2th lters, is obtained by convolving the l2th lter with the ith patch-mean removed ltered image, Iil1 , as,
Ol1;l2 = Iil1 i
Wl22 As can be seen, in the output O for each image, we have L1 L2 double ltered images with real values. To decrease the number of images, it is proposed to rst binarize them using Heaviside step function H( ). Next, for each pixel, we map L2 quantized binary bits to a decimal number as ̅ ̅ ̅ , , , , ℎ Input Layer
First Stage mean removal – applying PCA filters
Second Stage mean removal – applying PCA filters
Output Layer binarization and mapping – feature pooling
Finally, we partition each of L1 decimal images (Til1 ) into B blocks and compute block histograms (with 2L2 bins) for all L1 images as the features of ith image,
fi = [hist1(Ti1); :::; histB(Ti1); :::; hist1(TiL1 ); :::; histB(TiL1 )] 2 R1 2L2 L1B (7) where histj ( ) indicates the histogram of the jth block of the partitioned image. Utilizing local histograms provides translation invariance in the extracted features. Figure 1 displays the block diagram of a two-stage PCANet. 2.2
In order to process each plant image, color images are converted from RGB
to HSY color space [
For the observation-based task, we applied the Borda count fusion method [
In this section, we will explain the competition datasets and adjust optimal parameters of our proposed system by extracting validation sets from the training data. We then de ne the performance metrics and report the experimental results on the validation and o cial test sets. 3.1
The plant identi cation task in LifeCLEF 2015 involves identifying 1,000 species of trees, herbs, and ferns from photographs of their di erent organs mostly taken within France by di erent users. The collected dataset contains 113,205 pictures, 91,759 images for training and 21,446 images for testing. Table 1 shows the details of the provided datasets and their sample images.
To validate our results, we used the proportionate strati ed random sampling, i.e. we rst randomly split the training dataset into two subsets after placing images of each plant species into both the training and the validation sets. The proportion of validation sets to training sets is assumed to be around 1 to 3. In other words, for any dataset, we randomly selected one-fourth of available samples of each individual class, if possible, as samples of the validation set. We next applied our proposed plant identi cation system for di erent plant categories on the obtained training and validation sets. Table 2 shows the system performance in terms of the obtained rst rank classi cation accuracies. As expected, the ower, fruit, and stem photographs are relatively easier to classify compared to branch, leaf, and entire categories.
LifeCLEF lab itself employs a user-based metric called the average inverse rank score [17] instead of the total classi cation accuracy. The average inverse where U is the number of users who have taken the query pictures; Pu is the number of individual plants observed by the uth user; Nu;p is the number of pictures taken from the pth plant observed by the uth user; and su;p;n is the inverse of the rank of the correct species for the given image, ranging from 0 to 1. Considering this metric, we applied PCANet to the test sets using the learned parameters in the training step and submitted our prediction results to the organizers of LifeCLEF 2015 for o cial evaluation. Table 3 displays the inverse rank scores of our best run in the image-based LifeCLEF 2015 for di erent categories of plant identi cation task. In the observation-based task, our approach using Borda count achieved an inverse rank score of 0.162.
Bearing in mind that this large dataset consists of 1,000 classes of similar categories, our o cial overall rank score of 0.153 in the image-based task shows a fair performance for our submission. Comparing the o cial test results given in Table 3 with the higher accuracies reported in Table 2, we conclude that a considerable amount of over tting existed during validation. This situation could have been improved if we had used more data, but the time complexity was an issue even with the simple architecture.
Furthermore, since a very small subset of the test data in LifeCLEF 2015 belonged to scanned leaves, we skipped the preprocessing and segmentation phase [18] which had been applied to the LeafScan category in our previous submissions [8{11]. Once we applied PCANet to the segmented and preprocessed LeafScan images of LifeCLEF 2014, we achieved the inverse rank score of 0.51 as measured by the campaign organizers. 3.3
We measured the complexity of our system in terms of the running time for feature extraction and training the classi er. On average over all categories, PCANet took 1.37 seconds/image and 6.13 seconds/image for feature extraction and training, respectively. All codes were implemented in MATLAB (run in 80 GB RAM and 2.50 GHz CPU with two processors).
Although the campaign within the LifeCLEF 2015 had allowed using external training data, we restricted ourselves to the provided LifeCLEF datasets. That was due to the reasons that PCANet does not require large data to learn the weights and that using external datasets would increase the processing time for our system. 3.4
Parameters of the proposed PCANet-based plant identi cation system were selected experimentally through validation. Due to combinatorial increase, each time we adjusted only one of the parameters until nding the optimal point. For instance, we observed that by increasing or decreasing the image patch size from 7 7, the performance rapidly decreased. The same conditions were held for the block size of histograms (20 10).
On the other hand, we observed that by increasing the normalization size of input images and/or the number of lter banks (especially within the rst stage), the performance improved. However, there was a trade-o between complexity and accuracy, i.e. by increasing the input image size and/or the number of lters, the accuracy improved slightly while the size of the output feature vectors expanded as well. Consequently, the training time increased drastically. Therefore, we set the system parameters equal to values mentioned in the Section 2.2 to evaluate our system. 4
In this work, we used a simple deep convolutional approach called PCANet in order to identify the plant species within the LifeCLEF 2015 plant identi cation dataset. Our best run has shown a fair performance with overall inverse rank scores of 0.153 and 0.162 in the image-based and observation-based tasks, respectively. It seems that the proposed system is fast and e cient for aligned images such as preprocessed scanned leaves.
Acknowledgments. This project is supported by Turkish Scienti c and Research Council of Turkey (TUBITAK) under project number 113E499. 17. Muller, H., Clough, P., Deselarers, T., Caputo, B.: ImageCLEF: Experimental evaluation in visual information retrieval. Volume 32 of The Information Retrieval Series. Springer (2010) 18. Yanikoglu, B., Aptoula, E., Tirkaz, C.: Automatic plant identi cation from photographs. Machine Vision and Applications 25(6) (2014) 1369{1383