The paper describes an automatic system for recognition of 10,000 plant species from one or more images. The system nished 1st in the ExpertLifeCLEF 2018 plant identi cation challenge with 88.4% accuracy and performed better than 5 of the 9 participating plant identi cation experts. The system is based on the Inception-ResNet-v2 and Inception-v4 Convolutional Neural Network (CNN) architectures. Performance improvements were achieved by: adjusting the CNN predictions according to the estimated change of the class prior probabilities, replacing network parameters with their running averages, and test-time data augmentation.
The ExpertLifeCLEF [
The data provided for the challenge cover 10 000 species of plants { herbs,
trees and ferns { and consist from:
{ PlantCLEF 2017 EOL: 256K images from the Encyclopedia of Life3 provided
in the 2017 challenge [
The proposed classi cation system builds upon the state-of-the-art Convolutional Neural Network (CNN) architectures, described in Section 2.1. Section 2.3 discusses the use of running averages of the trained network parameters instead of values from the last training step which noticeably increased the accuracy of our models.
The class frequencies in the training data follow a long-tailed distribution.
It is reasonable to expect that the training data, whose signi cant majority was
downloaded from the web, have di erent class prior probabilities than the test
set. In section 2.4 we consider the problem of di erent class prior probability
distributions and implement an existing method [
Section 3 describes the 5 submissions we made. Results of the challenge are presented in Section 4. One of the submitted plant recognition methods achieved the best accuracy among automated systems, and thus placed 1st in the challenge and it outperformed 5 of 9 human experts.
Methodology
Convolutional Neural Networks
The proposed method is based on two architectures { Inception Resnet v2 and
Inception v4 [
Random crop, with aspect ratio range (0:75; 1:33) and with di erent area ranges listed in Table 2, Random left-right ip,
Brightness and Saturation distortion.
At test-time, 14 predictions per image are generated by using 7 crops and their mirrored versions: 1x Full image, 1x Central crop covering 80% of the original image, 1x Central crop covering 60% of the original image, 4x corner crops covering 60% of the original image. 4 https://github.com/tensorflow/models/tree/master/research/slim#
Pretrained Networks #1,..,#6, initialized from the ImageNet pre-trained checkpoints, were rst trained on PlantCLEF data from previous years (PlantCLEF 2017 EOL + PlantCLEF 2017 web + PlantCLEF 2016 subset). PlantCLEF 2017 test set was used for validation.
Another set of networks, denoted as #1clean,..,#6clean, was ne-tuned from models #1,..,#6 without using the noisy PlantCLEF 2017 web set. For this netuning, we also added most of the PlantCLEF 2017 test set, keeping only 1 000 observations (1 403 images) as a min-val set. Preliminary experiments, using the 2017 test set for validation, showed a signi cant improvement in accuracy when using running averages of the trained variables instead of the values from the last training step. Namely we used an exponential decay with decay rate of 0.999.
In this task where majority of the training data is noisy, we interpret this as
keeping a stable version of the variables, since mini-batches with noisy samples
may produce large gradients pointing outside of the local optima. Another
possible interpretation is that the learning rate was still too high. Unfortunately, we
did not have the computational time to experiment with di erent learning rate
schedules.
In many computer vision tasks, the class prior probabilities are assumed to be
the same for the training data and test data. In ExpertLifeCLEF, however, it is
reasonable to assume that class priors change: The largest part of the training
set comes from the web, where the class frequencies may not correspond with
the test-time priors (dependening on the species incidence, the interest of users,
etc.). The problem of adjusting CNN outputs to the change in class prior
probabilities was discussed in [
pe(ckjxi) = p(ckjxi) ppe(c(ckk)p)pe((xxii)) =
p(ckjxi) ppe((cckk)) P p(cj jxi) ppe((ccjj)) / p(ckjxi) ppe((cckk)) ; K j=1 (1)
The subscript e denotes probabilities on the evaluation/test set. The posterior probabilities p(ckjxi) are estimated by the Convolutional Neural Network outputs, since it was trained with the cross-entropy loss. For class priors p(ck) we have an empirical observation - the class frequency in the training set. The evaluation/test set priors pe(ck) are, however, unknown.
We follow the proposition from [
p(es)(ckjxi) = p(s+1)(ck) = e p(ckjxi) p(es)(ck)
p(ck) P p(cj jxi) p(es)(cj ) K j=1 p(cj ) 1 N
X p(es)(ckjxi); N i=1 ; (2)
In our submissions, we estimated the class prior probabilities for the whole
test set. However, one may also consider estimating di erent class priors for
di erent locations, based on the GPS-coordinates of the observations. Moreover,
as discussed in [
Submissions In the challenge, each team was allowed to submit up to 5 di erent run- les with predictions. We used this opportunity to evaluate the following 5 submissions: CMP Run 1 is an ensemble of 6 CNNs: #1clean,..,#6clean described in Section 2.2. This submission used the automatic test set class-prior estimation from the CNN outputs, discussed in Section 2.4.
CMP Run 2 was predicted by the ensemble from Run 1 without class prior estimation on the test data.
CMP Run 3 is an ensemble of 12 CNNs: #1,..,#6 described in Section 2.1 and #1clean,..,#6clean described in Section 2.2. This submission used the automatic test set class-prior estimation.
CMP Run 4 is an ensemble of 6 CNNs: #1,..,#6 described in Section 2.1. This submission used the automatic test set class-prior estimation.
CMP Run 5 is a single Inception-v4 model, denoted as CNN #4clean, using the automatic test set class-prior estimation.
In all runs, the predictions (optionally improved by the class prior estimation) for all crops of the test image are averaged to compute the nal image prediction. Moreover, for observations with several images (connected by the ObservationID values in the provided data), the nal classi cation decision is taken based on the average of all corresponding image predictions. The o cial results of the challenge are displayed in Figure 2. Our system achieved the best results among automatic methods: 88.4% accuracy on the full test set. The best scoring submission was the largest ensemble - CMP Run 3 - using all 12 models. Results of all CMP submissions are listed in Table 3.
When evaluated against human experts in plant sciences, the system (both CMP Run 3 and CMP Run 4) outperformed 5 of 9 tested human experts. That means that in the task of plant recognition from images, machine learning systems reached human expert performance - achieving better accuracy than the median of human experts. The detailed results are displayed in Figure 3.
Interestingly, while ne-tuning on "clean" data slightly improved the recognition accuracy on the full test set, it signi cantly decreased the accuracy on the test subset for human experts. Similarly, test-time prior estimation on the full test set noticeably improved the accuracy, but had an opposite e ect on the subset. We assume that the test subset selected for human experts was too small to provide a representative, identically distributed, sample of the full test set. Therefore the results on the test subset for human experts may be biased towards a small number of species contained in it. 5
Conclusions The proposed machine-learning system for recognition of 10 000 plant species achieved an excellent accuracy of 88.4% in the ExpertLifeCLEF 2018 challenge, scoring 1st among automated systems.
The ensemble of Convolutional Neural Networks bene ted from the following improvements: 1. Adjusting the CNN predictions according to the estimated change of the class prior probabilities. 2. Replacing network parameters by their running averages with exponential decay. 3. Test-time data augmentation.
The experiment with human experts shows that machine learning reached the expert knowledge in plant recognition: our system scored better than an average (median) human expert in plant recognition, achieving better recognition rate than 5 of the 9 evaluated human experts. MS was supported by the CTU student grant SGS17/185/OHK3/3T/13 and by the Electrolux Student Support Programme. JM was supported by The Czech Science Foundation Project GACR P103/12/G084 P103/12/G084. LP was supported by the UWB project No. SGS-2016-039.