In this paper we summarize the results of our second participation in the ImageCLEF2013 plant identification task. Again we used the combination of low-level features to identify similar pictures, identify adequate matchings and thus learn classifiers. This year, instead of using our workgroup's similarity query language “Commuting Quantum Query Language” (CQQL), we utilized support vector machines (SVMs) to classify the data. So we used classification on split subsets of the data instead of clustering similar results with the k-medoid method. For our experiments we used many di erent parameter combinations and feature combinations on the 2012 and 2013 data to compile four di erent runs.
In this paper we present the participation of the Database and Information
Systems Group (DBIS1) of the Brandenburg Technical University Cottbus,
Germany to the ImageCLEF 2013 plant identification task2 [
For the scope of this paper we take some distance to the CQQL part and try to gain some experience of classification techniques. We want to acquire some basic knowledge and in future combine the mechanisms of CQQL and classification techniques. The basis of our experiments in the plant identification task are still low-level visual features extracted from images.
In our contribution we use our own developed retrieval system PythiaSearch
[
For the task this year DBIS uses both, the last year’s data and experiences as well as the training data from 2013. One goal was to find out whether an optimization on last year data (including training and test data) yields good results. This point is interesting because an analysis of the last year´s data reveals a big di erence of classification results between training and test data.
Our main approach was to use the high dimensional classification technique SVM on high dimensional low-level features. Our submitted runs are based on several studies on di erent combinations of low-level features and various SVM kernels and parameters.
The ImageCLEF Plant Identification Task 2013 is the third iteration of this task and is focused on plant species identification based on images. There are some changes compared to the challenge last year. The number of species has increased from 126 to 250, the number of pictures has drastically gone up from 11572 to 26077. New subclasses were introduced to o er more di erentiation. The exact distribution can be found in the next subsection.
Furthermore the main goal changed from pure classification to a plant
species retrieval task. The data and their identification are described in the
following section. The complete description of the Plant Identification Task 2013
can be found in [
This year the Plant Identification Task is again based on the Pl@ntLeaves data
set [
These two classes are again divided into smaller subclasses. SheetAsBackground contains “Scan” (white paper as background) and “Scan-like” (laying on a table) pictures. “NaturalBackground” are pictures of di erent parts on the plants. It includes the five subclasses: “Entire”, “Flower”, “Fruit”, “Leaf” and “Stem”. The distribution of these subclasses is depicted in table 2.
SheetAsBackground Natural Background training data test data complete data set Entire Flower Fruit Leaf Stem Natural Background training data test data complete data set
The goal of this task is to identify the plant species which is shown on each test image. So for each combination of test image and plant species a prediction should be made, where a prediction is a score value between 0 and 1. The prediction scores of the di erent species for each image have to be ranked in descending order.
To evaluate the predictions the task organizers calculate a score which is
related to the rank of the correct species in the result list. Thereby a mean value
is built per author and plant in the collection. Here, an author is a person which
helped to built up the Pl@ntLeaves collection by contributing pictures and the
corresponding metadata. The score values are calculated with the following
formula [
S =
U 1 X U
1 XPu 1 XNu;p su;p;n u=1 Pu p=1 Nu;p n=1 (1) U : number of users (who have at least one image in the test data) Pu : number of individual plants observed by the u-th user Nu;p : number of pictures taken from the p-th plant observed by the u-th user su;p;n : score between 1 and 0 equals to the inverse of the rank of the correct species (for the n-th picture taken from the p-th plant observed by the u-th user) 2
In this section we give a short introduction to the used techniques of our participation. Firstly, a short description to our retrieval system PythiaSearch is given where the focus lies on the extraction of visual low-level features. Afterwards, the topic of image classification and the state of the art of support vector machiens is described. 2.1
Our retrieval system PythiaSearch is used to extract visual features from the
given plant images. PythiaSearch is a multimedia information retrieval system
supporting multiple search strategies. In order to formulate an information need
(e.g. using Query by Example) the user can choose from multimodal data such
as images, (multilingual) texts and various meta data formats. Additionally, it
features a relevance feedback process that can be used to adjust query results
based on the user’s interaction with the system. A detailed description of the
system can be found in [
PythiaSearch supports the extraction of low-level global and local visual features, e.g., color, edge and texture features or local features like SIFT and SURF. In total, the extraction component o ers more than 30 visual features.
Additionally, PythiaSearch allows the extraction of the most widely spread meta data collections EXIF, IPTC and XMP. These meta data, e.g. GPS coordinates, camera model, orientation, extend the variety of available features. Finally a full featured IR component is integrated.
For the scope of this paper we only extract available visual features. In a preparatory study (see section 3.1) we check the performance of all features and pick the best ones to create SVM models. The features were normalized to the interval [0; 1] so the SVMs can generate better results and the di erent dimensions have similar impact on the score values. 2.2
Classification is an important field in data mining with an ever increasing
relevance for real-world applications. Support vector machines (SVMs) are a
popular and powerful type of classification algorithms which achieves excellent
results. Furthermore this widespread technique can also classify large data sets
rapidly. Currently major database vendors are integrating SVMs and other data
mining capabilities in their systems, e.g. into Oracle Data Miner5 [
baseline_sys.zip
odm/
The basic idea is to map the raw data into a high dimensional vector space
and separate the data using a hyperplane. All data on one side of the hyperplane
is added to class +1, the rest to class 1. There are also several approaches to
generalize this behavior to multi-class classification [
For this paper we used the LIBSVM7 library which is a popular collection
of di erent SVM and regression models, e.g. radial basis functions (RBF), linear
and polynomial SVMs [
In this section we describe the selected features and how they were chosen. Afterwards, the type of SVMs and the final parameter combination are specified. 3.1
As mentioned in our last year’s participation in the plant identification task a good selection of visual features is crucial for a satisfying result. This year the challenge contains more complex and a much larger amount of data, thus reductions of the parameter combinations have to be performed. For our last year’s evaluation it was su cient to calculate a distance measure over the feature data. This year the SVM works directly on the raw feature data. Due to the fact that some features have a variable length it was impossible to use them directly as input for the SVMs. Another challenge was revealed in preliminary tests. We found out that feature combinations can lead to strongly varying results when changing the SVM kernels and parameters. This complicates obtaining a good model and increases the spanned space of feasible parameters combination enormously.
Nevertheless we used the last year’s experiments [
To construct the final runs we used SVM multi-class classification by applying the SVM library LIBSVM. A broad range of parameter combinations were tested and the best resulting runs selected. Parameters were amongst others the SVM type, kernel type, weights and costs for construction and movement of the hyperplane.
run o cial run name run1 1368038646069__DBISForMaT_run1_train2012_svm_Scan4_Photo2_1_2_3 run2 1368038721036__DBISForMaT_run2_train2012_svm_Scan12_Photo4_-_1_4 run3 1368045672892__DBISForMaT_run3_crossval2013_svm_feature4_config60_1_2_3 run4 1368045820175__DBISForMaT_run4_crossval2013_svm_feature5_config80_Photo14_1_3_3 The pictures were split into six subclasses, one for each of the five subcategories of ”Natural Background” and the sixth for pictures in ”SheetAsBackground”. For each of these six classes di erent models were learned.
We assign the test images deterministically to the predicted classes. That means we choose the best class (plant species) and ranked it #1 with confidence score 1.0, the remaining classes (species) are not present in the result for that picture, meaning the confidence score is 0.
We published four runs for this task. The association between the internal
and the o cial names of the runs is presented in table 4. The o cial names are
important to find these runs on the task website [
All resulting runs are computed without probability estimates because in our tests they archived worse results than the deterministic runs and the computation time is partly prohibitively high. For all runs the shrinking heuristics supplied by LIBSVM are used because it is recommended by the LIBSVM’s authors. This parameter seems to have no or a negligible e ect on the result. The parameter epsilon_loss_function was not set because it is only needed in the SVM type epsilon-SVR which is a regression type algorithm. For the plant identification task we needed no regression but classification. For our first two runs we used the score calculation script of the 2012 plant identification task using the 2012 training and test data to build classifiers. Our hope was that this script works analogously to its counterpart from the 2013 task. These approach holds two advantages: We learned with the ground truth of other data which can hopefully construct more general models reducing overfitting. Furthermore, using the score computation we can avoid problems which could arise when score computation and SVM classification results do not match completely. We chose the best performing parameter combinations for each subclass and applied the corresponding models to the 2013 test data. The parameters used for the first two runs are depicted in the tables 5 and 6. 3.4
In contrast to the first two runs run3 and run4 are constructed using the 2013 data set. This is done by performing 5-fold cross validations on the training data using many di erent parameter combinations. For this we computed the results and chose two promising feature combinations. The corresponding learned models are applied to the test data producing run3 and run4. The parameters used for the last two runs are depicted in table 7. In this section we compare the o cial scores of our four runs (see table 8) with the results of our competitors. For two of the five “Natural Background” subclasses we achieved good results. We were ranked the 4th best group for “Flower” (see figure 1) and 3th for “Entire” (see figure 2). Unfortunately, in the aggregated rank for “Natural Background” we were just 5th (see figure 3) because the results for the other three classes “Fruit” (5th), “Leaf” (7th) and “Stem” (6th) were just moderate. In the category of Scans and Scan-like photos of leafs (class “SheetAsBackground”) we were ranked 6th over all runs and 5th when just considering the automatic runs (see figure 4).
From our competitors the groups Inria and NlabUTokyo were always ranked better than we were and Sabanci Okan in most categories. This could be motivated in their additional experience and knowledge in the plant identification task10 or superior classification strategies. After release of the working notes 8 feature combination: FC_05 for class “Natural Background”, FC_14 for class “SheetAs
Background” 9 coef0: -0.5 for class “Entire”; -0.2 for class “Fruit”; -1 for other classes 10 Inria and Sabanci Okan also had excellent results in the 2011 and 2012 versions of the plant identification task. we will compare the di erent classification approaches for better results in the future.
We are pleased with the results of the class “Entire” because it is a di cult class, but also important for real-world applications. The poor results for “Leaf” (with natural background) and “SheetAsBackground” (scanned leafs) could be caused by the extensive research for this use case by other groups. Another reason could be the selection of wrong or not enough features for the di erent subclasses. Here, contour and shape features should promise better results.
It is interesting to observe that the results of the two di erent approaches for the run parametrization – training with the o cial score computation on the 2012 data and cross validation on the 2013 data – led to similar results. So the runs 1 and 3 have selected feature combination FC_04 for the class “SheetAsBackground” and the runs 2 and 3 selected FC_04 for the “Natural Background” subclasses. This means with the setup used by us both strategies can be used without losing too much classification success. Aggregated over all four runs run2 was the best, run1 the worst. This could mean that training with the last year’s ground truth has the potential to be better than with just the current training set, but the results and the small number of runs means that this result is not statistically significant and needs further research. Instead of using a weight-learning algorithm with CQQL we used SVMs to classify data this year. The above-average results show that this strategy is viable and further research in the parameter and feature selection should be considered.
In our 2012 participation [
While performing the task and after evaluating the results many possibilities for future changes and enhancements of our approach emerged. In order to obtain a deeper understanding of the classification results we will rerun the tests using the 2013 version of the score computation when available. We are interested whether we will achieve better results with other SVM parameter combinations. For this year’s task we used deterministic SVMs because the probabilistic runs achieves a lower percentage of correct classifications on the 2013 data and lower score values on the 2012 data set. Because the 2013 data set is very di erent to the last year’s version (see section 1) a probabilistic rerun with score computation of the 2013 makes sense. For both tests we need to wait for the ground truth and 2013 score computation script.
The usage of more and a larger spectrum of features could benefit the result. We could e.g. use good performing features described by other participants or more standard features. Furthermore instead of using several fixed feature combinations we could also test all reasonable feature combinations, train, test and choose the best one for each image subtype. Unfortunately, this procedure is very time-consuming and could also lead to overfitting of the classifiers.
For this task we restricted the usage of features to low-level visual features. We could improve our results by using also the metadata of the images, e.g. the GPS coordinates, time of shooting and possibly the photographer.
A further approach is the combination of SVM with the weighted learning
approach of our 2012 contribution [
In the next year we will try to generate an own CQQL kernel which can be easily used within LIBSVM. This kernel shall provide the capabilities of quantum logic an enable us to combine the di erent low-level features. We hope to gain more possibilities to adjust the training set and so separate the data in a more convenient way.