In this work, we summarize the results of our rst participation in the plant identi cation task. Unlike other contributors, we present a rather untypical approach, which does not rely on classi cation techniques. In contrast, logical combinations of low-level features expressed in a query language are used to assess a document's similarity to a species. Similar to ImageCLEF 2011, DBIS' approach is based on the commuting quantum query language (CQQL). CQQL was proposed by the workgroup to combine similarity predicates as found in information retrieval and relational predicates common in databases. In order to combine both predicate types, CQQL utilizes the mathematical formalisms of quantum mechanics and logic eventually forming a probabilistic logic. To test the utility of our query language, three di erent automatic approaches are discussed. First, a query by example approach towards plant identi cation is presented. Second, the approach is combined with a kmedoid technique to exploit relationships within the top-k results. To conclude with, the aforementioned techniques are compared with the utilization of the k-medoid method alone. With respect to the non-existent experience with the task, the results of the discussed approach are fairly decent but leave room for improvement being outlined as future work.
In this paper we present the results of the Database and Information Systems Group's (DBIS) participation in the plant identi cation task that was organized within ImageCLEF 2012. Because it is our rst try with the Pl@ntLeaves data set, our main objective was to gain experience with the data set and to investigate future directions of research. Unlike other contributors, we present a rather untypical approach, which does not rely on classi cation techniques. In contrast, logical combinations of low-level features expressed in a query language are used to assess a document's similarity to a species.
Similar to ImageCLEF 2011 [
The core idea of the presented approach can be summarized as follows. Based
on di erent document representations, e.g. color-based low-level features or
accompanying metadata such as GPS information, a logical CQQL query is
formulated, e.g. to de ne the species' characteristics. Based on the CQQL evaluation
rules describes in prior work [
The plant identi cation task is part of ImageCLEF for the second time. It is
focused on tree species identi cation based on leaf images. In comparison to
the last year's challenge there are some novelties. The number of species has
been increased from 70 to 126. Furthermore the main objective changed from
pure classi cation to a plant species retrieval task. In the following section we
describe the basic characteristics of the data and their identi cation as far as it
is needed for the understanding of this paper. The complete description of the
plant identi cation task 2012 be found in [
Training and Test Data The plant identi cation task is based on the
Pl@ntLeaves data set [
Scan Scan-like Photograph P 4,870 1,819 1,733 8,422 1,760 907 483 3,150 6,630 2,726 2,216 11,572 Identi cation and evaluation The goal of this task is to identify the tree species, whose leaf is depicted on a given test image. In consequence, for each test image, a prediction should be made for each of the 126 plant species. For the scope of the task, a prediction is a score between 0 and 1 expressing the con dence that a given sample images belongs to a given species.
To evaluate the prediction quality, the task organizers calculated a score
which is related to the rank of the correct species in the result list. Thereby a
mean value is built per author and per plant which is in the collection. An author
is a person which helped to built up the Pl@ntLeaves collection. The score is
de ned in the following formula [
Retrieval Model
f'1^'2 (d) = f'1 (d) f'2 (d) f'1_'2 (d) = f'1 (d) + f'2 (d)
(f'1 (d) ^ f'2 (d))
f:'(d) = 1
f'(d)
(1)
(2)
(3)
U : number of users (who have at least one image in the test data)
Pu : number of individual plants observed by the u-th user
Nu;p : number of pictures taken from the p-th plant observed by the u-th user
su;p;n : classi cation score (1 or 0) for the n-th picture taken from the p-th plant
observed by the u-th user
Repeating our summary found in prior work [
Given that f'(d) is the evaluation of a document d w.r.t. a CQQL query q. To form q, various conditions ' can be linked in an arbitrary manner using the conjunction (Equation 2), disjunction (Equation 3), or negation (Equation 4). If ' is atomic, f'(d) can be directly evaluated yielding a value out of the interval [0; 1] As stated before, the actual value of a representation can be calculated by a similarity measure or a Boolean evaluation carried out by a DB system or the like.
After a necessary syntactical normalization step [
In order to steer the in uence of certain conditions onto the query evaluation,
CQQL has been extended with a weighting scheme [
Equation 5 denotes a weighted conjunction, whereas Equation 6 states a weighted disjunction. A weight i is directly associated with a logical connector and steers the in uence of a representation 'i on the evaluation. To evaluate a weighted CQQL query, the weights are syntactically replaced by constant values according to the following rules: '1 ^ 1; 2 '2
('1 _ : 1) ^ ('2 _ : 2) '1 _ 1; 2 '2 ('1 ^ 1) _ ('2 ^ 2) (5) (6) 3
For the scope of this paper, experiments have been conducted on low-level features combined with some of the provided metadata. The discussed approach aims at improving the performance on the plant identi cation task by using combinations of di erent features. Hence, the three main objectives of the experiments are as follows:
First, we will investigate and optimize the e ciency of a combination of visual low-level features alone.
Second, the performance improvement using low-level features as well as metadata in a CQQL query will be examined.
Third, the discussed CQQL approach will be compared with other state-ofthe-art systems using classi cation systems like support vector machines (SVM). In order to investigate these points, three di erent approaches are used in combination with a preparatory study (see Section 3.1).
Section 3.2 describes the results of a query by example (QBE) approach, while Section 3.4 presents a solution of the plant identi cation task using a k-medoid clustering technique. Section 3.3 combines both approaches.
In order to conduct the experiments, a preparatory study has been carried out. All assumptions used later are based on this study.
To conduct the preparatory study, we used our own developed multimodal
retrieval system [
To get an overview of the performance of the individual low-level features we measured the accuracy of them with the Pl@ntLeaves data set. For our rst participation we decided not to evaluate each image type (scan, scan like, photographs) separately. An excerpt of the results of our initial evaluation runs is shown in Figure 1 displaying the values of precision at 5, 10, 20, 30, and mean average precision (MAP). Overall, we tested 15 low-level features in addition to the given GPS information. To calculate the similarity between the GPS data of two images we used the following similarity measure:
GP Ssim = 1 p(71:5 (longx longy))2 + (111:3 (latx 6378:388 laty))2 (7) whereas long stands for longitude and lat for latitude.
The MPEG-7 Color Structure Descriptor (CSD) [
Based on the results of Figure 1, we tried to nd di erent CQQL
combinations that would exceed the performance of the low-level feature runs. Our rst
combinations were based on our former evaluations with general purpose image
collections like Caltech 101 [
Consequently, new combinations using the best performing low-level visual features were examined. An excerpt of the used CQQL combinations can be found in Table 2.
Test Design We evaluated the performance using the given training data but using MAP and Precision at n evaluation metrics instead of the special plant identi cation metric. In total, 14 CQQL formulas containing visual low-level features and two combinations of visual and GPS data were tested. To evaluate the CQQL combinations, we used a Z-Score normalization for each feature similarity and the CQQL evaluation rules (see Formulae 2, 3, and 4). Together with the ground truth of the training data and our preference based approach we tried to optimize the weights used in the CQQL formulas in order to improve the retrieval metrics. Initially all weights ( i) are set to 1. After a learning run, the weights are set to values between 0 and 1 that ful ll the most preferences given by the ground truth. An excerpt from the nal evaluation results can be found in Figure 2.
The best visual-only run gives us a small performance boost of about 14% with a MAP value of 0.39 and a P@5 of 0.75 in comparison to the best single feature ColorStructure. In contrast, the best multimodal CQQL combination (UCC08) gives a clearly greater performance boost of about 39% with a MAP value of 0.47 and P@5 of 0.8. It should be noted that we use no optimization for each image type (scan, scan like or photograph) which should improve the values even further. We used a query by example (QBE) based approach for run 1 and 2 (see Figure 3). For this approach, test images are considered QBE documents and the training images form the collection to be used for retrieval.
The only di erence between both runs of the QBE approach is that the rst run (our main run) uses a multimodal CQQL combination (UCC08) whereas the second run is based on a CQQL combination consisting only of visual features (UCC06).
Based on the preparatory study, weighted CQQL queries were de ned using the best performing features. To learn the actual weight values, we picked a small random sample of images of all image types (scans, scan-like and photographs). With this training data, we evaluated the initial retrieval performance. To learn the best weights for a query, three steps were carried out: 1. A set of QBE images was chosen randomly from the training data. The QBE set includes all image types and some species which have the highest frequency in the training data. 2. For each of the QBE images, a ranking is calculated using a distinct set of weight values. 3. The results were interpreted using precision at n and MAP to reveal the best weight setting.
In order to nd out the optimal weight setting, each QBE image is used to retrieve similar images from the collection eventually generating a ranking. As we know the species for each QBE document the ranking can then be compared with the optimal ranking of a species. This comparison is needed for the automatic preference input in order to learn weights with the presented approach. In order to provide preferences, the rst 500 (at most) documents of the generated ranking are checked for irrelevant documents preceding relevant ones regarding the examined species de ned by the current QBE document. If such an order dirrelevant > drelevant is detected, and inverted preference dirrelevant < drelevant is de ned. After the rst 500 (at most) documents have been tested, these preferences serve as input for the weight learning algorithm. Using these weights, a new ranking is generated. This ranking is then evaluated for the aforementioned retrieval metrics. To conclude with, all runs are averaged to determine an average set of weight values for a given query. 3.3
For our second test (run 3), we used an approach combining a QBE-based approach with image clustering. The rst part of this approach uses the same techniques as described in the last section. To reveal the relationship of all images found in the top-k result, we applied an image clustering method. We expected to nd a homogeneous group of images which could then be used to determine the species of the query document. Therefore we used a distance based clustering approach, a k-medoid clustering. It was necessary to use such a method because we want use the CQQL similarities between all top-k images.
The result of the k-medoid clustering is a set of clusters. A cluster contains a mixture of training and test data (cluster members). To make a prediction which species should be associated with a given test image, we inspect the cluster containing the test image. Analyzing the rest of the cluster members (considering only training images of which the species is known), we check to which species the training data belongs. Taking the distance between the test image and the nearest training image, a ranking is created. The ordered list gives us the probability of a membership for all species within the cluster. To get a prediction for each type of species, we take the other clusters and continue following the same principle. To optimize our approach, we evaluate which values for top-k and the number of clusters reached the best quality. The idea of the used quality measure is straightforward: a correct classi cation on the rst predicted species will de ne a score value of 1. If the correct species is on the n-th position we de ne a score value of n. Then a minimized summed score value de nes the best quality.
The usage of this combined approach gives us the opportunity to use the frequency of occurrence of the plant species. In the next approach, multiple occurrences of a species are ignored. Instead, only the rst appearance is regarded important. Considering all top-k documents, we expect some species to occur repeatedly. A clustering on these images (with multiple occurrences of a species) yields a (good) probability that images with the same species belong to the same cluster (because of their high similarity). For a given test image which belongs to the same cluster there is a high probability that this test image belongs to the species too. The handling of images which occur very rarely or never in the training data set is complicated because our prediction is unde ned. 3.4
For our last try (run 4), we used a pure image clustering approach. Unlike our rst methods we do not calculate an individual species prediction for each test image. Instead, we cluster the complete data set including the train and test images. We used a k-medoid clustering approach, which works on distances, calculated by a logical combination of global visual features combined with GPS data (UCC08). In this run, we used no special optimization techniques, so we apply initial weights to the CQQL formula. The species prediction of each test image is similar to the method used in the top-k clustering. For a given test image we analyzed the result list of the cluster which holds these object. To make a prediction for all categories we analyzed the nearest clusters and took all unused categories like we did in top-k clustering approach.
With this approach we are able to exploit the relationship of the test images and training images as additional information to identify the species. But without an equal distribution of all species we get some problems with categories which occur rarely as we already discussed in last section. 4
According to the o cial results, our best run achieved place 20 in the overall ranking. Considering only automatic runs, we achieved rank 11. At a closer look, the score values of our results for scans and scan-like photographs are very poor. We cannot fully explain the failure with these image types as there were no indications for the weak performance during the training stage. One reasonable interpretation of the results is that the conducted learning runs led to an overtting. Another reason could be the gap between the nal metric used in the task and the metrics (precision at n, MAP) used during our preparatory study. A detailed analysis is not yet possible until the exact score calculation method and the ground truth of the test data is released. To conclude with, further research has to be carried out to nd satisfying answers.
Regarding our general approach and the results in the photograph category we obtained a decent rank 6 (3rd best group) considering only automatic runs as illustrated in Figure 3. The fact that our run based on visual features alone (run 2) performed better or as well as the combined run (run 1) for all image types is surprising. This is contradictory to the results we had observed during the training runs.
In comparison to the best submission, our results are far o . One reason for the large distance to the best group can be the training which was realized for all three image types in the same run, while other teams tried to di erentiate. 5
Our participation on the ImageCLEF plant identi cation task poses a lot of questions. The results are not devastating for our rst participation. Anyhow, we are not satis ed because the results during the training were auspiciously. Furthermore, our run relying on visual features alone performed best and the di erences between scan, scan-like and photographs were very small. Further research might reveal the reasons. For our next participation, we have to consider some general points that were neglected. First, we learned that concentrating on each individual image type de nitely improves performance and should be incorporated in our approach. Additionally, we acknowledge that a reasonably working image segmentation is important for natural photographs in order to extract shape features which could be included into our retrieval model.
This research was supported by a grant of the Federal Ministry of Education and Research (Grant Number 03FO3072).