This paper summarizes the results of the BTU DBIS research group's participation in the Personal Photo Retrieval subtask of ImageCLEF 2013. In order to solve the subtask, a self-developed multimodal multimedia retrieval system, PythiaSearch, is used. The discussed retrieval approaches focus on two di erent strategies. First, two automatic approaches that combine visual features and meta data are examined. Second, a manually assisted relevance feedback approach is presented. All approaches are based on a special query language, CQQL, which supports the logical combination of di erent features. Considering only automatic runs without relevance feedback that have been submitted to the subtask, DBIS reached the best overall results, while the relevance feedback-assisted approach is placed second amongst all participants of the subtask.
This paper summarizes the results of the BTU DBIS research group's
participation in the Personal Photo Retrieval subtask of ImageCLEF 2013 [
As in DBIS' participations in various ImageCLEF tasks between 2011 and
2012, the discussed approaches rely on the commuting quantum query language
(CQQL) [
CQQL is an extension of the relational domain calculus, i.e., it can be
directly executed within a relational DB system [
In the scope of this paper, CQQL is used for the matching within the used
multi-modal multimedia retrieval system PythiaSearch [
The personal photo retrieval subtask 2013 is an extension of 2012's pilot task.
The current subtask uses 5,555 image documents that have been sampled from
personal photo collections. In contrast to the pilot phase of the task, 2013's
focus lies on the evaluation of retrieval algorithms using di erent search
strategies and user groups. One objective of the task is to assess whether a retrieval
algorithm's e ectiveness is stable for di erent user groups [
The subtask does not provide any training data. Hence, it has to be solved
ad-hoc. The participants are given multiple query-by-example (QBE) documents
and/or browsed documents and are asked to nd the best matching documents
illustrating an event or depicting a visual concept. In total, 74 topics are
available. In contrast to the last year, the topics are no longer separated into visual
concepts or events [
The interactive retrieval system PythiaSearch [
Let f'(d) be the evaluation of a document d w.r.t. a CQQL query. To construct a CQQL query, various conditions ' can be linked in an arbitrary manner using the conjunction (1), disjunction (2), or negation (3). If ' is atomic, f'(d) can be directly evaluated yielding a value out of the interval [0; 1]. For the scope of this paper, an atomic condition is the result of a similarity measure, e.g., the similarity of the QBE document's color histogram and d's color histogram; or a Boolean evaluation calculated by a DB system or the like.
After a necessary syntactical normalization step [
f'1^'2 (d) = f'1 (d) f'2 (d) f'1_'2 (d) = f'1 (d) + f'2 (d)
(f'1 (d) ^ f'2 (d)) f:'(d) = 1 f'(d) (1) (2)
An example of the arithmetic evaluation of the query that is used in this paper is given in Section 3. In accordance with the Copenhagen interpretation of quantum mechanics, the result of an evaluation of a document d yields the probability of relevance of d w.r.t. the query. This probability value is then used for the ranking of the result list of documents.
Weighting in CQQL In order to re ect the need for the personalization of a
query and as a necessary step for the support of relevance feedback (RF), CQQL
has been extended with a weighting scheme [
The weighting in CQQL is fully embedded into the logical query. That is, a query maintains its logical properties while weights are used. To illustrate, Equation 4 denotes a weighted conjunction, whereas Equation 5 states a weighted disjunction. A weight i is directly associated with a logical connector and steers the in uence of a condition 'i on the evaluation. To evaluate a weighted CQQL query, the weights are syntactically replaced by constant values according to the following rules: '1 ^ 1; 2 '2 '1 _ 1; 2 '2 ('1 _ : 1) ^ ('2 _ : 2) ('1 ^ 1) _ ('2 ^ 2) (4) (5) 2.2
As implied before, the relevance judgement of a query's results is very subjective
with respect to the user's IN. To re ne a subjective IN, PythiaSearch supports
a gradual relevance feedback on the basis of partially ordered sets (posets) [
Internally, a machine-based learning algorithm (a downhill simplex variant)
is used to nd appropriate weight values for a given CQQL query ful lling the
input preferences. The actual algorithm and its properties is described separately
in [
Motivated by CQQL's support for formulating multi-modal queries, DBIS
participated in the 2011 Wikipedia Retrieval task at ImageCLEF [
Our experiments for the Personal Photo Retrieval subtask can be subdivided into two types of runs. First, fully automatic runs which demonstrate the e ectiveness of a CQQL-based logical combination of features with di erent origin, i.e., visual and meta data features. Second, the performance of the aforementioned RF mechanism is investigated (see Section 3.4). 3.1
Over the last years, the DBIS working group conducted a lot of experiments on
various image collections, ranging from the Caltech collections [
PythiaSearch supports the extraction of low-level global and local visual features, e.g., color, edges and texture features or local features like SIFT and SURF. In total, the extraction component o ers more than 30 visual features. Additionally, the system allows the extraction of common image meta data such as Exif, IPTC, or XMP. This meta data, e.g., GPS coordinates, the camera model, or the image orientation extends the variety of features that can be used for the matching of documents. In accordance with the rules of the subtask, IPTC-based data is ignored in the following experiments. Table 1 lists all features that are used in the experiments.
The feature extraction and similarity calculation functionality used by
PythiaSearch resembles the baseline system that is provided by the subtask organizer1.
For a description, see [
Based on preparatory study on the retrieval e ectiveness of various visual lowlevel features and an examination of the subtask's thematic orientation on both visual concepts and events, an appropriate CQQL query had to be de ned. The core idea of the examined CQQL query, which is shown in Equation 6, is to use low-level features that did show a good performance over all six test collections 1 Please note that the organizer of the subtask did not actively participate in the experiments described in this paper nor did he release additional information to the working group that other participants could not obtain. Alas, he carried out many of the pre-studies including the investigation of generally e ective CQQL queries. Furthermore, he had a major impact on the development of PythiaSearch and the underlying learning algorithm. a picture, spatial and temporal proximity as well as a similar camera model are valid indicators. Hence, the core CQQL query is enriched by a person presence condition in form of a Boolean predicate and the aforementioned features derived from Exif meta data.
This concept results in the following CQQL query that uses a weighted conjunction of 18 conditions, whereas all weights are set to 1 initially to express the equal importance of all conditions.
^(ACCsim ; BICsim ; CEDDsim ; ColorHistBordersim ; ColorHistCentersim ; i
ColorHistsim ColorLayoutsim ; ColorStructuresim ; DominantColorsim ; (6) EdgeHistsim ; FCTHsim ; Regionshapesim ; ScalableColorsim ; Tamurasim ; GPSsim ; modelsim ; timesim ; Personsim )
The value of each condition is determined by a distance measure such as the
Euclidean distance of the corresponding feature between the QBE document and
the retrieved document which is then transformed into a similarity measure in the
interval of [0; 1]. Boolean conditions are evaluated traditionally. The calculation
of the GPS coordinate similarity is carried out as we did for the ImageCLEF
2012 Plant Identi cation task [
GP Ssim = 1 p(71:5 (longx (7) (8) (9) (10) (11) 3.3
The 2013 Personal Photo Retrieval subtask provides QBE documents as well as browsed documents. For the approaches without RF, we use the provided data in two ways.
First, we use only the QBE de ned documents (run1 ) because of the fact, that the provided browsed documents can contain misleading information, respectively contain images that did not ful ll the users IN. For the topics for which no QBE document is available, we use all browsed documents instead. Whether the browsed documents are really relevant for the user's IN cannot be determined automatically. Thus, this approach might be a ected by irrelevant input to the retrieval system.
Second, we assume that all documents (no matter if they are QBE or browsed documents) are equally meaningful regarding the user's IN. Hence, we use all documents as QBE documents with no special ranking for the labelled QBE document. This approach is labeled run2 in Figure 4. 3.4
The main objective for the manually assisted approach (run 3 ) is to remove misleading browsed image documents from the initial query and to improve the retrieval quality using the graded RF approach that is described in Section 2.2. The experiment is carried out interactively with the PythiaSearch GUI (see Section 2). Using the aforementioned preference-based approach, irrelevant, relevant documents and the relationship between them can be expressed as a poset. To simplify the user interaction, the GUI o ers 3 levels of relevance and a \garbage can" to collect completely irrelevant documents. Figure 2 shows the three levels where the center (level 1) contains the query document(s). All documents in level 2 are more relevant than documents in level 3 and 4, whereas documents in level 3 are more relevant than documents in level 4. All preferences together create a poset. Documents marked as completely irrelevant are removed from the query results and have a negative impact on the machine-based learning algorithm similar to negative QBE documents.
In order to keep control over the time consumption of submitting 74 queries manually, we have de ned some restrictions for the RF-based experiment.
First, at most one RF iteration is carried out, i.e., we model the behavior of an impatient user.
Second, the assessment of the quality of the results is based on the top-30 results only. This number of documents can be easily inspected without scrolling and requires signi cantly less time than inspecting the top-50 or top-100. Because of this strategy, it may happen that no RF is carried out all because the top-30 results seem relevant to the interacting user.
Third, to simulate a user that avoids a large amount of interaction with the system, a total of 6 images is used to de ne the preferences used during RF.
In general, obviously irrelevant images from the given IN speci cation were removed from the input. Nevertheless, during the submission of all 74 sample queries it was not always possible to identify the IN without background knowledge. In these cases, the RF process is skipped. 4
With reference to the o cial results (see Figure 4), our best run, i.e., run 3, achieves rank 5 in the overall ranking for the average user. Compared to the results of all participants of the subtask, DBIS is ranked second. Focussing on the NCDCG cut 5 we reach about 97%, on NDCG cut 100 about 87% and on MAP cut 100 78% of the best obtained retrieval score.
When only automatic runs are considered, i.e., runs without RF, DBIS achieves the best results. Unfortunately no other runs without RF that use all available modalities (visual and meta data features) and IN information (QBE and browsed documents) were submitted. Due to these circumstances, a resilient interpretation of our results is hardly possible. Anyhow, we assume that the inclusion of meta data helps to distance our from the other approaches. In any case, further information about the techniques used by the other participants is needed.
Generally speaking, the outcome of the presented experiments is fully satisfying. Though, we acknowledge room for improvements for the RF-based run. We assume that an inspection of more than the top-30 results and the inclusion of more preferences might have an impact on the retrieval e ectiveness. Furthermore, a speci cation of the actual IN in textual form will help human assessors during RF because it will enable them to provide RF for every topic. As said before, we could not provide RF for all topics because of the lack of this kind of information. In consequence, we expect an improvement of the RF e ectiveness when this information can be used.
One objective of this subtask is to examine the robustness of a retrieval approach with respect to di erent user groups (e.g. IT experts, non-IT users or gender-speci c groups). Figure 3 shows the variance of the MAP cut 100 scores of our three submitted runs between the di erent user groups. The di erences between all groups is relatively small, e.g., 0.395 vs. 0.425 for run 3. The difference in MAP cut 100 between the best and the worst run is about 7-9%. Interestingly, the results for female users tend to be the best, whereas the average user score tends to be worst. This e ect can also be observed in the results of the other groups. Alas, we are not sure why this e ect is present amongst all groups. The results of our participation in the ImageCLEF 2013 Personal Photo Retrieval subtask are motivating. Although, DBIS achieved a good e ectiveness rank, there are areas that need further research.
First, we plan to analyze single features and meta data in more detail to nd out which features or meta data contributes most to the retrieval quality. As said before, for the RF-supported approach there are various optimizations possible. In particular, the restriction to one RF iteration seems to limit the retrieval quality. First informal experiments show that up to three iterations can give a great performance boost. Furthermore, we assume that the inclusion of RF on all topics will lead to a performance improvement. Another interesting research question is the development of the weight values during the RF iterations in order to reveal whether some features do not contribute to the retrieval e ectiveness at all.