This paper presents the participation of the Pattern Recognition and Application Group (PRA Group) and the Ambient Intelligence (AmILAB) in the ImageCLEF 2012 Personal Photo Retrieval Pilot Task. This is a pilot task that aims to provide a test bed for QBE-based retrieval scenarios in the scope of personal information retrieval based on a collection of 5,555 personal images plus rich meta-data. For this challenge we used Image Hunter, a content based image retrieval tool with relevance feedback previously developed by ourselves. The results show that we obtained good results by taking into account that we used only visual data, moreover we were the only one that used relevance feedback.
The personal photo retrieval is a pilot task introduced in the ImageCLEF 2012
competition. This pilot task provides a test-bed for query by example (QBE)
image retrieval scenarios in the scope of personal information retrieval. In fact,
instead of using images downloaded from Flickr or other similar web resources,
the dataset proposed re ects an amalgamated personal image collection that has
been taken by 19 photographers. The aim of this pilot task is to create an image
retrieval scenario where a normal person (i.e., not expert in image retrieval tasks)
searches in its personal photo collections some \relevant" images, i.e. the search
for similar images or images depicting a similar event, e.g. a rock concert. This
pilot task is divided into two tasks: retrieval of visual concepts and retrieval of
events. A detailed overview of the dataset and the task can be found in [
We took part only to the Task 1 of this competition. In this task we have a
set of visual concepts with ve QBE associated to be used in the retrieval process
to retrieve relevant images. To perform the task we used Image Hunter, a content
based image retrieval tool with relevance feedback previously developed at the
AmILAB [
With the aim of building a practical application to show the potentialities of Content Based Image Retrieval tools with Relevance Feedback, we developed Image Hunter. Image Hunter is a prototype that shows the capabilities of an Image Retrieval engine where the search is started by an image provided by the user, and the system returns the most visually similar images. The system is enriched by Relevance Feedback capabilities that let the user specify which results match the desired concepts through an easy user interface. 2.1
Architectural description
This tool is entirely written in JAVA, so that the tool is machine independent.
For its development, we partially took inspiration from the LIRE library [
The main core of Image Hunter is a full independent module, thus allowing the development of a personalized user interface. A schema of the whole system can be seen in Figure 1.
The core of the system is subdivided into three main parts: { Indexing and Lucene interface for data storing; { Feature extraction interface; { Image Retrieval and Relevance Feedback.
The Indexing part has the role of extracting the visual features and other informations from the images. The visual features and other descriptors of the images are then stored in a particular structure de ned inside Image Hunter. The tool can index di erent types of image formats and it can be built in an incremental way. Lucene turned out to be well suited for the storage needs of Image Hunter. The core of Lucene's logical architecture is a series of document containing text elds, where we have associated di erent features ( elds) to each image (document).
As we said before, for the feature extraction we took inspiration from the
LIRE library that it is used as an external feature extraction library. We
expanded and modi ed its functionalities by implementing or reimplementing in
Image Hunter some extractors. The Feature extraction interface allows to
extract di erent visual features based on di erent characteristics: color, texture
and shape. They are:
{ Scalable Color [
One of Image Hunter's greatest strengths is its exibility: in fact, its structure was built in a way that it is possible to add any other image descriptor. The choice of the above mentioned set is due to the \real time" nature of the system with large database. In fact even if some local features such as SIFT or SURF could improve the retrieval performance for some particular kind of searches, on the other hand they are more time expensive in the evaluation of the similarity between images.
The core adopts three relevance feedback techniques [
The user interface is structured to provide just the functionalities that are strictly related with the user interaction (e.g., the list of relevant images found by the user).
Image Hunter employs a web-based interface that can be viewed and experienced at the address http://prag.diee.unica.it/amilab/WIH. This version is a web application built for the Apache Tomcat web container by using a mixture of JSP and java Servlet. The graphic interface is based on the jQuery framework, and has been tested for the Mozilla Firefox and Google Chrome browsers. The Image Hunter homepage let the user choose the picture from which starting the search. The picture can be chosen either within those of the proposed galleries or among the images from the user hard disk. In order to make intuitive and easy the features o ered by the application, the graphical interface has been designed relying on the Drag and Drop approach. From the result page, the user can drag the images that her deems relevant to her search in a special box-cart, and then submit the feedback. Then the feedback is processed by the system, and a new set of images is proposed to the user. The user can iterate the feedback process as many times he/she wants. Figure 2 summarizes the typical user interaction within Image Hunter.
Example-Image chosen by a user to perform a search in a Digital Library
Compute the similarity between the query imageand the images in the database
After 3 Compute the iteractions new query using the user's hints
Relevant images found at the previous steps relevant not-relevant The system outputs the results
The user drags the relevant images into the
relevant's box for the relevance feedback
In this section we brie y describe the two relevance feedback techniques
implemented in the core that we have used in this competition. The use of the
nearest-neighbor paradigm is motivated by its use in a number of di erent
pattern recognition elds, where it is di cult to produce a high-level generalization
of a class of objects, but where neighborhood information is available [
n=t 1 + n=t relBQS (I) +
relNN (I) 1 1 + n=t where n and t are the number of non-relevant images and the whole number of images retrieved after the latter iteration, respectively. The two terms relNN and relBQS are computed as follows: relNN (I) = kI
kI N N r (I)k + kI
N N nr (I)k
N N nr (I)k where N N r(I) and N N nr(I) denote the relevant and the non relevant Nearest Neighbor of I, respectively, and k k is the metric de ned in the feature space at hand, relBQS (I) = 1 e 1 dBQS (I) max dBQS (Ii)
i
1
e
where e is the Euler's number, i is the index of all images in the database and
dBQS is the distance of image I from a reference vector computed according to
the Bayes decision theory (Bayes Query Shifting, BQS) [
kN maxfkR; kN g (mR mN ) (4) where mR and mN are the mean vectors of relevant and not-relevant images respectively, is the standard deviation of the images belonging to the neighborhood of the original query and kR and kN are the number of relevant and not-relevant images, respectively.
If we are using F feature spaces, we have di erent scores rel(I) for each f feature space. Thus the following combination is performed to obtain a \single" score: rel(I) =
F X wf relf (I) where the wf is the weight associated to the f -space.
wf =
f i2R
f i2R
f i2R SVM based Relevance Feedback Support Vector Machines are used to nd a decision boundary in each feature space f 2 F . The SVM is very handy for this kind of task because, in the case of image retrieval, we deal with high dimensional feature spaces and two \classes" (i.e. relevant and not-relevant). For each feature space f , a SVM is trained using the feedback given by the user. The results of the SVMs in terms of distances from the hyperplane of separation are then combined into to a relevance score through the Mean rule as follows relSV M (I) = 1 F
F f=1 3
Image Hunter at ImageCLEF
For the participation at the ImageCLEF competition we mainly used Image
Hunter as it is. This means that as visual features we have used only those
listed in the previous section, that are partially part of those provide for the
competition [
We took part only to the task 1 (retrieval of visual concepts). In the task different visual concepts are provided and to each concept ve QBE are associated. However Image Hunter is designed to trigger the image retrieval process only with one QBE. Thus, instead of performing ve di erent runs for each concept starting with a di erent QBE and averaging the results, we slightly modi ed Image Hunter at the rst interaction step (i.e., the rst content based image retrieval before the relevance feedback steps) to take into account the ve QBE. In this case we adopted two di erent techniques: the \mean" of the QBEs and a mixed multi query approach. In the rst case, the query used to trigger Image Hunter is the mean vector of the ve QBE in each visual feature space:
Qmean = mQBE that is the case of BQS presented in Equation (4) when there are only relevant images. In the second case we performed one content based image retrieval for each QBE, and then we mixed the results by assigning to each retrieved image the minimum distance from the ve query images. After this rst automatic interaction, the tool is ready to interact with real users by using one of the relevance feedback methodologies above.
The interaction with real users was performed by 10 di erent people. The only constraint that we gave to each person was in the minimum number of interactions (i.e., 3), but letting them free to choose when stop the retrieval process. In Figure 3 some snapshots of the tool in action are reported. (6) (7) (8)
We submitted four runs to the competition combining the two methodologies for the rst step and the relevance feedback methods used: Qmean + kNN (Run11 in tables), multi query + kNN (Run12 ), Qmean + SVM (Run31 ), multi query + SVM (Run32 ). Unfortunately, Run31 was not evaluated due some duplicates in the nal le. In each run 100 retrieved documents are reported. In our case they are sorted with the relevance score obtained at the last step, these means that in some cases the rst entry is not the rst relevant image retrieved. This fact, derives from the methodologies used for relevance feedback. In particular the kNN, by means of the BQS, \moves" the query in the visual spaces, and with respect to the last BQS query the rst relevant images retrieved could be far than other images.
In Table 1 the methodologies used by the competitors in the competition are presented. Among all the competitors we were the only one that used only the visual features for all the runs, and the only one that used a real user interaction relevance feedback.
In Table 2 the precision after N docs retrieved is reported. REGIM obtained the bet results until 20 retrieved documents, after the best results are obtained by the run IBMA0 of KIDS. Instead, our results are generally good if we take into account the kNN retrieval (i.e., Run11 and Run12 ), and are mostly better than the only other method based only to visual features.
In Table 3 normalized discounted cumulative gain and mean average precision after N docs retrieved is reported. For these measures our run Run11 obtained better results than those from REGIM (that were the best in the previous table), and we can claim that it is the best second run if we look at the overall of the measures presented in this table.
Group Run ID P5 P10 P15 P20 P30 P100 KIDS IBMA0 0,8333 0,7833 0,7222 0,6896 0,6347 0,4379 KIDS OBOA0 0,8000 0,7292 0,6667 0,6354 0,6083 0,4117 KIDS IOMA0 0,7667 0,6583 0,6222 0,6104 0,5639 0,3925 KIDS OBMA0 0,6500 0,6500 0,6083 0,5771 0,5611 0,3925 REGIM run4 0,9000 0,8375 0,7917 0,7333 0,6292 0,3992 REGIM run2 0,9000 0,8417 0,7917 0,7292 0,6278 0,3975 REGIM run1 0,9000 0,8417 0,7889 0,7292 0,6278 0,3967 REGIM run5 0,9000 0,8458 0,7889 0,7292 0,6278 0,3971 REGIM run3 0,9000 0,8458 0,7889 0,7292 0,6278 0,3975 Image Hunter - Lpiras Run12 0,7917 0,7667 0,7361 0,6938 0,6083 0,3417 KIDS IOOA4 0,6750 0,6125 0,5778 0,5354 0,4486 0,3054 Image Hunter - Lpiras Run11 0,8000 0,7083 0,6222 0,5646 0,4903 0,2825 Image Hunter - Lpiras Run32 0,6583 0,5667 0,4667 0,3958 0,2972 0,1425
In our participation to the personal photo retrieval pilot task of ImageCLEF, we tested the e ciency of our previous tool Image Hunter. As our intention was to benchmark this tool on this task, we did not make any modi cation. The only modi cation made was about the use of ve QBE instead of one. The results obtained are encouraging, especially if we think that the results were obtained using only visual features. Future improvements of the tool will focus on the use of combination of the meta-data features with the visual ones, and on the improvement of our ranking system that is not actually designed for scienti c evaluation.