The personal photo retrieval task at ImageCLEF 2012 is a pilot task for testing QBE-based retrieval scenarios in the scope of personal information retrieval. This pilot task is organized as two subtasks: the visual concepts retrieval and the events retrieval. In this paper, we develop a framework of combining different visual features, EXIF data and similarity measures based on two clustering methods to retrieve the relevant images having similar visual concepts. We first analyze and select the effective visual features including color, shape, texture, and descriptor to be the basic elements of recognition. A flexible similarity measure is then given to achieve high precise image retrieval automatically. The experimental results show that the proposed framework can provide good effectiveness in distinct measures of evaluation.
The main aim of the ImageCLEF 2012 personal photo retrieval task is providing a test
bed for image retrieval based on some given query images [
The visual concepts retrieval is a great challenge to the developers. Some of the concepts are abstract like the topic “sign,” and some of them are very subjective like “art object.” Even different people would draw different opinions on the same image. The events retrieval is to find the images with the same kinds of events. Some of the target topics like “fire” and “conference” are too general to define in visual concept. Parts of the events in this subtask connect with geographical topics. Thus, most of the topics are difficult to retrieve in visual. In such a case, EXIF features may support much more information about the event concept.
In our participation to the ImageCLEF 2012 personal photo retrieval task, we developed a framework for the visual concepts retrieval and the events retrieval. First, we selected 7 visual features from the given features set for the task. Each selected feature is used to cluster all the images into groups individually. We first define the similarity degree for visual features and EXIF’s information. Then, the similarity measures for different image features are integrated to estimate the similarity scores between each image and the query image. The cluster of each feature is used to help weighting the image similarity. Finally, the framework combines and ranks the similarity degrees between an image and the different QBE images to retrieve the photos with the same concept.
The remainder of this paper is organized as follows. We describe the used features provided by the organizers in Section 2. Section 3 introduces the proposed similarity measures and retrieval methods. In Section 4, we present the experimental results of our proposed framework. Finally, we conclude the paper with a discussion and future work. 2 2.1
The original datasets in the personal photo retrieval task provided 19 extracted visual
features. After our estimating test, 7 features were selected from the 19 features. They
are AutoColorCorrelogram [
SURF feature clustering. The SURF descriptor is the feature with scale-invariant and rotation-invariant. In this paper we defined the matching pair to measure the similarity between two images. If the SURF descriptor di for the image Ii matches another descriptor dj in the image Ij and vice versa, the descriptors di and dj form a matching pair. The distance between two images Ii and Ij is defined as dist SURF ( Ii , I j )
1 N mp ( I i , I j ) , (1) where Nmp(Ii, Ij) is the number of matching pairs between the two images Ii and Ij. The larger Nmp is, more similar two images are. Based on the measure of the matching pair, we propose the clustering algorithm for SURF descriptors, shown as Table 2. Before describing the detailed algorithm, we define two cluster distances: the intracluster Dintra(Ck) and the inter-cluster Dinter(Ck, Cl).
Algorithm: SurfCluster Input: the set of images I Output: the clusters of images C C = {}; while ( min{distSURF(Ii, Ij)} < ) // is the threshold of SURF distance Case 1: Ii Ck and Ij Cl for Ck, Cl C and Ck Cl if ( Dintra(Ck ∪ Cl ) ≦1 × min{Dintra(Ck), Dintra(Cl)}) // 1 is a constant.
C = C ∪ {Ck ∪ Cl} - {Ck} - {Cl}; else
SurfCluster(Ci ∪ Cj); end if Case 2: Ii Ck and Ij Ck for Ck C if ( Dinter(Ij, Ck) ≦ 2 × Dintra(Ck) ) // 2 is a constant.
C = C ∪ {Ck ∪ {Ij}}; else
C = C ∪ {{Ii, Ij}}; end if Case 3: Ii Ck and Ij Ck for all Ck C
C = C ∪ {{Ii, Ij}}; end while
According to our observation, if the number of matching pairs is larger than four, the images look similar in visual. Hence, we define the similarity for SURF feature as 1 Dintra (Ck ) | Ck |2 distSURF ( Ii , I j ), for Ii, Ij Ck ; Dinter ( I j ,Ck )
1 | Ck |
distSURF ( Ii , I j ), for Ii Ck. SSURF ( I i , I j ) max1,
N mp ( I i , I j ) 4
.
4
(2)
(3)
(4)
Other Visual Features. For other visual features, the clustering methods consider only
the similarity between two images using the distance measures of Table 3. The
detailed algorithm is list as Table 4.
The textual features are mainly extracted from EXIFs. There are totally 63 features in
EXIFs; for example, ApertureValue, BrightnessValue, ColorSpace,
CompressedBitsPerPixel, Contrast, etc. However, only two features, the GPS and the time, were
considered and used in our methods. The values of the GPS and the time are also
clustered by the same clustering algorithm of general visual features shown in Table 4
using L1 distance measure.
The ranges of feature distances are quite different for all visual features. Before
combining all the features to measure the similarity of images, the normalization process
is necessary. We use the approximation proposed by Abramowitz & Stegun [
visual similarity between the images Ii and Ij, SV(Ii, Ij), is defined as
SV (I i , I j ) S SURF (I i , I j ) wk S fk (I i , I j ) , k where Sfk(Ii, Ij) means the similarity between the images Ii and Ij of the k-th feature, wk is the weight of the k-th feature. Two weighting methods, the cluster weighting and the non-cluster weighting, are proposed as follows: ●
Cluster Weighting. We use the clustering results of Section 2 to automatically weight the features. If a query image belongs to a cluster for a specific visual feature, the average similarity between the query image and each image in the cluster is computed as the weight of the specific visual feature. (5) (6) ● Non-Cluster Weighting. In this method, the weights wk are set to 1, except for the weights of AutoColorCorrelogram, Color Structure, and SURF features double other visual features.
proposed for the geographical distance: ● Boolean measure. The Boolean measure of the GPS feature is defined as 1 if GPS (Ii ) and GPS (I j ) are in the same cluster, SG(B) (Ii , I j )
0 otherwise; where GPS(Ii) and GPS(Ij) denote the values of the GPS feature in EXIF for Ii, Ij. ● Similarity measure. The continuous similarity measure on geographical distance is defined as
SG(S) (Ii , I j ) 1 1 e dist(GPS(Ii ),GPS(I j ))radius , where and radius are smoothing parameters; dist(GPS(Ii), GPS(Ij)) means the real geographical distance on earth between the two positions GPS(Ii), GPS(Ij). (7) (8) (9) (10)
proposed for time duration: ● Boolean measure. The Boolean measure of the time feature is defined as 1 if T (Ii ) and T (I j ) are in the same cluster, ST (B) (Ii , I j )
0 otherwise; where T(Ii) and T(Ij) denote the time feature in EXIF of Ii and Ij. ● Similarity Measure. The continuous similarity measure on time is defined as
ST (S) (I i , I j ) 1 dist(T (I t ),T (I j )). where dist(T(Ii), T(Ij)) denote the real time difference in second between two timestamp T(Ii) and T(Ij). 3.3
Finally, we define the similarity between two images Ii and Ij by integrate the features SV(Ii, Ij), SG()(Ii, Ij), and ST()(Ii, Ij) into a linear combination. The image similarity Sim(Ii, Ij) is defined as
Sim(Ii , I j ) wV SV (Ii , I j ) wG SG() (Ii , I j ) wT ST () (Ii , I j ) . (11)
Given a set of query images Qj, 1 j m, the similarity of each query image Qj and the image Ii in the image set is measured by Sim(Ii, Qj). The maximum similarity Sim(Ii, Qj) is the similarity degree of the image Ii for the visual concept via the m query images Qj. It can be formally defined as max{Sim( Ii ,Q j )} . 1 jm (12) 4 4.1
The system is implemented on a Microsoft Windows XP SP 3, 2.33 GHz PC with
3.00GB RAM. The developed software and related systems are written in Java
language, so the system is cross-platform. The methods in five runs used different image
features, which are shown in Table 5. The notations in the table are: V stands for the
visual features; G denotes the GPS feature; T is the time feature. While the parameter
C, N means the cluster weighting and the non-clustering weighting, respectively.
Finally, the parameter B represents the Boolean measures and S is the similarity
measures.
In this subtask, 24 visual concept queries were given to be evaluated from the totally
32 concepts. The retrieval results for the visual concepts are evaluated by three
different measures: precision, NDCG (normalize discount cumulative gain) [
As Table 6 shows, the Run 5 using all of the image features is the best one for all measures. The second place is the Run 2 which uses the time feature only. The Run 3 with the visual features and the GPS feature is the third place. The Run 1 and the Run 4 are worse than the above three runs.
The results show that the visual features are not useful for most of the visual concepts in the task. The reason is that most of the concept topics are semantically related to each other. There is not much common characteristic in visual features among QBE images. While combining the visual features with the EXIF features, the performance Run 1
Run 2 Features Weights
V 0.6750 0.6125 0.5778 0.5354 0.4486 0.3054 0.5701 0.5062 0.4798 0.4545 0.4016 0.3303 0.0632 0.0930
T wT 1 increases obviously. The GPS feature can help us to find the images photographed in the neighboring positions easily. The geographic-related topics like “Asian temple & palace” and “temple (ancient)” have good results. However, some topics are not expected to be good, like “animals” and “submarine scene,” which returned high precision. The main reason is that a photographer generally tries to take pictures with the similar topics at the same place. Although the GPS feature is precise for geographicrelated topics, the missing values on the GPS feature will degrade the precision greatly. Some non-geographical topics have obviously bad results in the runs of using the GPS features, like “clouds.” The time feature is also an important factor for searching personal photos. Since the images photographed in short time are usually very similar or dependent in visual concept. As the above discussion, the Run 5 getting the best results shows that our image similarity measure method can combine the different image features effectively. In the subtask, totally 15 different events queries are given to find the pictures with the same event. Each query contains three QBE images. The evaluations are done by precision, NDCG, and MAP as the subtask 1. The experimental results are shown in Table 7.
As Table 7 shows, the best results are the Run 2 and Run 5. The Run 1 using the visual features is still the worst as the subtask 1. The Run 3 using the visual and the GPS features is a little better than the Run 4 taking the visual and the time features.
Owing to the event queries usually describe the images with the properties of happening in specific time duration or location area, the time and the GPS features are relatively important here. For example, the topics “Australia,” “Bali,” and “Egypt” are related in geographical; the topics of activities like “conference,” “party,” and “rock concert” are temporal-related. Hence, the provided EXIFs of the images are very useful in this subtask of events retrieval. The Run 5 combining all features is not expected to be the best as the subtask 1. The reason might be that the event queries are not so related with the visual concept, but highly dependent on time and location. However, the proposed similarity measure method did not degrade the precision much.
Run 1
V wV 1 In this paper we proposed a framework and similarity measure methods to combine different image features for retrieving images from a set of conceptual photos. The proposed method can handle the visual concepts retrieval subtask in part. However, the time and position information are more important than other visual features in the event retrieval subtask. Although the proposed method could adjust the weights to fit the requirements, it has still a lot of problems to be solved. The proposed framework retrieved the relevant images weighted by manual in most of the cases. As we know, the feature selection is important in retrieval individual concept. For example, the experimental results show that the GPS and the time features are very useful for retrieval in this dataset. However, it may be not so effective in other dataset. The problem of selecting and weighting the features automatically is a challenge in the task.
This pilot task is its first year announced at ImageCLEF. The dataset seems too small for evaluating modern applications. Further, the concept queries often contain some irrelevant images in visual. The procedure of determining concepts and their relevant images may need to be fixed for providing as a benchmark.