This paper considers the strategies of query expansion, relevance feedback and result fusion to increase both relevance and diversity in photo retrieval. In the text-based retrieval only experiments, the run with query expansion has better MAP and P20 than that without query expansion, and only has 0.85% decrease in CR20. Although relevance feedback run increases both MAP and P20, its CR20 decreases 10.18% compared with non-feedback run. It shows that relevance feedback brings in relevant but similar images, thus diversity may be decreased. The run with both query expansion and relevance feedback is the best in the four text-based runs. In the content-based retrieval only experiments, the run without feedback outperforms the run with feedback. The latter has 10.84%, 9.13%, and 20.46% performance decrease in MAP, P20, and CR20. In the fusion experiment, integrating text-based and content-based retrieval not only reports more relevant images, but also more diverse ones.
In the photo retrieval task of ImageCLEF 2008, the focus is shifted from cross language image retrieval to promote diversity. Besides relevance, retrieving diverse items representing different subtopics is also concerned. How to balance the relevance and diversity is challenging. This paper studies the strategies of query expansion and relevance feedback in text-based and content-based retrieval, and shows how to merge the results of text and image queries to increase both relevance and diversity.
This paper is organized as follows. Sections 2, 3 and 4 present text-based retrieval, content-based retrieval and combination of both, respectively. Section 5 shows the runs submmited for formal evaluation in the photo retrieval task, and discsses the effects of different retrieval and fusion strategies.
In text-based retrieval, we consider the strategies of query expansion and relevance feedback. Assume the text corpus T is composed of n terms, t1, t2, …, tn, and a query Q contains m query terms, q1, q2, …, qm. We expand Q in the following way.
(1) For each corpus term ti and query term qj, compute P(qj | ti)=P(ti,qj)/ P(ti). (2) For each corpus term ti, compute OverlapNum(ti,Q) defined below.
OverlapNum(ti,Q) = cardinality{q | q∈Q, P(q | ti)>0}
m (3) For all ti∈T, if ∑ P( q j | ti ) × OverlapNum( ti ,Q ) > thd, then ti will be added j=1 to new query Q’. In the experiments, thd is set to 1. In other words, the original query terms which also appear in the corpus will be added into Q’.
We adopt Lemur as our text IR system. The weighting function is BM25 with parameters (K1=1.2, B=0.75, K3=7). For relevance feedback, we select the top-10 terms of the highest BM25 scores from the top-5 retrieved documents, and add them to the query. The expanded terms have 1/2 weight of the original query terms.
For each image gi, we extract two kinds of features: SizeFeature(gi) and ColorFeature(gi). These two functions are defined below.
(1) SizeFeature(gi) = 0, if height(gi)>width(gi)
= 1, if height(gi)<=width(gi) (2) ColorFeature(gi): divide gi into 32×32 blocks, and extract their RGB values. The similarity of two images, gi and gj, is computed as follows.
(1) Compute the color similarity of gi and gj based on their color features.
ColorSimilar(gi, gj) = number of blocks in gi and gj, whose R, G and B value differences are not larger than 10. (2) Compute the size similarity of gi and gj based on their size features.
If SizeFeature(gi) and SizeFeature(gj) is the same, then SizeSimilar(gi, gj)=1.5. Otherwise, SizeSimilar(gi, gj)=1.0. (3) The similarity of gi and gj is in terms of SizeSimilar and ColorSimilar:
Similar(gi, gj)= SizeSimilar(gi, gj) × ColorSimilar(gi, gj)
In image retrieval, we compute the similarities of the query images and all the
images in the data set and select the most similar image for media mapping
We submit 7 runs shown below for the formal evaluation. (1) NTU-EN-EN-AUTO-NOFB-TXT
This run is baseline. We employ Lemur for text-based retrieval without query expansion and relevance feedback. (2) NTU-EN-EN-AUTO-FB-TXT
This run employs Lemur for text-based retrieval with relevance feedback. (3) NTU-EN-EN-AUTO-QE-NOFB-TXT
This run employs Lemur for text-based retrieval with query expansion. (4) NTU-EN-EN-AUTO-QE-FB-TXT
This run employs Lemur for text-based retrieval with query expansion and relevance feedback. (5) NTU-IMG-EN-AUTO-NOFB-TXTIMG.
This run employs content-based retrieval first, then adopts media mapping to transform the image query to text query, and employs Lemur for text-based retrieval without relevance feedback. (6) NTU-IMG-EN-AUTO-FB-TXTIMG
This run is similar to NTU-IMG-EN-AUTO-NOFB-TXTIMG except that relevance feedback is done. (7) NTU-EN-EN-AUTO-QE-FB-TXTIMG
This run merges the results of NTU-IMG-EN-AUTO-FB-TXTIMG and NTU-EN-EN-AUTO-QE-FB-TXT.
The evaluation of the formal runs is based on mean average precision (MAP), precision at 20 (P20) and instance recall at rank 20 (CR20), which calculates the percentage of different clusters represented in the top 20. Table 1 lists the experimental results of employing text query only. The run with query expansion has better MAP and P20 than that without query expansion, and only has 0.85% decrease in CR20. Although relevance feedback increases both MAP and P20 in EN-EN-AUTO-FB-TXT run, its CR20 decreases 10.18% compared with EN-EN-AUTO-NOFB-TXT. It shows that relevance feedback brings in relevant but similar images, thus diversity may be decreased. The run with both query expansion and relevance feedback is better than the other three runs. Compared with baseline, it has 33.79%, 44.44%, and 0.27% increase in MAP, P20 and CR20, respectively.
Table 2 lists the experimental results of employing sample images. In the experiments, 3 example images are considered. The run without feedback outperforms the run with feedback. The latter has 10.84%, 9.13%, and 20.46% performance decrease in MAP, P20, and CR20. The possible reason of the drop in precision is the top-5 retrieved images for feedback may be very specific. That may introduce noises. Consider topic 43, sunset over water, as an example. The correct image should contain both sunset and water. The query without feedback is “Sunset at the sea the dark outlines of a mountain in the foreground the sun is rising over the sea behind it a light orange sky in the background peru”. In the top-5 retrieved images, only one contains both scenes, but all of them contain sunset scene. There are 34 relevant images in the result list before feedback, and only 25 relevant images after feedback. The MAP decreases from 0.1776 to 0.0535 after feedback. CR20 decreases more than MAP and P20. It shows pure relevance feedback is harmful to diversity.
Table 3 compares the performance of employing text query only, image query only, and both. The fusion run, which achieves MAP 0.2809, P20 0.3769 and CR20 0.2763, is the best of our 7 submitted runs in the formal evaluation. It shows that integrating text-based and content-based retrieval not only reports more relevant images, but also more diverse ones.
This paper considers query expansion, relevance feedback and result fusion to deal with relevance and diversity in image retrieval. Query expansion is useful to increase the precsion in text-based retrieval, but has a little negative effect on the diversity. Relevance feedback is harmful to diversity when this strategy is used independently or in single type of queries. Text-based and content-based retrievals have their own special capability, so that both relevance and diversity are improved.