In the paper, we introduce our systems and methods to promote the diversity of the ad-hoc photog retrieval in ImageCLEF 2009. The image database in this year is quite different from the previous years, not only increasing the corpus size from 20,000 images to half millionm but also changing the domain from the travel to news. Most of queries are related to person names and the text information in image documents is rich. Thus we put a lot of effort on text while using the visual information to elimiate or down-rank similar images in order to make top images visually dissimilar. To reduce the ambiguarity of query and infer the implicit dimension of diversity space, name entity extraction is applied on the top documents in order to extract informative phrase patterns to faciliate query expansion. Name entity based query expansion makes our system placed in the top performance.
1. Introduction In the past years, the task of the photographic image retrieval works on the images in the travel domain and the database only has 20,000 images. The image document includes two modalities i.e. visual image and text description. Thus the information needs of the user query are formulated as keywords or image exemplars. In this year, the domain of the benchmark image database is changed to the news. The image content (visual and text description) in the news domain is quite different from the previous. So do the information needs, i.e. query types. In news domain, most of the queries and text descriptions of images are related with the personal names and events. Here are two query examples, one having the description about the meaning of diversity in the field of <clusterTitle> besides the key information need in the field of <title> while another only has key information frequently having one keyword.
Example 1 <top> <num> 1 </num> <title> leterme </title> <clusterTitle> yves leterme </clusterTitle> <clusterDesc> Relevant images contain photographs of Yves Leterme. Images of Leterme with other people are relevant if Leterme is shown in the foreground. Images of Leterme in the background are irrelevant. </clusterDesc> <image> belga28/05980958.jpg </image> <clusterTitle> leterme albert </clusterTitle> <clusterDesc> Relevant images contain photographs of Yves Leterme and King Albert II. Images with only one of them are considered to be irrelevant. </clusterDesc> <image> belga27/05960161.jpg </image> <clusterTitle> leterme -albert </clusterTitle> <clusterDesc> Images which contain photographs of Leterme which are not part of the above categories are relevant to this cluster. </clusterDesc> <image> belga32/06229323.jpg </image> </top> <title> obama </title> <image> belga30/06098170.jpg </image> <image> belga28/06019914.jpg </image> <image> belga30/06107499.jpg </image> </top>
descriptions attached to the image. Thus we can improve the diversity performance by grouping the visually-similar
images and only selecting one or a few representative images in a group to represent the whole cluster. The visual
similarity can be measured by matching one against another to see whether they are significantly different [
ranking system with the visual-based system. Thus our system architecture in this year also uses the two modalities.
Simple and computation efficient visual features such as global colour moments and histogram, which work well in the
previous data, do not operate well in this new dataset, because the content of images in this year are objects such as
person. It requires us to extract other effective visual features, e.g. histogram of orientation which achieves higher
performance in person detection [
As introduced in the above, the text-based ranking system and the content-based image ranking system are individually
built based on the text descriptions and visual descriptors respectively. To down-rank the visually similar images, we
also apply the visual descriptors to re-rank the output from the text-based system. Before we discuss the details of
submitted runs, we first discuss the index structure for large-scale retrieval system. This year the database has about half
million image documents while it only has ~20,000 images in the previous corpus. It is crucial to build an index
structure for efficiently and effectively processing the query rather than linearly scanning the database. For the
contentbased image retrieval system, the visual content of each image is characterized by a high dimensional feature. Although
there are a lot of methods to index the documents, we apply the multi-probe locality sensitive hashing (MPLSH) 1[
Run 1: LRI2R_TCT_TXT It is the basic run submitted in order to validate the efficiency of name entity extraction for the query expansion. In the run, only the text modality is used. The keywords of the query are the words occurred in the fields of title and cluster title without using the cluster description. Each field represents a different information need of the user. For example, the title is a general information need while the cluster title gives a detailed information need. Thus, we formulate multiple sub-queries using these filed and then linearly combine multiple rank lists to reach a final rank list for the query. For example, for the first query discussed in Section 1, we have 4 sub-queries and thus the final rank is from fusing four rank lists. Without any prior knowledge of which field should be weighed more, the equal weight is used. Run 2: LRI2R_TI_TXT The run is to evaluate query expansion using name entity extraction. The name entity extraction operates on the text descriptions corresponding to the top-N (In the experiment, N=20) image documents in the Run 1. The name entity extraction tags the words in the text description using four categories: person, organization, location and others. We will only consider the first two categories. Then we search the phrase patterns occurred closely with the query keywords occurred in the title and cluster title used in Run 1, in a window size of 10 words. We assume that each phrase pattern
may represent one dimension in the diversity space and we treat each phrase pattern as an individual sub-query similar to Run 1. Thus, we have mined multiple cluster titles automatically for each query. Then the final ranking is gotten using the same operation in Run 1.
Example 1 (query 1) leterme Yves leterme Belga Photo Mark Renders leterme Elio Di Example 2 (query 26) obama Barack obama Senate obama Michelle Run 3: LRI2R_FUSE_TCTI_TXT
Run 4: LRI2R_DIVERSITY_TCTI_TXTIMG
feature we used is derived from the SIFT descriptor using the bag-of-visterm. In our implementation, two visual
dictionaries with the size of 512 and 256 are individually generated using k-means on 20,000 randomly selected images
from the database. We use the heuristic way similar to [
3. Do a. Choose next document in initial rank and calculate its visual similarity with the documents in diversity stack b. If maximal similarity is lower than a threshold, then the document is novel, push it into diversity stack.
c. Remove the document from the initial rank
see the best run is the Run 2, which uses name entity extraction for query expansion. Comparing with its baseline, i.e.
entity extraction can mine the useful pattern for disambiguate the query and to discover the other dimensions of diversity space which are not in the query. The discovered diversity dimensions play an even important role for the queries in part 2, where the query only has a few words, most only having one word and without any diversity indicator, i.e. cluster title. This observation can be enhanced when analysing the performance on the queries in part 2 in Table 3.
(CR@10, P@10, F-measure) is even bigger. The F-measure in Run 2 is 0.7528 compared with 0.6556 in Run 1.
better than the Run 2. It indicates that the manually selected cluster titles are superior to our automatically selected. This is reasonable because the automatically found phrase patterns may have a few noises. But the performance gap in this case only has ~3% in F-measure, not such bigger as in Table 3, where the F-measure difference is ~9%. The analysis strongly demonstrates name entity extraction is a very useful technology for diversity search, especially when the query is ambiguous and diversity dimensions are not indicated.
is a combination of Run 1 and Run 2. From Table 1, we find that the Run 3 is the worst of all 5 runs, although individually the Run 1 and Run 2 are in the second and first rank position. The Run 5 is a combination of Run 3 and a
significant as our past years’ systems [
0.6710 0.848 0.429 0.6570 0.79 0.5033 0.6239 0.794 0.4219 0.6005 0.804 0.4779 0.5965 0.802 0.4778 0.7492 0.7174 0.6987 0.6875 0.6842 0.7329 0.828 0.4904 0.6519 0.868 0.4324 0.6055 0.852 0.5129 0.6082 0.832 0.4445 0.5975 0.848 0.5127 0.7760 0.7446 0.7079 0.7027 0.7010 0.6901 0.818 0.426 0.6395 0.756 0.399 0.5955 0.756 0.443 0.5955 0.756 0.443 0.5811 0.752 0.516 0.7528 0.6929 0.6662 0.6662 0.6556
In the paper we describe the details of our submitted runs and analyze their diversity performance. Our analysis indicates two useful technology for promoting diversity search: 1) name entity extraction based query expansion and 2) visual based re-ranking. Name entity extraction can mine the phrase patterns to characterize the diversity space of information need of the user while the visual based re-ranking can further update ranking to make the top document visually dissimilar. Our successfull systems do not exploit the query image exemplars. The system of combining textbased retrieval and content-based image retrieval does not give an obvious gain. In future, we will dig into the two successful technologies to further improve the diversity performance.