In this paper, we present our solutions for the WikipediaMM task at ImageCLEF 2008. The aim of this task is to investigate effective retrieval approaches in the context of a large-scale and heterogeneous collection of Wikipedia images that are searched by textual queries (and/or sample images and/or concepts) describing a user's information need. We first experimented with a text-based image retrieval approach with query extension, where the expansion terms are automatically selected from a knowledge base that is (semi-)automatically constructed from Wikipedia. We show how this open, constantly evolving encyclopedia can yield inexpensive knowledge structures that are specifically tailored to effectively enhance the semantics of queries. Encouragingly, the experimental results rank in the first place among all submitted runs. The second approach we experimented with is content-based image retrieval (CBIR), in which we first train 1-vs-all classifiers for all query concepts by using the training images obtained by Yahoo! search, and then treat the retrieval task as visual concept detection in the given Wikipedia image set. By comparison, this approach performs better than other submitted CBIR runs. Finally, we experimented with a cross-media image retrieval approach by combining and re-ranking text-based and content-based retrieval results. Despite the final experimental results were not formally submitted before the deadline, this approach performs remarkably better than the text-based retrieval or CBIR approaches.
ImageCLEF 2008 contains five different tasks (i.e., photo retrieval, medical retrieval, visual concept detection, medical annotation, and WikipediaMM). In this paper we present our efforts in the WikipediaMM task of ImageCLEF 2008. This is our first year at ImageCLEF and the WikipediaMM task we participated in is also offered for the first time. We participated in all steps of the task, including topic creation, retrieval experiments, and relevance assessment.
The aim of WikipediaMM 2008 task is to investigate effective retrieval approaches in the context of a large-scale and heterogeneous collection of Wikipedia images that are searched by textual queries (and/or sample images and/or concepts) describing a user’s information need. Towards this end, WikipediaMM 2008 task needs to deal with searching 75 topics from approximately 150,000 images in the Wikipedia collection. Roughly speaking, there are three challenges that the task participants must deal with: y Scalability. A good retrieval approach should perform well on such a large image set. y Robustness. A natural, and also maybe the most effective solution for this task is text-based image retrieval approach since each image in this dataset is associated with user-generated alphanumeric, unstructured metadata (e.g., a brief caption or description of the image). However, text-based retrieval methods should be robust to noisy textual description that many images may be annotated with. y Multi-modal Fusion. Considering the fact that some images in this dataset have few or even no descriptive texts, it may be more preferable to combine text-based and content-based image retrieval (CBIR) for better performance. However, a pure combination of traditional text-based and content-based approaches is not adequate for dealing with the problem of Wikipedia image retrieval [1]. Therefore, how to effectively exploit the correlation between different modalities of retrieval clues remains challenging.
In this participation, we carry out three kinds of experiments. Our key idea is to experiment with different query extension techniques to help the retrieval system get close to users’ real intent. In general, the oft-used approach of query extension is to add terms to queries or modify preliminary queries. In our experiments, query extension techniques are used in different situations. In the case of text-based image retrieval, the expansion terms are automatically selected from a knowledge base that is (semi-)automatically constructed from Wikipedia. This open, constantly evolving encyclopedia can yield inexpensive knowledge structures that are specifically tailored to effectively enhance the semantics of queries. Encouragingly, the experimental results rank in the first place among all submitted runs. In the case of CBIR, 1-vs-all classifiers are trained for all query concepts by using the training images obtained by Yahoo! search. Then the retrieval task can be implemented as visual concept detection in the given Wikipedia image set. Clearly, the training images obtained by Yahoo! search are used to enhance the image retrieval task. In the last case, we experimented with a cross-media image retrieval approach by combining and re-ranking text-based and content-based retrieval results. Here the CBIR results are used to modify preliminary text-based retrieval results. Despite the final experimental results were not formally submitted before the deadline, this approach performs remarkably better than the single text-based retrieval or CBIR approaches.
The rest of this paper is organized as follows. First of all, we present the architecture of our system for the WikipediaMM task in section 2. Then three different approaches are described respectively in the following three sections. The experimental results of our approaches are presented in section 6. Finally we draw a conclusion and propose our future work.
In our system, we implemented a retrieve engine as a test environment for all methods. Fig.1 shows the
architecture of our system, illustrating how different components works together to generate retrieval results. The
components include:
returns relevant images.
(
A natural solution for WikipediaMM 2008 task is text-based image retrieval method. To help the retrieval system get close to users’ real intent, query extension techniques are often used by adding terms to queries or modifying preliminary queries. In this participation, we focus on how to automatically extract the expansion terms from a knowledge base that is (semi-)automatically constructed from Wikipedia. Organized with concept identified by URL (one user defined concept refers a single page) and links between concepts and external nodes, Wikipedia is not only a Web collection but also an online knowledge center which assembles all users’ intelligences. Thereby, it is naturally attractive and promising that this open, and constantly evolving encyclopedia can yield inexpensive knowledge structures that are specifically tailored to effectively enhance the semantics of queries. Recently, “Wikipedia mining” has been addressed as a new research area. WikiRelate [2] used links-based path length for computing relatedness for given concepts; Nakayama et al. [3] proposed a link mining method called PFIBF (Path Frequency – Inversed Backward link Frequency) as a base for Web thesaurus construction. However, none of work is made on using Wikipedia as the knowledge base in information retrieval. Fig. 2 shows the system framework and data flow of our text-based retrieval approach. As mentioned above, we first construct a knowledge base from Wikipedia pages for query extension. Specifically, each non-administrative Wikipedia page is used as a term/concept describing individuals (e.g., Jingtao Hu), concepts (e.g., Emissions trading), locations (e.g., Big Ben), events (e.g., collapse of the World trade Center), and categories (e.g., microbiology). For a given term, the related terms can be easily extracted from the corresponding Wikipedia page. Thus given a textual query (and/or concept), the query constructer searches the knowledge base with the given query term and then extends the query with extracted terms. Finally, the extended query is given to the retrieval engine to generate the final search results.
(a) (b)
Fig.2. Text-based retrieval with query extension. (a) System framework, and (b) Data flow.
In our system, the construction of the knowledge base from Wikipedia includes the following steps:
(
In our system, we employ an unsupervised keyphrase extraction algorithm presented in our previous work [5].
This algorithm uses several set-independent feature weights, treating text in a page as a semantic network.
Several structure variables of Small-World Network (SWN) are used to select key nodes from the network as
keyphrases K = {(tk , P(tk ))} , each with a probability score P(tk ) indicating the importance of the extracted
keyphrase tk .
(
ibf = log(bf (tN) + β ) , (
Therefore, the final weight of a keyphrase can be computed as:
wtk ∈K = P(tk ) ⋅ ibf (tk )
(
Fig. 3 Examples for topic “daily show” and topic “Big Ben” in the knowledge base For each topic, we can extract a knowledge tree which consists of the given topic, and keyphrases extracted from Wikipedia pages. Moreover, this knowledge tree can be pre-constructed or online constructed. By treating each query as a topic, the text-based retrieval system finds the topic in the knowledge base, and uses the weighted keyphrases as query extension to enhance the system performance. It should be noted that if no relevant keyphrases can be found in the base with respect to the query, it is very easy to perform the above steps online.
For the retrieve engine, we use the TF-IDF paradigm which is widely used in text mining and information retrieval. It is defined as follows:
In many real-world image datasets such as Wikipedia image collection, there are some images have few or even no descriptive texts. To address this problem, content-based image retrieval (CBIR) is used in our system. Given the pre-defined query concepts, 1-vs-all classifiers are trained for all these concepts by using the training images obtained by Yahoo! search. Then the retrieval task can be treated as visual concept detection in the given Wikipedia image set. Fig is a summary of our approach. Clearly, the training images obtained by Yahoo! search are used for enhancing the image retrieval task.
“skyscraper” SVM
SVM training / testing
Search engine
Top k images
topic representation
In our implementation, the training process includes the following three steps:
(
For better retrieval performance, we study cross-media image retrieval by combining both text-based retrieval and CBIR methodologies. In our system, we use the re-ranking scheme to combine the retrieval results of the two engines. Given a query topic, either text-based retrieval or CBIR engine can return a set of result images. By combining the two result sets, the returned images are re-ranked with weighting factors
WeightedScore(q, d ) = w1 * Score(qtext , dtext ) + w2 * Score(qvisual , dvisual ) . (
G = {(di1, d 2j ) : di1 = d 2j , i < M 1, j < M 2} , where (di1, d 2j ) stands for an image both returned by the two engines. The numbers of overlap images in Top N ranked images are computed as:
H = #{di1 : di1 ∈ G, i < N}, H2 = #{d 2j : d 2j ∈ G, j < N}
1 and the weight of each engine is calculated as: wl = σ / 2 + Hl / N , σ + ∑ Hl / N
l where l is the engine identifier and σ is an adjusting parameter.
In this section, we describe our experiments for WikipediaMM task at the ImageCLEF 2008. Note that some of experimental results reported in this section were not formally submitted before the deadline.
(
In the experiments, we found that with a high threshold, the number of similar article titles is limited and there are few terms to be added to the query. From the experimental results shown in Table 2, we can see that using similar article titles in Wikipedia with respect to the queries directly does not help improving the performance of text-based retrieval system.
Link structure mining of Wikipedia can be used in Web thesaurus construction or query expansion. However, due to the complexity of Wikipedia hyperlink network, there were too many noises in these links, including Wikipedia system-generated links, and links that are rarely referenced. Therefore, we obtained even worse results than text-based retrieval without query extension.
For query expansion with the automatically constructed knowledge base, we found that the performance of the text-based retrieval system using keyphrases with just high probabilities is worse than text-based retrieval without any query extension. This means that keyphrases with general meanings have negative effects on retrieval results.
In contrast, by using the Keyword-IBF method, general keyphrases referenced by many articles are granted a lower weight in the extended query. As a consequence, this query extension method tends to choose terms that are more specific. The average improvement of all queries in this case is around 0.5% over text-based retrieval methods without query extension.
Text-based image retrieval approach with query extension can perform better than traditional text retrieval Fig. 7. Experimental Results of Precision at top N results of text-based retrieval with query extension from the semi-automatically constructed Wikipedia knowledge base.
By reasonably manual confirmation of the knowledge base, we obtained an obvious improvement of the retrieval results. From Table 3 and Fig. 7, we can see that this approach performs much better than all other models. This shows that query extension by using a knowledge construction procedure with a good keyphrase extraction algorithm and reasonably manual confirmation can remarkably improve the retrieval performance.
For content-based image retrieval, the experimental results show that our BOW methods perform better than other CBIR systems. Compared with the text-based systems, our CBIR obtained a comparable MAP (0.1912 of CBIR vs. 0.21003 of text-based retrieval), and higher precisions in the top-ranked images (P@5=0.5333 and P@10= 0.448 of CBIR vs. P@5=0.405333 and P@10=0.346667 of text-based retrieval). Although visual content ambiguity hampers the overall performance (MAP) by returning images with similar low-level features, the experimental results show that learning visual models from Web images (e.g., from Yahoo! search) do help to rank the content-relevant images in higher places.
Table 4: The testing results of CBIR.
Run ID QE Modality MAP P@5 P@10 R-Prec CBIR run1 with IMG 0.1912 0.5333 0.442667 0.292887 CBIR run2 with IMG 0.1928 0.5307 0.4507 0.2295 It also should be noted that, our CBIR approach performs best in all submitted CBIR runs in WikipediaMM 2008 task.
In the last set of experiments, text-based and content-based image retrieval approaches are combined so as to
achieve better performance. In the experiments, we set M2 smaller than M1. This means that only the top-ranked
images returned by the CBIR approach are included in the re-ranking phase since the lower-ranked images may
have much higher probabilities to be noises in the CBIR system. Table 5 shows the testing results on the given
collection, where ReRank-Text-Visual-N denotes the combination of CBIR and text-based retrieval without query
extension, and ReRank-Semi-Visual-N denotes the combination of CBIR and text-based retrieval with query
extension from the semi-automatically constructed knowledge base, and N denotes the corresponding parameter
in formula (
Run ID
ReRank-Text-Visual-10
ReRank-Text-Visual-20
ReRank-Text-Visual-40
ReRank-Text-Visual-60
ReRank-Text-Visual-80
ReRank-Text-Visual-100
ReRank-Semi-Visual-10
ReRank-Semi-Visual-20
ReRank-Semi-Visual-40
ReRank-Semi-Visual-60
ReRank-Semi-Visual-80
ReRank-Semi-Visual-100
(a) (b)
Fig. 8 Performance of cross-media re-ranking: (a) Precision at top N results of cross-media re-ranking. (b) MAP
of re-ranking with different values of N in formula (
We used 6 values of N (N=10, 20, 40, 60, 80, 100) in formula (
As shown in Table 5 and fig. 8, the re-ranking approach performs remarkably better than the single text-based or content-based retrieval approaches. For the combination of CBIR and text-based retrieval without query extension, the average improvement of all the queries in ReRank-Text-Visual-10 is around 5.34% over the single text-based retrieval approach (25.6498% of MAP, as shown in Table 1). While for the combination of CBIR and text-based retrieval with query extension from the semi-automatically constructed knowledge base, the average improvement for all the queries in ReRank-Semi-Visual-10 is around 1.403% over the single text-based retrieval approach (34.44% of MAP, as shown in Table 3).
Fig. 9 shows two examples of the re-ranking results. For query “golden gate bridge”, text-based retrieval returns more relevant images than visual-based information retrieval. By re-ranking the two results, we get a better performance and a higher precision of top ranked images than the previous two methods. For query “Singapore”, visual-based retrieval does not perform well, however, combining CBIR result with text-based retrieval can still improve the system performance.
We also observed that the re-ranked results have higher precision of top-ranked images than results returned by text-based retrieval or CBIR. There are some possible reasons. Text-based retrieval can return more relevant images by searching keywords with image descriptions, while CBIR can obtain high precision of top-ranked images but too many noises in lower-ranked images. Thus combining CBIR with text-based retrieval can help increase the precision of top-ranked images.
In conclusion, the cross-media re-ranking approach performs remarkably well. This indicates that cross-media fusion is definitely a promising direction to investigate effective retrieval approaches in the context of a large-scale and heterogeneous collection of images.
Fig. 9 Search results for (a) topic “Golden Gate Bridge” and (b) topic “Singapore” using text-only (left), visual-only (middle) and their combination (right).
In this paper, we reported our solutions for WikipediaMM task at ImageCLEF 2008. We experimented with text-based, CBIR and cross-media image retrieval approaches with query extension. Encouragingly, the experimental results of our text-based approach rank in the first place among all submitted runs. Moreover, our CBIR approach also performs better than other submitted CBIR runs. Despite the final experimental results were not formally submitted before the deadline, our cross-media approach performs remarkably better than the single text-based or content-based retrieval approaches.
However, our experiments are just a first attempt towards effective cross-media retrieval in the context of a large-scale and heterogeneous collection of images. The query extension techniques can still be improved. Specifically, our knowledge base construction procedure strongly depends on the keyphrase extraction algorithm. Thus how to more effectively extract concepts and their relationships from Wikipedia is still a challenging future work.
This work is supported by grants from Chinese NSF under contract No. 60605020, National Hi-Tech R&D Program (863) of China under contract No. 2006AA01Z320 and 2006AA010105.