In this paper we discuss the CELI's first year of activity at CLEF in the context of Cross Language Image Retrieval Track (ImageCLEF). The proposed system is an upgrade of CELI's cross language delegated search system (www.elois.biz). The system is meant to perform CLIR on the web by using Google and Yahoo indexes. Therefore the goal is to provide reasonable translations of queries with no direct access to the corpus. This, in turn, means absence of tuning procedure for the system and impossibility to impose restrictions in terms of domain, style, etc. Our approach is based on bilingual dictionaries and the main research effort was devoted to filter out the noise introduced by translation ambiguities. We experimented a disambiguation strategy based on Latent Semantic Analysis which allow us to compute the degree of semantic coherence of possible translation candidates. We also tested some query expansion methods in order to “balance” the fact that the system does not perform any kind of visual retrieval. Being the system basically the same, we are in a good position to compare results in the Image track with the one in the Ad-hoc track. Surprisingly it emerges that, contrary to what happened in the Ad-hoc exercise, expansion strategies improve results in image retrieval tasks.
CELI (www.celi.it) is an Italian company active in the Natural Language Processing and Document Retrieval field. Over the years CELI developed several cross language information retrieval systems, both in the context of European projects and commercial applications. Recently an internal project started, with the goal of providing cross language access to the indexes of Google and Yahoo (cf. http://www.elois.biz). It is in the context of such a project that the participation to CLEF was decided. The goal was mainly to compute figures of performance of several different strategies of query disambiguation and query expansion as well as obtain a comparison with the best systems in the cross language information retrieval arena.
We were particularly interested to Image track for the following reasons:
• There is a renewed commercial interest for image search, and cross language image search is
one of the few domains where appreciation of the results can take place irrespective of the
query language.
• By working just on textual metadata the system is easily extensible to different kinds of
digital objects, such as music and images.
• Thanks to a collaboration with the University of Bolzano, CELI is active in the domain
of digital libraries, with a particular regard to the migration from standard OPAC search
to natural language access to catalogues [
The system which has been used is identical to the one described in [
These have been computed by using Latent Semantic Analysis. • Besides query translation, the system is able to perform query expansion on the target language. Three strategies are tested, namely: – The use of a generic semantic net. – The use of a “prioritized” semantic net that integrate domain specific extensions (legal, medical, geographical, etc.). – The use of “related concepts” as derived from the application of Latent Semantic Analysis. 3
Participation to ImageCLEF was aimed at testing the system in a completely new and unseen setting. CELI subscribed to the ImageCLEF experiment just few days before the deadline, so no tailoring of the system on the basis of the corpus was performed, nor any kind of experimental run using topics from previous years.
We submitted a total of 8 runs, each one with Italian topic descriptions (the title tag) targeting an English corpus of image metadata. From the IAPR corpus we indexed the title, description, notes and location fields. Out of eight runs, half of them put all translations in OR, whereas the remaining ones use AND among different translated query terms and OR to relate different translations and/or expansions of the same term. Each quartet of runs is then partitioned into the above mentioned query expansion strategies plus a a run with no expansion at all. 4
As we stated in the introduction, the goal of the system is to act as cross language interface to
mainstream search engines. It is not therefore surprising that, after research by [
Run name CELI-AND CwnExpansion CELI-AND NOEXPANSION CELI-OR NOEXPANSION CELI-AND CwnCascadeExpansion CELI-OR CwnExpansion CELI-AND LisaExpansion CELI-OR LisaExpansion CELI-OR CwnCascadeExpansion Here we report results only for runs based on titles, as these share obvious similarities with ImageCLEF topics, for which, for Italian, only the title was available.
The first thing we notice is basically an average similar performance for precision at 10 and 20, in spite of a difference of about 10% in terms of mean average precision. Given the fact that rarely users inquiry beyond the 20th result, this is an indication that metadata-based cross language information retrieval can perform as well as full text information retrieval. This is in turn a good new for digital libraries for which only a small part of texts has been digitalized.
The second important thing is that while in the Ad-hoc track absence of query expansion seems to provide better results, in the case of image retrieval, the reverse applies. We think that this is an effect of the fact that metadata are usually shorter and the language is generally more controlled. It is therefore unlikely that synonyms might occur in the description of the same image. Query expansion will then intervene to bridge between the uncontrolled nature of the query and the controlled nature of the metadata.
Finally we observe that whereas LSA based expansion was ranked as the best expansion strategy in the Ad-hoc task, it performed very badly in the Image track. So far we do not have a clear explanation for this behavior. However we suspect that this is due to the fact that LSA expansion focuses on generic “related terms”. These are likely to occur in documents centered around a given topic, but they are usually absent in the description of a picture. For instance if a news article is about a church, it might contain also the adjective Christian or the noun religion. By contrary, the description of a photo of a church is not likely to contain anything else apart from visually perceivable features. 5
The most evident limit of CELI’s participation to ImageCLEF was of course the lack of any capability of performing visual retrieval. We estimate, however, that in the context of delegated search this is not a major drawback, as, in any case, mainstream image search engines are not likely, in the immediate future, to provide search restrictions based on visual patterns. If this were the case, we also estimate that standard web users would go on looking for content based features rather than visual features. A query for pictures of a “Mercedes E200 coup´e” will probably be always more common than “a brown Jaguar on a green background”.
In terms of retrieval based on metadata we estimate that big improvements are possible according to the following lines: • Better recognition of named entities and evaluation if they should be translated or not. • Expansion of named entities, in particular as far as geographical data are concerned. • More selective use of metadata (currently they are just indexed as if their concatenation was the text of a document). • Introduction of some basic reasoning capabilities. These might be considered as a sort of query expansion, but their implementation might vary. We are thinking to basic skeletons of temporal reasoning (an old car might have been indexed as a 1974 car) as well as some ontology based inferences (A lion is an animal) .
Improvements on these points will be our goal for the next ImageCLEF experiment.