In this paper, we present our participation in the ImageCLEF 2005 ad-hoc track. First, we describe a preliminary pool of cross-language experiments with the ImageCLEF 2004 testbed performed in order to evaluate the impact of different-size dictionaries using three distinct approaches. These differences are not remarkable, however recognizing named entities and launching structured queries over the metadata improve the results in all cases. Then, we decided to refine our named entities recognizer and repeat the three approaches with the 2005 topics, achieving the best result among all cross-language European Spanish→English runs.
conrfim interesting differences among approaches: runs based on structured queries were substantially better than the others.
On the other hand, we describe the UNED’s participation in the ImageCLEF 2005 track. Given the benefits of recognizing named entities in the topics in order to structure the queries, we decided to improve our recognition process. Now we are able to locate and identify more complex proper nouns, temporal references and numbers. Then, we performed the three approaches over the 2005 testbed obtaining the first and second best cross-lingual runs in European Spanish, representing the 94% of our monolingual experiment.
The structure of this paper is the following: in Section 2, we explain the preliminary set of experiments using the ImageCLEF 2004 testbed and different-sized bilingual dictionaries. Then, in Section 3, we focus on our participation in ImageCLEF 2005. In section 3.1, we describe this year’s settings, in section 3.3 the runs submitted to this year’s edition of ImageCLEF are presented and then, in section 3.4, we comment the results obtained. Lastly, in Section 4, we draw some conclusions. 2 2.1
For our preliminary series of experiments, we used the Spanish-English ImageCLEF 2004 ad-hoc
track testbed [
Every original topic in English consisted of a title and a more extensive narrative fragment describing an information need. We used the Spanish version of the topics, in which only titles were translated.
Images had an accompanying textual description consisting of eight human-annotated metadata fields, such as: a unique ID, both short and long titles, the location where the image had been taken, a description of the image, the date, the author’s name and some general categories in which the photograph may be included (e.g. [ferries], [woods & forests], [panoramic views], . . . ). Even though this information was not always complete, we decided to use it in order to improve our retrieval of relevant images.
All retrieval runs have been done with the Inquery search engine [
In a rfist step, our simple recognizer uses a straightforward set of rules in order to locate all named
entities appearing in the ImageCLEF topics and looking like such (see [
Then, we attempt to classify these possible proper nouns and temporal references by checking if they appear as author’s names, locations and dates in the collection. If they do, we build the query re-structuring this information so that our search engine favors those images with such metadata.
The procedure is the following:
1. If it is a proper noun, we ask the search engine to find any document containing the entity
in the “author” or “location” efilds. If the search is non-nil, we assume that the role of the
entity is the efild in which it was found.
2. If it is a numerical expression, we ask the search engine to find any document containing
the entity in the “date” efild. If the search is non-nil, we assume that the cardinal number
represents a date.
3. Finally, if it is a temporal reference, we check if it is a date in a similar fashion (see [
We compared the three following approaches:
i) A naive baseline using a word by word translation. Words which were not present in our
bilingual dictionary were left untranslated. For instance, the query for topic 2 (Fotos de Roma
que fueron tomadas en Abril de 1908 ) was translated and built using Inquery’s operators as:
#sum(pointless unpointed tomada taken taken take assume take take get espouse take on
take take capture seize have take imbibe take april 1908)
ii) strong baseline following Pirkola’s proposal [
We tried all three conditions with six different bilingual dictionaries: FreeDict a freely available on line dictionary; EWN generated from the official EuroWordNet multilingual semantic network; EWN2 compiled from an updated version of the Spanish Wordnet; Vox an electronic version of the Vox-Harraps Spanish-English dictionary; 1 All-Vox a combination of all the dictionaries above except Vox; and finally, All a merged version of all four dictionaries.
Finally, we evaluated three additional runs for comparison purposes: two monolingual runs (a straight run with the English version of the query, and an enhanced run with the field search strategy described in Section 2.2) and an additional cross-language run where named entities and temporal references are annotated manually. The latter was intended to evaluate the effects of errors in the automatic location of entities. 2.4
For all bilingual dictionaries, our structured query approach was better than the naive and Pirkola baselines. Pirkola’s approach was, in turn, substantially better than its naive counterpart in all cases. However, only the differences between our structured query approach and the naive baselines were relevant according to a non-parametric Wilcoxon sign test (in half of the cases).
The differences among dictionaries were not statistically significant either. For most of the runs, the translations provided by EWN2, All-Vox or Vox separately were enough to reach the highest precision values.
1This is the only genuine bilingual dictionary that we used and we took it as the basis for a merged version of all dictionaries.
Our best runs achieved an average precision of .54, which represents 91% of our best
monolingual run. This result slightly outperformed the best official cross-language run in the ImageCLEF
2004 evaluation (which was .53, obtained by Dublin City University with the DE → EN language
pair). For further details about the results, see also [
Remarkably, the manual annotation of named entities did not improve the results obtained with our simple automatic recognition strategy. This is an indication that the field search strategy is reasonably robust: for instance, if an expression is misinterpreted as a person name, it will probably not appear in the author efild and, therefore, precision will hardly be affected. 3 3.1
As in the previous edition, the testbed provided to ImageCLEF 2005 ad-hoc task participants was the St Andrews image collection. In this case, the participants were given 28 topics, each containing a title and a narrative fragment with verbose details about an information need.
Besides, this year there have been proposed two distinct set of Spanish topics which tried to show the local variants of the language: one European Spanish translation and another Latin American version. Even though the topics had been translated into Spanish wholly, we only took the short titles in our experiments. 3.2
Regarding our last year’s experience, some improvements have been done in our named entities recognition process. Now, we can locate more complex multi-word proper nouns and temporal references by attaching several simple entities of the same type usually connected by articles, prepositions and conjunctions. And so, our recognizer is able to locate some Spanish named entities such as the ones shown in Table 1.
In Table 2, we show the named entities located in the ImageCLEF 2005 European Spanish topics.
It is worth mentioning that topics proposed contained fewer expressions likely to be named entities than last year. Indeed, no temporal reference or number was located and we only could take advantage of the improvements of the recognizer in 6 out of 28 topics. Regarding the precision of the recognition, notice that the entities located in this year’s topics are the same that a user would have manually selected. 3.3
We submitted to ImageCLEF 2005 track vfie different runs, based on the same runs we had already tested in Section 2.3. First, one monolingual run in order to establish the maximum precision that we could achieve using our resources. Then, a naive run building the queries with a simple word by word translation. We also submitted two runs based on the strong baseline with the synonymy’s operators which allowed us to enrich and expand the translations while minimizing the noise. Lastly, we repeated the run adding the field search operators.
The features that define each of these runs are shown in Table 3. 3.4
The official results obtained by our five runs are shown in Table 4. First of all, it is worth mentioning that our cross-lingual run enriched with named entities unedESENEnt obtained the best MAP score among all official cross-lingual runs having European Spanish as the source language. Its counterparts without using the named entities unedESEN and unedESAmerEN got comparable results: .28 (3rd position in European Spanish) and .26, respectively. On the other hand, our simpler cross-lingual run achieved .19. In this paper, we have presented our participation in the ImageCLEF 2005 ad-hoc track.
First, we have described a preliminary pool of cross-language experiments with the ImageCLEF 2004 testbed performed in order to evaluate the impact of different-size dictionaries using three distinct approaches. We outperformed our ImageCLEF 2004 participation but the differences among dictionaries were not remarkable. However, in all cases, our results dramatically improved when recognizing named entities and launching structured queries over the metadata. So, we decided to renfie our named entities recognizer and repeated the three approaches with the 2005 topics, achieving the best result among all cross-language European Spanish→English runs.
Therefore, automatic query structuring seems an effective strategy to improve cross-language retrieval on semi-structured texts. Remarkably, no sophisticated named entity recognition machinery is required to benefit from query structuring. Of course, it remains to be checked whether this result holds on collections with different metadata fields and different textual properties.
This work has been partially supported by the Spanish Government under project R2D2-Syembra (TIC2003-07158-C04-02). Vcı´tor Peinado holds a PhD grant by UNED ( Universidad Nacional de Educacoi´n a Distancia ). Dimensions Query language Initial query Query type Feedback/expansion Modality unedmono unedESENnaive unedESEN
EN ES (Eur.) ES (Eur.) title title title automatic automatic automatic
X √ √ text text text unedESamerEN unedESENent
ES (Amer.) ES (Eur.)
title title automatic automatic √ √ text text
run unedmono unedESENEnt unedESEN unedESAmerEN unedESENnaive