In this paper, we present the experiments for the first participation of the UNED's NLP Group in the ImageCLEF bilingual ad hoc task. Two different strategies are attempted: a) Expanding and translating queries with noun phrases, and b) Performing structured searches using entities located in topic titles over image caption fields. All our experiments obtain results above the average MAP for the Spanish task. Our best result (using only structured searches over image captions) obtains 88.18% regarding the monolingual experiment and behaves a 8.3% better than Pirkola's structured queries. On the other hand, noun phrases expansion, which had proved successful in interactive tasks, influences negatively the results, due to the mismatch between the training corpus and the ImageCLEF collection.
1. Expand queries with noun phrases. We expanded the queries using a bilingual SpanishEnglish phrase-based dictionary whose phrases had been automatically extracted from the CLEF news comparable corpus. ∗ NLP Group website available at http://nlp.uned.es 1See http://ir.shef.ac.uk/imageclef2004/stand.html for further details. 2. Perform structured searches using the named entities and dates over the image caption fields, so that: a) Proper names automatically detected in the query are searched in the “author” and/or “location” fields. If the search is non-nil, the search engine will favor images containing these entities tagged as author or location within the caption. b) Temporal references and numbers in the query are searched in the same fashion on “date” fields. The latter were used on the assumption that they may be understood as years.
Two ideas, based on previous works, made us choose these strategies:
1. Expansion with Natural Language techniques is more beneficial with short queries [
This paper is structured as follows: in section 2, we firstly discuss the possibilities of using noun phrases in query expansion and interactive tasks, we present the linguistic resources used (section 2.1), the preliminary CLIR experiments we performed previously over the CLEF collection (section 2.2) and we explain the settings of the ImageCLEF experiments (section 2.3). Below, we explain the procedure of structured searches over image captions used in this work (section 3) and comment the results of our official experiments (section 4). Finally, in section 5, we draw some conclusions and propose future work. 2 2.1
We used two comparable corpora, the Spanish and English CLEF corpora made up of the Spanish collection from the EFE news agency and the English collection from the Los Angeles Times newspaper, both of them from 1994.
We also had a bilingual dictionary containing a set of more than vfie million aligned noun
phrases2 which had been automatically recognized and extracted from the CLEF corpora using
statistical data such as the frequency of sequences of two or three informative words (nouns and
adjectives) in both languages [
Previous UNED participations in the iCLEF tracks3 showed how helpful noun phrases could
be in document selection [
3The Interactive track for the Cross-Language Evaluation Forum webpage is available at http://nlp.uned.es/iCLEF.
Orange County abortion issue free trade agreement World War II
Condado de Orange
tema del aborto acuerdo de libre comercio
Segunda Guerra Mundial word translation (which can be made using a simple bilingual dictionary) and the translation of more complex structures such as sentences (often performed using a machine translation system).
Finally, we had a new bilingual Spanish-English dictionary made up from heterogeneous
lexicographic resources such as usual dictionaries (some of them freely distributed), word lists and
semantic networks such as WordNet [
In order to check the usefulness of noun phrases for a Cross-Language batch retrieval, rfistly, we proposed the three following baselines based on the translation word by word using bilingual dictionaries: naive baseline Word by word translation, building a bag of words with all the possible translations appearing in our dictionaries. frequencies In this experiment, we built a bag of words from only those possible translations appearing in more than one lexicographic source, assuming that those ones which had high frequencies in our dictionary were the most common and reliable.
Following this strategy, we pursued two goals: on one hand, we used only those translations
considered reliable, on the other hand, we rejected residuary translations from semantic
networks in order to reduce the noise produced by the expansion.
strong baseline We used Pirkola’s proposal [
Then, we decided to compare these baselines with more sophisticated combinations of the available resources, such as: systran The queries were translated using the Systran machine translation system. phrases + pirkola We used our noun phrases dictionary to expand the query with related noun phrases and translate them using a bilingual dictionary. Our strategy was the following: firstly, we expanded each topic term with the ten noun phrases containing it with the highest frequencies in the CLEF collection and then we translated them using their appropriate aligned phrases. Those query terms from which no phrases were identified were included in the translated query using Pirkola’s approximation, i.e. using synonymy operators. “multi-lemma” phrases + pirkola In order to limit the noise produced by the phrase expansion of the previous experiment, we made use only of those phrases which contain at least two query terms.
4Besides, when using this merged dictionary instead of the original VOX dictionary that we used in previous approaches, we obtain improvements in our CLIR experiments of about 36% phrases + pirkola + systran As a last experiment, we performed one run combining the three resources, i.e. noun phrases, structured translations from our dictionaires and the Systran machine translation system.
As shown in Table 2, our experimental proposals using the noun phrases outperformed the baselines. The differences were statistically significant, according to a non-parametric Wilcoxon sign test. Our strong baseline even obtained the same average precision than Systran’s translations, showing that the combination of Pirkola’s structured queries and reliable lexicographic resources are good enough when searching over a collection of short documents.
run naive baseline frequencies strong baseline systran phrases + pirkola phrases + pirkola + systran ‘‘multi-lemma’’ phrases + pirkola
There is no need to use external machine translation systems, at least when translating small documents.
The “multi-lemma” variant of noun phrase expansion performs slightly better for batch CLIR, although the difference is not statistically relevant according to a Wilcoxon sign test.
The quality of a translation strongly depends on the resources used. 2.3
Extending the above results to the ImageCLEF bilingual ad hoc task, we have used the corpus and the set of Spanish topics provided by the organization, our bilingual XML dictionary, the Systran machine translation system5 and the set of aligned noun phrases between English and Spanish.
Topic title were manipulated as usual in Cross-Language Information Retrieval tasks. Punctuation and stopwords were removed6 and informative words were lemmatized before attempting the translation using the dictionary or the expansion by means of the noun phrases.
Since the image captions contain structured information, we decided to use it identifying which query terms could be understood as authors, locations or dates.
• • • 3 3.1
Before performing structured searches over the caption fields, we needed to locate the entities appearing in each topic title. So, we used our own recognizer which is able to find: 5Systran web-based interface available at http://www.systransoft.com 6In order to adapt the stopword list to this specific task, we included as stopwords fotograıaf´ s , fotos (photographs), retrato (portrait). . .
Named entities Expressions in uppercase wherever uppercase is not prescribed by punctuation rules.
Temporal references Those ones matching words such as names of weekdays, months or seasons.
Numbers Those ones matching any numerical expression or words from a given list (e.g. dos (2), cien (100), mil (1,000) . . . ) 3.2
After having located these entities, we performed the following procedure to identify the role of every entity in the ImageCLEF collection and to structure the query using the information appearing in the caption efilds.
For each entity located in the Spanish topic titles: • If it is a named entity, we ask the search engine to find any document containing the entity in the “author” or “location” fields, rfistly in Spanish and secondly in English. 7 If the search is non-nil, we assume that the role of the entity is the field in which it was found. • If it is a cardinal number, we ask the search engine to find any document containing the entity in the “date” field. If the search in non-nil, we assume that the cardinal number represents a date.
• If it is a temporal reference, we check if it is a date, in the same fashion. 3.3
In Table 3, we show the entities found for each Spanish topic title. Our recognizer located 31 entities (named entities, temporal references and cardinal numbers), although some of them are incorrect. For instance, on topic 5 Irlanda and Norte should have been identified as a unique named entity. Elisabeth, on topic 14, was not identified as a possible entity. Besides, on topics 11 and 13, expressions such as Postales and Campeonato Abierto were misidentified as named entities.
Entities such as Postales, Campeonato Abierto, Reina Madre and Segunda Guerra Mundial did not represent any author, location or date. In any case, our strategy did not identify them as such either.
Regarding the other located entities, a manual analysis about their roles showed that: authors Every possible author (Thomas Rodger, John Fairweather and George Middlemass Cowie) was correctly identiefid using this strategy. locations On one hand, Roma, Irlanda, Norte, British Columbia, Canada´ , Egipto, Londres, Bute, Escocia and York were correctly identiefid. On the other, St. Andrews, Cambridge, Tay Bridge, Crail Camp, North Street and Edimburgo was not. dates Abril, 1908, 1879, 1939, 1954, 1900. All dates were identified with this strategy.
Thus, this strategy strongly depends on the quality of the manual tagging of the collection. Sometimes it is not possible to identify the role of entities such as Tay Bridge or St. Andrews because they are not tagged this way in the image captions.
7We perform the search in both languages because there is no general rule for translating proper names. Entities were translated using Systran because of the lack of proper names in our dictionary.
Retratos de ministros de la iglesia por [NE Thomas Rodger].
Fotos de [NE Roma] que fueron tomadas en [DAT E Abril] de [CARD 1908]. Vistas de la catedral de [NE St. Andrews] por [NE John Fairweather]. Hombres vestidos militarmente, [NE George Middlemass Cowie]. Buques de pesca en [NE Irlanda] del [NE Norte].
Vistas panoar´micas en [ NE British Columbia], [NE Canada´].
Vistas exteriores de templos en [NE Egipto].
Edificios de la universidad o colegios universitarios, [ NE Cambridge]. Fotos de faros ingleses.
Calles en plena actividad en [NE Londres].
Tarjetas [NE Postales] con mu´ltiples vistas de [ NE Bute], [NE Escocia]. Desastre ferroviario en el [NE Tay Bridge], [CARD 1879].
Torneo del [NE Campeonato Abierto] de golf, [NE St. Andrews] [CARD 1939]. Elizabeth la [NE Reina Madre], en su visita a [NE Crail Camp], [CARD 1954]. Dan˜os provocados por bombas en la [NE Segunda Guerra Mundial]. Fotos de la catedral del [NE York].
Vistas de [NE North Street], [NE St. Andrews].
Fotos del castillo de [NE Edimburgo] antes de [CARD 1900] Gente marchando o desfilando.
Roı´ con un viaducto al fondo.
Monumentos a los cıad´ os en la guerra en forma de cruz.
Fotos mostrando tradicionales bailarines escoceses.
Fotos de cisnes en un lago.
Golfistas golpeando con sus palos de golf.
Barcos en un canal. 4 4.1
Given the experiences achieved after having performed the approaches explained on section 2.2, we decided to use the following strategies in our ImageCLEF experiments: • Naive baseline using a word by word translation (UNEDESBASE).
topic 13: turn tourney tournament tourney joust tilt championship title open frank open-minded opened overt unconcealed undone up extrovertish unfastened unlatched unlocked unsecured exposed hospitable forthright open-ended unresolved outgoing assailable undefendable undefended unhealed open undo dig head lead blossom unlock spread unfold brighten clear golf st andrews 1939 • Strong baseline using a structured query, following Pirkola’s approach (UNEDES).
This is the core of the structured query for the next approaches, using INQUERY’s synonymy operators: topic 13: #syn( turn tourney ) #syn( tournament tourney joust tilt ) #syn( championship title ) #syn( open frank open-minded opened overt unconcealed undone up extrovertish unfastened unlatched unlocked unsecured exposed hospitable forthright open-ended unresolved outgoing assailable undefendable undefended unhealed ) #syn( open undo dig head lead blossom unlock spread unfold brighten clear ) golf st andrews 1939 • Structured query using INQUERY’s operators and structured search over captions (UNEDESENT).
If some entity is located and identiefid as a possible author name, location or date, we include the structured search over the caption efilds. In this case, the search engine will favor those images in whose caption efilds 1939 is tagged as a date. So, the following operator is added to the previous query: #field( DDATE #sum( 1939 ) ) • Structured query using INQUERY’s operators and structured search over captions + noun phrases (UNEDESENTNOO and UNEDORENTNOO).
We detected several errors in the original Spanish query set. These were xfied and sent to ImageCLEF organizers for distribution among other participants. However, for completeness, we submitted the most complex runs both with the original and the xfied query set (UNEDORENTNOO and UNEDESENTNOO, respectively).
In order to expand the queries, we added the set of noun phrases extrated from the query terms using the “multi-lemma” phrases strategy. For topic 13 and UNEDESENTNOO, the phrases included are: #phrase( golf course manager ) #phrase( world golf championship ) #phrase( world championship tournament ) #phrase( first golf tournament ) #phrase( day after a golf tournament ) #phrase( chiefs into the title ) #phrase( clear the tournament ) #phrase( champions tournament ) #phrase( conference tournament title ) #phrase( day golf tournament ) #phrase( golf tournament last ) #phrase( league golf tournament ) #phrase( tournament of champions ) #phrase( phoenix golf tournament ) #phrase( tennis tournament in st ) #phrase( championship golf course ) #phrase( ups for golf ) #phrase( championships golf tournament ) #phrase( gains after a bond ) #phrase( final of the tournament of champions ) #phrase( tournament at st ) #phrase( bond gains ) #phrase( title of chief ) To sum up, the set of submitted runs and its features are shown on table 4. Our five experiments obtained very promising results, 8 and they all are placed above the average (notice that average MAP for Spanish is 0.30). The best one, UNEDESENT, has even gotten the second best result in the ranking among all the participants in the Spanish-English task, with losses of only 11.82% and 0.77% with regard to the monolingual experiment and the Sheffield best run,9 respectively, as shown in Table 5.
Structured queries over image captions (UNEDESENT) obtained an improvement of about 8.3 % with regard to the Pirkola’s proposal (UNEDES). It seems to indicate that such a simple detection 8Official results are available at http://ir.shef.ac.uk/imageclef2004/adhoc results/adhoc results.html. 9Sheffield’s es es fb has been the best run among all the Spanish participants. of entities can improve the average results, taking advantage of the information contained in the image captions.
Unsurprisingly, expansion using noun phrases does not improve the structured search. Our set of aligned noun phrases had been previously extracted from a different collection, out of the photographic domain. As shown on section 4.1, the expansion inserted too much noise in the form of unrelated noun phrases, such as “gains after a bond”, “chiefs into the title” . . .
Finally, notice that our weakest baseline (UNEDESBASE) is about 26% better than the average MAP for the Spanish participants which seems to point the reliability of our dictionary. run Sheffield UNEDESENT UNEDES UNEDESENTNOO UNEDORENTNOO UNEDESBASE average
In this paper, we have presented two different strategies applied to the ImageCLEF bilingual ad hoc task.
• Expand queries with noun phrases, translating and expanding the queries with noun phrases automatically extracted from a different corpus.
The results obtained by using this strategy seems to indicate that noun phrases should be extrated from the appropriate corpus, in order to avoid noise. • Perform structured search using entities located from topic titles over the image caption fields.
This straightforward refinement obtains an improvement of 8.3 % in relation to Pirkola’s baseline and can be easily extended to other searches over structured documents. • Given that our weakest baseline is above the average MAP, our basic linguistics resources have proved very useful.
As possible lines of future work, we are working on: • Extracting the noun phrases directly from the appropriate collection. So far, we have been using the CLEF comparable corpora due to their size but this decision implies too much noise. • Refining our recognizer in order to make a better detection of entities.
• Quitting using commercial machine translation systems which are out of our control.
This work has been partially supported by a grant from the Spanish Government, project R2D2 (TIC2003-07158-C0401), and a grant from the UNED (Universidad Nacional de Educacoi´n a Distancia).