This paper presents the 2007 MIRACLE's team approach to the AdHoc Information Retrieval track. The work carried out for this campaign has been reduced to monolingual experiments, in the standard and in the robust tracks. No new approaches have been attempted in this campaign, following the procedures established in our participation in previous campaigns. For this campaign, runs were submitted for the following languages and tracks: - Monolingual: Bulgarian, Hungarian, and Czech. - Robust monolingual: French, English and Portuguese.
The MIRACLE team is made up of three university research groups located in Madrid (UPM, UC3M and UAM) along with DAEDALUS, a company founded in 1998 as a spin-off of two of these groups. DAEDALUS is a leading company in linguistic technologies in Spain and is the coordinator of the MIRACLE team. This is our fifth participation in CLEF. As well as monolingual and robust multilingual tasks, the team has participated in the ImageCLEF, Q&A, and GeoCLEF tracks.
The MIRACLE Information Retrieval toolbox is made of basic components: stemming, transformation (transliteration, elimination of diacritics and conversion to lowercase), filtering (elimination of stop and frequent words), proper nouns detection and extracting, and paragraph extracting, among others. Some of these basic components can be used in different combinations and order of application for document indexing and for query processing. Through our participation in previous campaigns, the integration procedure of the different modules is stable and, to some point, optimized.
MIRACLE makes use of its own indexing and retrieval engine, which is based on the trie data structure 0. Tries
have been successfully used by the MIRACLE team for years, as an efficient storage and retrieval of huge
lexical resources, combined with a continuation-based approach to morphological treatment [
• Robust monolingual: French, English and Portuguese.
MIRACLE toolbox has already been described in previous campaigns papers [
Extraction: The extraction treatment has a special filter for extracting topic queries in the case of the use of the narrative field: some patterns that were obtained from the topics of the past campaigns are eliminated, since they are recurrent and misleading in the retrieval process. For example, for English, we can mention patterns as “… are not relevant.”, or “…are to be excluded”. All the sentences that contain such patterns are filtered out.
Paragraphs extraction: We have not used paragraph indexing this year, since the results we have obtained in this campaign and past ones have been disappointing.
Tokenization: This process extracts basic text components, detecting and isolating punctuation symbols. Some basic entities are also treated, such as numbers, initials, abbreviations, years, and some proper nouns (see next item). The outcomes of this process are only single words, years that appear as numbers in the text (e.g. 1995, 2004, etc.), or entities.
Entities: We consider that entities detection and normalization plays a central role in Information Retrieval, but it is a difficult task. For this year we have integrated a special module in the tokenization process that detects and marks some entities that have been previously collected from several sources into a lexical database for entities. These entities, which can be people names, place names, initials, abbreviations, etc., can consist of one or more words and special symbols, and their correct treatment is integrated into the tokenizer. For now, no entity normalization is done, so the same entity can appear in different forms and these are treated as different entities.
Filtering: Stopwords lists in the target languages were initially obtained from [
Transformation: The items that resulted from tokenization were normalized by converting all
uppercase letters to lowercase, and accents eliminated. This has not been done for Bulgarian.
Stemming: We used standard stemmers from Porter [
Indexing: When all the documents processed through a combination of the former steps are ready for
indexing, they are fed into our indexing trie engine to build the document collection index.
Retrieval: When all the documents processed by a combination of the former steps are topic queries,
they are fed to an ad-hoc front-end of the retrieval trie engine to search the previously built document
collection index. In the 2007 experiments, only OR combinations of the search terms were used. The
retrieval model used is the well-known Robertson’s Okapi BM-25 [
The following table and graphic representation summarize the performance of our official experiments in the monolingual tasks (using the topic fields title/description).
lang bg cz hu In the case of the monolingual robust task, only the results for English will be shown, as our PT and FR runs did not match the interpretation made in the assessment concerning the available collections and topics included in each experiment. In the results, the mean average precision figures are given.
lang en
Average Precision 0.3778
This year we have not changed our previous processing scheme, using the same improvements incorporated last year regarding proper nouns and entities detection and indexing. For this reason, the obtained results must be quite similar to previous ones. The only element that makes the processing of each language different has to do with the stemming components and stopword lists.
It is clear that the quality of the tokenization step is of paramount importance for precise document processing. We still think that a high-quality entity recognition (proper nouns or acronyms for people, companies, countries, locations, and so on) could improve the precision and recall figures of the overall retrieval, as well as a correct recognition and normalization of dates, times, numbers, etc. Although we have introduced some improvements in our processing scheme during the last years, a good multilingual entity recognition and normalization tool is still missing.
This work has been partially supported by the Spanish R+D National Plan, by means of the project RIMMEL (Multilingual and Multimedia Information Retrieval, and its Evaluation), TIN2004-07588-C03-01; and by the Madrid’s R+D Regional Plan, by means of the project MAVIR (Enhancing the Access and the Visibility of Networked Multilingual Information for Madrid Community), S-0505/TIC/000267.
Special mention to our other colleagues of the MIRACLE team should be done (in alphabetical order): Ana María García-Serrano, Ana González-Ledesma, José Mª Guirao-Miras, José Luis Martínez-Fernández, Paloma Martínez-Fernández, Antonio Moreno-Sandoval and César de Pablo-Sánchez.