The main goal of the bilingual and monolingual participation of the MIRACLE team at CLEF 2004 was testing the effect of combination approaches to information retrieval. The starting point is a set of basic components: stemming, transformation, filtering, generation of n-grams, weighting and relevance feedback. Some of these basic components are used in different combinations and order of application for document indexing and for query processing. Besides this, a second order combination is done, mainly by averaging or by selective combination of the documents retrieved by different approaches for a particular query.
The MIRACLE team is constituted by three university research groups located in Madrid (UPM, UC3M and UAM) along with a company, DAEDALUS, started up in year 1998 as a spin-off from two of them. DAEDALUS is a leading company in linguistic technologies in Spain1, and acts as the coordinator of the MIRACLE team. This is the second participation in CLEF, after year 2003 [6] [7]. Besides bi and monolingual tasks, the team has participated in the ImageCLEF and Q&A tracks.
The main purpose of the bi and monolingual participation was testing the effect of combination approaches to information retrieval. The starting point is a set of basic components: stemming, transformation (transliteration, elimination of diacritics and conversion to lowercase), filtering (elimination of stop and frequent words), generation of n-grams, weighting (giving more importance to titles) and relevance feedback. Some of these basic components are used in different combinations and order of application for document indexing and for query processing. Besides this, a second order combination is done, mainly by averaging or by selective combination of the documents retrieved by different approaches for a particular query. When evidence is found about better precision of one system at one extreme of the recall level (i.e. 1,0), complemented by better precision of another system at the other recall end (i.e. 0,0), then both of them are combined to benefit from their complementary results.
On the other hand, our group has been developing during the last year an indexing system based on the trie data structure [5]. Tries [4] are successfully used by the MIRACLE team for an efficient storing and retrieve of huge lexical resources, combined with a continuation-based approach to morphological treatment. However, the adaptation of these structures to manage efficiently document indexing and retrieval for commercial applications has been a hard task. The currently available prototype shows a strong improvement of performance (both indexing and retrieval times are considerably reduced). However, this system was not fully operative for this CLEF campaign. So, the Xapian [3] indexing system, robust, efficient, and well suited for our purposes, was used as in the last campaign. 1 DAEDALUS clients include leading companies in different sectors: media (EL PAÍS), publishing (Grupo SM), telecommunication (Grupo Telefónica), digital rights management (SGAE), photography (StockPhotos) and the reference institution for the Spanish language, Instituto Cervantes. Its portfolio of solutions includes STILUS® (professional spell, grammar and style checking of texts in Spanish), K-Site® (information retrieval, fuzzy search and knowledge management), LUCAS (universal locator of audiovisual contents, an Internet spider), etc. For this year, we have submitted runs for the following tracks: a) Monolingual Russian. b) Monolingual French. c) Bilingual Dutch to French.
d) Bilingual German to French.
The Xapian system has been the basic indexing and retrieval tool for bilingual and monolingual experiments for the MIRACLE group. Before being indexed, document collections have been pre-processed using different combinations of scripts, each one oriented to a particular experiment. For each one of these, topic queries have been also processed by the same combination of scripts (although in some cases some variants have been used, as will be described later).
The baseline approach to processing documents and topic queries is composed of the following sequence of steps: 1. Extraction: Ad-hoc scripts are run on the files that contain particular documents or topic queries collections, to extract the textual data enclosed in XML marks. We used all the marks permitted for automatic runs (depending on the particular collection, all of the existing TEXT, TITLE, LEAD1, TX, LD, TI, or ST for document collections, and the contents of the TITLE and DESC marks for topic queries – NARR marks contents were systematically ignored). The contents inside these marks were concatenated to feed the following steps. However, in some experiments only the titles were extracted (including in the run identifier the strings titnormal, titnostem or titngrams), and in some normal experiments in monolingual Russian, the terms appearing inside the TITLE marks were given more importance by repeating them several times more (these experiments include in the identifiers the strings normaltit1, normaltit2 or normaltit3, when the titles terms are included one, two or three times more). 2. Parsing: A simple parsing process is made for eliminating punctuation signs and detecting basic indexing chunks (usually words, by some basic entities can be detected, as compounds, proper nouns, and so on). We think that the quality of this step is of paramount importance in all the document processing. A high-quality entity recognition (proper nouns or acronyms for people, companies, countries, locations, and so on) could improve the precision and recall figures of the overall retrieval, as well as it could a proper recognition and normalization of dates, times, numbers, etc. 3. Lowercase words: All document words are normalized by converting all uppercase letters to lowercase. 4. Stopwords filter: All the words known as stopwords are eliminated from the document. Stopwords in the target languages were initially obtained from [1], but were completed using other several sources and using own knowledge and resources. 5. Stemming: The process known as stemming is applied to each one of the words of the document.
The stemmer used is the one referenced in [2]. 6. Remove accents: All document words are normalized by eliminating accents in stemmed words.
Note that this process can be done before stemming, but resulting lexemes are different. In spite of that, some experiments have been made doing this step before stemming.
a. Indexing: When all the documents processed by the former steps are to be indexed, they are fed to a Xapian ad-hoc front-end to build the Xapian document database. b. Retrieval: When all the documents processed by the former steps are topic queries, they are fed to a Xapian ad-hoc front-end for searching the previously built Xapian index. In our experiments for this year we have only used OR combinations of the search terms.
In the case of the Russian language the basic processing steps described above is slightly changed, due to the different encodings of the Russian files and the resources used for Russian: while document collection and topics files were encoding in UTF8, as well as stopwords resources, the stemming resources worked in KOI8, so some recoding steps were added in appropriate processing points. In addition to that, some other tools did not work properly with the UTF8 encoding, so some workarounds had to be added: (a) The parsing process was simplified even more, using a sed script to achieve basic punctuation processing, and (b) a transliteration of the files to the ASCII charset was needed in order to get the XAPIAN indexing system to work. The transliteration script used was the one available in reference [1].
In addition to the baseline experiments (identified with the suffix normal in the run identifiers), other experiments have been also defined as variations of these: If the stemming step is not made, we identify the resulting experiments with the suffix nostem, where the actual word forms appearing in the documents are used for indexing and retrieval. We also tried a variant of the nostem experiments, where we obtained particular ngrams from each of the actual word forms in the documents. We denoted these with identifiers of the form ngramsXY, where X is the length of the n-grams and Y the number of characters that overlap between two consecutive n-grams. (For example, in an experiment denoted by an identifier with the suffix ngrams54, from president we would obtain the n-grams: “_pres”, “presi”, “resid”, “eside”, “siden”, “ident”, and “dent_”. The symbol “_” is introduced to denote word boundaries. Note that four characters overlap between two consecutive n-grams).
In the case of the topic queries, an additional variation is introduced: the FW (Frequent Words) filter is applied by filtering out of the queries the 20 most frequent words, or stems, that appeared in the corpora, as well as some typical query terms. These variants were identified by using the FW string in the run identifier. The Xapian engine allowed us to use relevance feedback, so we used such technique in several experiments. When the terms of the first documents retrieved in the first retrieval step are fed back to a second retrieval step, we used the strings R1, R2, R3, R4 or R5, in the run identifier depending on the actual number of documents used. Note that using relevance feedback does not affect to an indexing process, and can be used in any of the variants used for processing the documents collections or the topics queries.
For translation purposes, the SYSTRAN system was used. Our tests done on the collections and topics of CLEF 2003, showed that SYSTRAN outperformed other on-line translators on the selected pairs of languages (Dutch to French and German to French) when used to find documents in the French collections from queries in Dutch or German. As other pairs as Finnish and Swedish to French where not available on-line in SYSTRAN, other translators where tested, with very poor results.
Not all the possible combinations of the variants described in the previous section were tried in the experiments, due to evident limitations of resources and time. The experiments were tried in a rather intuitive, non-systematic way, trying to test a wider and richer set of trials. To compare these approaches, we used these techniques following the instructions given for CLEF 2003 (corpora and topic queries) and using the appropriate qrels available at the beginning of this campaign. The appendix includes all the data that compares the results that we obtained in the experiments. The tables show the precisions at recall points 0 and 1, the average precision, the percentage of the latter with respect to the best average precision experiment (the first one in each table) for each of the experiments. The best value is marked with the symbol “*”. The comb column in each table indicates if the experiment is a combined experiment, what will be described in the next section, and the sel column shows what experiments were selected for CLEF 2004, usually the ones with a better result in precision (regarding CLEF 2003 experiments).
The appendix also includes the results for the same experiments for the CLEF 2004 campaign (the relevant tables mark the experiments submitted in the sub column), once the qrels for this campaign have been available. In the following figures, the results obtained by the best baseline experiments submitted to CLEF 2004 are compared with the results obtained by exactly the same system when applied to the 2003 tasks. The comparison shows qualitative differences between the 2003 and 2004 topics. No figure is presented for French, as all the submitted runs were in this case obtained through combination (see the next section).
0,7 0,6
In this campaign, we have made some tests that try to combine the results from the basic experiments in different ways. We thought that to some extent, the documents that have a good score in almost all experiments are more suitable to be relevant that other documents that have good score in one experiment but a bad one in others. We have tried two strategies for combining experiments:
Average: The relevance figures obtained in the Xapian probabilistic retrieval in all the experiments to be combined for a particular document in a given query are added. This approach combines the relevance figures of the experiments without giving more importance to a particular experiment.
Asymmetric DWX combination: In this particular type of combination, two experiments are combined in the following way: The relevance of first D documents for each query of the first experiment is preserved for the resulting combined relevance, whereas the relevance for the remaining documents in both experiments are combined using weights W and X. We have only ran experiments labeled “101” and “201”, that is, the ones that get the one (or two) documents more relevant from the first basic experiment and all the remaining documents retrieved from the second basic experiment, then re-sorting all these results using the original relevance.
Average combinations get better figures in average precision or in precisions at 0 or 1 points of recall, than the original basic experiments. The reason could be that good relevant documents that appear with a high score in the experiments combined are strengthened. The asymmetric “101” or “201” that we have used do not get improvements as the average combinations do.
frnormal frnormal frR1FWnormal frnormalinv frnormal frR1FWnormal frnormalinv frnormal frR1FWnormal frnormalinv frnostem frnormal frR1FWnormal frnormalinv frnostem Frtitnormal
Basic experiments ruR1FWnormal ruR1FWnormal ruR1FWnormal ruFWnormal ruR1FWnormal ruFWnormal ruR2FWnormal ruR1FWnormal ruFWnormal ruR2FWnormal runostem
nl2frcomb1s101
Asym101 nl2frav de2frcomb1s101 de2frcomb2s201 de2frav Average Asym101 Asym201 Average nl2frFWnormal nl2frR1FWnormal nl2frR4FWnormal de2frFWnormal de2frR3FWnormal de2frR1FWnormal de2frR4FWnormal
nl2frR4FWnormal nl2frR2FWnormal nl2frR5FWnormal fe2frR3FWnormal De2frFWnormal de2frR2FWnormal de2frR5FWnormal In the following figures, the results obtained by the best combined experiments submitted to CLEF 2004 are compared with the results obtained by exactly the same systems when applied to the 2003 tasks. The comparison shows again the qualitative differences between the 2003 and 2004 topics.
Monolingual Rusian - run ruav7
Monolingual French - run frcomb1s201
Runormal
rungrams54 nl2frR3FWnormal
nl2frFWnormal de2frR3FWnormal de2frFWnormal 0,7 0,6 0,5 ion0,4 s iec0,3 r p0,2 0,1 0 0,9 0,8 0,7 n0,6 iso0,5 ice0,4 rp0,3 0,2 0,1 0 0,0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1,0 0,7 0,6
The combination approach seems to improve slightly the precision results for IR retrieval tasks, although an indepth analysis of the reasons for that is still needed. The differences shown between the 2003 and 2004 experiments seem to be highly idiosyncratic, dependent to a great extent on the different topics selected each year. This is particularly true in the case of Russian, due to the low number of documents relevant for the topics set. Regarding the basic experiments, the general conclusions were known in advance: retrieval performance can be improved by using stemming, filtering of frequent words, appropriate weighting and relevance feedback with a few documents. On the other side, n-grams performed worse than expected.
Future work of the MIRACLE team in these tasks will be directed to several action lines: (a) Getting better performance in the indexing and retrieval phases, to be able to make experiments in a more efficient way (indexing times for huge documents collection is now excessive for a flexible scheduling of experiments). This will be achieved using our own trie-based libraries for the indexing and retrieval phases. (b) Improving the first parsing step: we think that this is one of the most critical processing steps that can improve the overall results of the IR process. A good entity recognition and normalization is still missing in our processing scheme for these tasks.
This work has been partially supported by the projects OmniPaper (European Union, 5th Framework Programme for Research and Technological Development, IST-2001-32174) and RIMMEL (Multilingual and Multimedia Information Retrieval and its Evaluation”, Spanish Ministry of Science and Technology, years 2004-2007). The participation of the MIRACLE team in year 2003 was partially funded by the Regional Government of Madrid through the research project “MIRACLE: Multilingual Information Retrieval System and its Evaluation under the CLEF European Initiative” (07T/0055/2003) and through its Entrepreneurship Innovation Programme (Madrid Innova, project PIE/594/2003). [1] University of Neuchatel page of resources for CLEF (Stopwords, transliteration, stemmers …). On line http://www.unine.ch/info/clef/. [Visited 11/08/2004] [2] Snowball stemmers and resources. On line http://www.snowball.tartarus.org. [Visited 11/08/2004] [3] Xapian: an Open Source Probabilistic Information Retrieval library. On line http://www.xapian.org.
[Visited 11/08/2004] [4] Jun-Ichi Aoe, Katsushi Morimoto, Takashi Sato: An Efficient Implementation of Trie Structures.
Software Practice and Experience 22(9): 695-721, 1992. [5] Goñi-Menoyo, José Miguel; González-Cristóbal, José Carlos and Fombella-Mourelle, Jorge. An optimised trie index for natural language processing lexicons. MIRACLE Technical Report.
Universidad Politécnica de Madrid, 2004. [6] Martínez, J.L.; Villena-Román, J.; Fombella, J.; García-Serrano, A.; Ruiz, A.; Martínez, P.; Goñi, J.M. and González, J.C. (Carol Peters, Ed.): Evaluation of MIRACLE approach results for CLEF 2003.
Working Notes for the CLEF 2003 Workshop, 21-22 August, Trondheim, Norway. [7] Villena-Román, J.; Martínez, J.L.; Fombella, J.; García-Serrano, A.; Ruiz, A.; Martínez, P.; Goñi, J.M. and González, J.C. (Carol Peters, Ed.); MIRACLE results for ImageCLEF 2003. Working Notes for the CLEF 2003 Workshop, 21-22 August, Trondheim, Norway.
X X X
X comb
X X X X X X X X X X sub X X X X X X
X comb Sub X X X X
run id de2frR5FWnormal de2frR4FWnormal de2frR2FWnormal de2frR3FWnormal de2frav de2frR1FWnormal de2frcomb1s101 de2frFWnormal de2frcomb2s201 de2frnormal comb
X X comb sub comb
X X X X X X