This paper presents the 2005 Miracle's team approach to Monolingual Information Retrieval. The goal for the experiments in this year was twofold: continue testing the effect of combination approaches on information retrieval tasks, and improving our basic processing and indexing tools, adapting them to new languages with strange encoding schemes. The starting point was a set of basic components: stemming, transforming, filtering, proper nouns extracting, paragraph extracting, and pseudo-relevance feedback. Some of these basic components were used in different combinations and order of application for document indexing and for query processing. Second order combinations were also tested, by averaging or selective combination of the documents retrieved by different approaches for a particular query.
The MIRACLE team is made up of three university research groups located in Madrid (UPM, UC3M and UAM)
along with DAEDALUS, a company founded in 1998 as a spin-off of two of these groups. DAEDALUS is a
leading company in linguistic technologies in Spain and is the coordinator of the MIRACLE team. This is the
third participation in CLEF, after years 2003 and 2004 [
The starting point was a set of basic components: stemming, transformation (transliteration, elimination of diacritics and conversion to lowercase), filtering (elimination of stop and frequent words), proper nouns extracting, paragraphs extracting, and pseudo-relevance feedback. Some of these basic components are used in different combinations and order of application for document indexing and for query processing. Second order combinations were also tested, mainly by averaging or by selective combination of the documents retrieved by different approaches for a particular query. When evidence is found of better precision of one system at one extreme of the recall level (i.e. 1), complemented by the better precision of another system at the other recall end (i.e. 0), then both are combined to benefit from their complementary results.
In addition, during the last year our group has been improving an indexing system based on the trie data
structure, which was reported last year [
Document collections were pre-processed before indexing, using different combinations of elementary processes, each one oriented to a particular experiment. For each of these, topic queries were also processed using the same combination of processes. (Although some variants have been used, as will be described later.) The baseline approach to document and topic query processing is made up of a combination of the following steps:
Extraction: The raw text from different document collections or topic files is extracted with ad-hoc scripts that selected the contents of the desired XML elements. All those permitted for automatic runs were used. (Depending on the collection, all of the existing TEXT, TITLE, LEAD1, TX, LD, TI, or ST for document collections, and the contents of the TITLE, DESC, and NARR for topic queries.) The contents of these tags were concatenated, without further distinction to feed subsequent processing steps. This extraction treatment has a special filter for extracting topic queries in the case of the use of the narrative field: some patterns that were obtained from the topics of the past campaigns are eliminated, since they are recurrent and misleading in the retrieval process. For example, for English, we can mention patterns as “… are not relevant.”, or “…are to be excluded”. All the sentences that contain such patterns are filtered out.
Paragraphs extraction: In some experiments, we indexed paragraphs3 instead of documents. Thus, the subsequent retrieval process returned document paragraphs, so we needed to combine the relevance measures from all paragraphs retrieved for the same document. We tested several approaches for this combination, for example counting the number of paragraphs, adding relevance measures or using the maximum of the relevance figures of the paragraphs retrieved. Experimentally, we got best results using the following formula for document relevance:
1 relN =relmN +ξ ⋅ ⋅ ∑ rel jN ,
n j≠m
where n is the number of paragraphs retrieved for document N, reliN is the relevance measure obtained
for the i-th paragraph of document N, and m refers to the paragraph with maximum relevance. The
coefficient ξ was adjusted experimentally to 0.75. The idea behind this formula is to give paramount
importance to the maximum paragraph relevance, but taking into account the rest of the relevant
paragraphs to a lesser extent. Paragraph extraction has not been used for topics processing.
Tokenization: This process extracts basic text components, detecting and isolating punctuation
symbols. Some basic entities are also treated, such as numbers, initials, abbreviations, and years. For
now, we do not treat compounds, proper nouns, acronyms or other entities. The outcomes of this
process are only single words and years that appear as numbers in the text (e.g. 1995, 2004, etc.).
Filtering: All words recognized as stopwords are filtered out. Stopwords in the target languages were
initially obtained from [
resources. We have also other lists of words to exclude from the indexing and querying processes, which were obtained from the topics of past CLEF editions. We consider that such words have no semantics in the type of queries used in CLEF. As example, we can mention some of the English list: appear, relevant, document, report, etc.
Transformation: The items that resulted from tokenization were normalized by converting all uppercase letters to lowercase, and accents eliminated. This process is usually done after stemming, although it can be done before, though the resulting lexemes are different. We ought to make it before stemming in the case of the Bulgarian and Hungarian languages, since these stemmers did not work well with uppercase letters. Note that the accent removal process is not applicable for Bulgarian language.
Stemming: This process is applied to each one of the words to be indexed or used for retrieval. We
used standard stemmers from Porter [
Proper noun extraction: In some experiments, we try to detect and extract proper nouns in the text. The detection was very simple: Any chunk that outcomes from a the tokenization process is considered a proper noun provided that its firs letter is uppercase, unless such word is included in the stopwords list or in a specifically built list of words that are not suitable to be proper nouns (mainly verbs and adverbs). We opted for this simple strategy4 since we had not available huge lists of proper nouns. In the experiments that used this process, only the proper nouns extracted from the topics fed a query to an index of documents of normal words, where neither proper nouns were extracted nor stemming was done.
Final use:
o Indexing: When all the documents processed through a combination of the former steps are
ready for indexing, they are fed into our indexing trie engine to build the document collection
index.
o Retrieval: When all the documents processed by a combination of the former steps are topic
queries, they are fed to an ad-hoc front-end of the retrieval trie engine to search the previously
built document collection index. In the 2005 experiments, only OR combinations of the search
terms were used. The retrieval model used is the well-known Robertson’s Okapi BM-25 [
After retrieval, some other special processes were used to define additional experiments:
Pseudo-relevance feedback: We used this technique in some experiments. After a first retrieval step, we processed the first retrieved document to get their indexing terms that, after a standard processing5 (see below) are fed back to a second retrieval step, whose result is used.
Combination: The results from some basic experiments can be combined in different ways. The underlying hypothesis is that, to some extent, the documents with a good score in almost all experiments are more likely to be relevant than other documents that have a good score in one experiment but a bad one in others. Two strategies were followed for combining experiments: o Average: The relevance figures obtained using the probabilistic retrieval in all the experiments to be combined for a particular document in a given query is added. This approach combines the relevance figures of the experiments without highlighting a particular experiment. o Asymmetric WDX combination: In this particular type of combination, two experiments are combined in the following way: The relevance of the first D documents for each query of the first experiment is preserved for the resulting combined relevance, whereas the relevance for the remaining documents in both experiments are combined using weights W and X. We have only run experiments labeled “011”, that is, the ones that get the document more relevant from the first basic experiment and all the remaining documents retrieved from the second basic experiment, re-sorting all the results using the original relevance measure value.
5 Both retrieval processes can be independent from each other: we could have used two different treatments for the queries and documents, so using different indexes for each of the retrievals. In our case, only standard treatments were used for both retrieval steps.
For this campaign we have designed several basic experiments in which the documents for indexing and the topic queries for retrieval are processed using the same combination of the steps described in the previous section. For ease of reference we used letters that denote a set of these basic processes or variations of the extraction process for topic queries:
S: Standard or baseline treatment: tokenization, filtering, stemming, and transformation6.
N: Non-stemming treatment: tokenization, filtering, and transformation.
R: Use the narrative field in the topics.
T: Do not use narrative field.
So, the basic experiments are denoted SR, ST, NR, or NT. We designed additional experiments:
H: The standard treatment (S) is made on the index built from the paragraphs of the documents from the collection. The combination of the paragraphs retrieved is done as stated in the previous section. Depending on the fields selected in the topic queries, we can denote these experiments as HR or HT. r1: Pseudo-relevance feedback, as described above. As we only used SR runs to feed this type of run, we denoted it as r1SR. x<run1>WD<run2>X: This denotes an asymmetric DWX combination from experiments <run1> and <run2>. For example, we run experiment xNP01HR1, which gets the first document retrieved from the NP run, with its original relevance value and all the documents retrieved in the experiment HR also with their original relevance values, then resorting all these relevance figures. a<run1>….<runZ>: This denotes the average run for all the Z runs stated. For example, we run the experiment denoted aHTSTxNP01ST1, which adds all the relevance measure values obtained for each document in the experiments HT, ST, and xNP01ST1.
We tried a wide set of experiments, running several combinations of the variants described in the previous section, although not all the possible tried due to evident limitations of computing resources and time. The experiments were tried trying to test a wider and richer set of trials.
To compare these approaches, we used these techniques following the instructions given for CLEF 2004 (corpora and topic queries) and using the appropriate qrels available at the beginning of this campaign. The experiments that provided the best precision results in the CLEF 2004 scenario were selected for submission to CLEF 2005.
The results from our experiments follow. For each of the monolingual tracks, we show a table with the precision at 0 and 1 points of recall, the average precision, the percentage deviation (in average precision) from best one obtained, and the run identifier. The results are sorted in average precision ascending order, but an asterisk marks all the best precision values for each column. The submitted runs to CLEF 2005 are shown in boldface, and the figures show the precision-recall graphs for the submitted runs as well as our best7 runs, provided that these were not submitted. 6 As it was commented before, in some cases these two steps are made in reverse order. 7 Be the best run in average precision, or in precision at 0 or 1 points of recall.
Results for monolingual Bulgarian
1 0.8 0.6 0.4
At 0 At 1 Avgp % 0.4453 0.0291 0.1804 -36.01% 0.4419 0.0315 0.1886 -33.10% 0.5445 0.0393 0.2191 -22.28% 0.5174 0.0678 0.2200 -21.96% 0.5552 0.0665 0.2328 -17.42% 0.5924 0.0659 0.2676 -5.07% 0.6173* 0.0816* 0.2819* -0.00%
Run HT HR r1SR NR NT ST
SR Monolingual runs: Bulgarian
SR
ST r1SR
HT 0.2 0.4 0.6 0.8 Results for monolingual French
1 0.8 0 Results for monolingual Hungarian
1 0.8 0 Results for monolingual Portuguese
1 0.8 0.6 0.4
At 0 0.4586 0.6718 0.7035 0.7217 0.7723 0.8074 0.8232* 0.8160 0.8217
At 1 Avgp % 0.0396 0.1669 -54.87% 0.0070 0.2214 -40.13% 0.0232 0.2358 -36.24% 0.0290 0.2832 -23.42% 0.0455 0.2957 -20.04% 0.0469 0.3198 -13.52% 0.0536 0.3456 -6.54% 0.0561 0.3628 -1.89% 0.0566* 0.3698* -0.00%
Run NP xNP01r1SR1 r1SR aSRr1SR NT NR ST xNP01SR1
SR Monolingual runs: Portuguese
SR xNP01SR1
ST aSRr1SR 0.2 0.4 0.6 0.8
Except for Portuguese, the best results obtained came from runs that were not submitted, since we obtained worse results using the 2004 queries and qrels than with the submitted ones. We think that this behavior can be explained since the results depend to a great extent on the different topics selected each year. It is worth to note that we obtained the best results using the narrative field of the topic queries in all cases, as well as the standard processing approach (SR runs).
We expected to have had better results using combinations of proper nouns indexing with standard (SR or ST) runs, as it seemed to follow from the results from 2004 campaign, but it has not been the case. It is clear that the quality of the tokenization step is of paramount importance for precise document processing. We still think that a high-quality entity recognition (proper nouns or acronyms for people, companies, countries, locations, and so on) could improve the precision and recall figures of the overall retrieval, as well as a correct recognition and normalization of dates, times, numbers, etc. Pseudo-relevance feedback has not performed quite well, but we run quite few experiments to extract general conclusions. On the other hand, these runs had a lot of querying terms, what made them very slow.
Regarding the basic experiments, the general conclusions were known in advance: retrieval performance can be improved by using stemming, filtering of frequent words and appropriate weighting. 6
Future work of the MIRACLE team in these tasks will be directed to several lines of research: (a) Tuning our indexing and retrieval trie-based engine in order to get even better performance in the indexing and retrieval phases, and (b) improving the tokenization step; in our opinion, this is one of the most critical processing ones and can improve the overall results of the IR process. A good entity recognition and normalization is still missing in our processing scheme for these tasks. We need better performance of the retrieval system to drive runs that are efficient when the query has some hundred terms, as occurs when using pseudo-relevance feedback. We need also to explore further the combination schemes with these enhancements to the basic processes.
This work has been partially supported by the Spanish R+D National Plan, by means of the project RIMMEL (Multilingual and Multimedia Information Retrieval, and its Evaluation), TIN2004-07588-C03-01. Special mention to our colleagues of the MIRACLE team should be done (in alphabetical order): Ana María García-Serrano, Ana González-Ledesma, José Mª Guirao-Miras, Sara Lana-Serrano, José Luis MartínezFernández, Paloma Martínez-Fernández, Ángel Martínez-González, Antonio Moreno-Sandoval and César de Pablo-Sánchez.
Martínez, J.L.; Villena-Román, J.; Fombella, J.; García-Serrano, A.; Ruiz, A.; Martínez, P.; Goñi, J.M.; and González, J.C. (Carol Peters, Ed.): Evaluation of MIRACLE approach results for CLEF 2003.
Working Notes for the CLEF 2003 Workshop, 21-22 August, Trondheim, Norway.