This paper presents the 2006 MIRACLE's team approach to the AdHoc Information Retrieval track. The experiments for this campaign keep on testing our IR approach. First, a baseline set of runs is obtained, including standard components: stemming, transforming, filtering, entities detection and extracting, and others. Then, a extended set of runs is obtained using several types of combinations of these baseline runs. The improvements introduced for this campaign have been a few ones: we have used an entity recognition and indexing prototype tool into our tokenizing scheme, and we have run more combining experiments for the robust multilingual case than in previous campaigns. However, no significative improvements have been achieved. For the this campaign, runs were submitted for the following languages and tracks: - Monolingual: Bulgarian, French, Hungarian, and Portuguese. - Bilingual: English to Bulgarian, French, Hungarian, and Portuguese; Spanish to French and Portuguese; and French to Portuguese. - Robust monolingual: German, English, Spanish, French, Italian, and Dutch. - Robust bilingual: English to German, Italian to Spanish, and French to Dutch. - Robust multilingual: English to robust monolingual languages.
The MIRACLE team is made up of three university research groups located in Madrid (UPM, UC3M and UAM) along with DAEDALUS, a company founded in 1998 as a spin-off of two of these groups. DAEDALUS is a leading company in linguistic technologies in Spain and is the coordinator of the MIRACLE team. This is our fourth participation in CLEF, after years 2003, 2004, and 2005. As well as bilingual, monolingual and robust multilingual tasks, the team has participated in the ImageCLEF, Q&A, and GeoCLEF tracks. The starting point was a set of basic components: stemming, transformation (transliteration, elimination of diacritics and conversion to lowercase), filtering (elimination of stop and frequent words), proper nouns detection and extracting, and paragraph extracting, among others. Some of these basic components are used in different combinations and order of application for document indexing and for query processing. Results combinations were also tested, mainly by averaging or by selective combination of the documents retrieved by different approaches for a particular query. When evidence is found of better precision of one system at one extreme of the recall level (i.e. 1), complemented by the better precision of another system at the other recall end (i.e. 0), then both are combined to benefit from their complementary results.
Our group has used its own indexing and retrieval engine, which is based on the trie data structure [
Monolingual: Bulgarian, French, Hungarian, and Portuguese.
Bilingual: English to Bulgarian, French, Hungarian, and Portuguese; Spanish to French and Portuguese; and French to Portuguese.
Robust monolingual: German, English, Spanish, French, Italian, and Dutch.
Robust bilingual: English to German, Italian to Spanish, and French to Dutch.
Robust multilingual: English to robust monolingual languages. 2
MIRACLE toolbox has already been described in previous campaigns papers [11], [12], [16]. We will say here that document collections and topics were pre-processed before feeding the indexing and retrieval engine, using different combinations of elementary processes. We will repeat here some relevant facts about these:
Extraction: The extraction treatment has a special filter for extracting topic queries in the case of the use of the narrative field: some patterns that were obtained from the topics of the past campaigns are eliminated, since they are recurrent and misleading in the retrieval process. For example, for English, we can mention patterns as “… are not relevant.”, or “…are to be excluded”. All the sentences that contain such patterns are filtered out.
Paragraphs extraction: We have not used paragraph indexing this year, since the results we have obtained in this campaign and past ones have been disappointing.
Tokenization: This process extracts basic text components, detecting and isolating punctuation symbols. Some basic entities are also treated, such as numbers, initials, abbreviations, years, and some proper nouns (see next item). The outcomes of this process are only single words, years that appear as numbers in the text (e.g. 1995, 2004, etc.), or entities.
Entities: We consider that entities detection and normalization plays a central role in Information Retrieval, but it is a difficult task. For this year we have integrated a special module in the tokenization process that detects and marks some entities that have been previously collected from several sources into a lexical database for entities. These entities, which can be people names, place names, initials, abbreviations, etc., can consist of one or more words and special symbols, and their correct treatment is integrated into the tokenizer. For now, no entity normalization is done, so the same entity can appear in different forms and these are treated as different entities.
Filtering: Stopwords lists in the target languages were initially obtained from [
Transformation: The items that resulted from tokenization were normalized by converting all
uppercase letters to lowercase, and accents eliminated. This has not been done for Bulgarian.
Stemming: We used standard stemmers from Porter [28] for most languages, except for Hungarian and
Bulgarian, where we used stemmers from Neuchatel [
Indexing: When all the documents processed through a combination of the former steps are ready for indexing, they are fed into our indexing trie engine to build the document collection index.
Retrieval: When all the documents processed by a combination of the former steps are topic queries, they are fed to an ad-hoc front-end of the retrieval trie engine to search the previously built document collection index. In the 2006 experiments, only OR combinations of the search terms were used. The retrieval model used is the well-known Robertson’s Okapi BM-25 [32] formula for the probabilistic retrieval model, without relevance feedback.
Combination: After retrieval, some other special combination processes were used to define additional experiments: The results from some basic experiments can be combined in different ways. The underlying hypothesis is that, to some extent, the documents with a good score in almost all experiments are more likely to be relevant than other documents that have a good score in one experiment but a bad one in others. Two strategies were followed for combining experiments: ► ►
Average: Relevance figures obtained in all the experiments to be combined for a particular document in a given query are added. This approach combines the relevance figures of the experiments without highlighting a particular experiment.
Asymmetric WDX combination: In this particular type of combination, two experiments are combined in the following way: The relevance of the first D documents for each query of the first experiment is preserved for the resulting combined relevance, whereas the relevance for the remaining documents in both experiments are combined using weights W and X. For example, for experiments labeled “011”, the most relevant document from the first basic experiment is considered, and then all the remaining documents retrieved from the second basic experiment. Then all the obtained results are re-sorted using the obtained relevance measure values. 3
The experiments name reflects the processes made on the documents collections and the topic sets. The naming scheme we have used this year is, for basic experiments, as follows:
<doclang><docproc><topiclang><topicproc>
<doclang> and <topiclang> are the standard two letter abbreviations for the documents or topic
languages1 (i.e. bg, de, en, es, fr, hu, it, nl, and pt), or ml for documents language in multilingual robust runs.
Except for multilingual runs, should be identical.
<docproc> reflects the processes made on the documents collection for the experiment. The first letter is
always2 F (for indexing the full texts). The second letter is S or W. Letter S is used for the standard or baseline
treatment: tokenization, filtering, stemming, and transformation; whereas W is used for a non-stemming
treatment: tokenization, filtering, and transformation.
<topicproc> reflects the processes made on the topics collection for the experiment. For monolingual runs, it
also consists of two characters, being the second S or W, having the same meaning that in documents texts
processing. The first character is one of the digits 2, 3, 4, 5 or 6; reflecting how many times the title (T),
description (D) or narrative (N) of the topic has been taken into account, according to the following scheme: 2
(TD), 3 (TDN), 4 (TTTDN), 5 (TTTTDN), and 6 (TTTTDDN). For bilingual runs, some information is added
before these two characters: the translation engine used and the standard two-letter code of the source topic
language. For robust runs, the letter R is present in the first position of this field. The translation engine cndes
used have been the following: L (Wordlingo [
For combining experiments, we depart from this scheme. In the <docproc> position we indicate the type of
combination: xWDX, for asymmetric WDX combination (see the meaning of the digits D, W, and X in the
previous section); or y, for average combination of runs. In the <topicproc> position the runs that are
combined are indicated in an ad-hoc, rather weird encoding. For example, run “nlx021nlRLfrFW4FS4” refers to
a combined (robust) experiment made on the Dutch collection, with W=0, D=2, X=1, using French topics
1 These refer to the topic language of monolingual runs or the target language in a translated topic for cross-lingual runs.
2 The letter H was reserved for paragraphs indexing, process which was not made this year, as mentioned.
1
0.8
0.6
0.4
0.2
0 0
lang
bg
en
fr
hu
pt
translated into Dutch using the WordLingo [
For (robust) multilingual runs, a special naming convention is used. It will be described in a later section. 4
The following figures and tables resume the performance of our best experiments in the monolingual and bilingual tasks. The details of all the experiments run and their performance figures and some graphic representations can be found in the appendix.
Best monolingual runs
Best bilingual runs bgx101bgFS4FS5 enx101enFS3FS4
frFSfr6S hux101huFS3FS4 ptx101ptFS3FS6 bgFSbgWen3S frFSfrVen3S huFShuMen4S ptFSptSfr3S 1 0.8 0.6 0.4 0.2
The following figures and tables resume the performance of our best experiments in the monolingual and bilingual robust tasks. The details of all the experiments run and their performance figures and some graphic representations can be also found in the appendix. We have not used a different system or different types of runs for the robust case, so we just present the results obtained. Please, note that in these tables, geometric mean average precision figures are given instead average precision figures. 3 We have tested only some WDX sets: 011, 021, 091, 101, and 153. Regarding combined experiments, we tested these combinations: FS3FS4, FS3FS6, FS4FS5, FW3FS3, FW4FS4, FW4FS5, and FW4FS6. 0.6 0.4 0.2 0 0
Best robust monolingual runs
Best robust bilingual runs dex0e1n1ydeenRRFFWS33F4S563 esFSesR3S frFSfrR3S itFSitR6S nlFSnlR4S esx011esRLitFW3FS3 deFSdeRSen3S nlFSnlRLfr6S 0.8 0.6 0.4 0.2 In the monolingual case, results for Spanish are much better than those obtained for the rest of the languages. In all cases the use of baseline runs has obtained results better than the use of combined ones. Curiously, target language Dutch runs have results better than the runs in other languages. Note that in all cases, the experiments having into account the topic narrative have best results, as happened in the non-robust case. We used the traditional approach to multilingual information retrieval that translates topic queries to each of the documents collections. The probabilistic BM25 [32] approach used for monolingual retrieval gives relevance measures that depends heavily on parameters that are too dependent on the monolingual collection, so it is not very good for this type of multilingual merging, since relevance measures are not comparable among collections. In spite of this, we made merging experiments using the relevance figures obtained from each monolingual retrieval process, considering three cases:4
Using original relevance measures for each document as obtained from the monolingual retrieval process. The results are composed of the documents with greater relevance measures.
Normalizing relevance measures with respect to the maximum relevance measure obtained for each topic query i (normal normalization): reli norm = reli
.
reli max The results are composed of the documents with greater normalized relevance measures.
Normalizing relevance measures with respect to the maximum and minimum relevance measure obtained for each topic query i (alternate normalization): reli alt =
reli − reli min reli max − reli min
The results are composed of the documents with greater alternate normalized relevance measures. We denote if normalization is done in the run identifier using the last character: n means normal normalization whereas l denotes alternate normalization. When neither l nor n is present, no normalization has been made for that run. For this “standard multilingual approach”, the run naming convention follows this pattern: mlRSFS(de|en|es|fr|it|nl)([23456])S([ln]?) 4 Round-robin merging for results of each monolingual collection has not been used. where usual regular expression patterns are used, but inclosed in “()”. The meanings of the letters used should be evident from the described naming conventions for monolingual runs that are combined. Note that S is used both for “stemmed” and “Systran”.
In addition to all this, we tried a different approach to merging: Considering that the more relevant documents for each of the topics are usually the first ones in the results list, we will select from each monolingual results file a variable number of documents, proportional of the average relevance number of the first N documents. Thus, if we need 1,000 documents for a given topic query, we will get more documents from languages where the average relevance of the first N relevant documents is greater. We did all this in two cases:
Using not normalized runs (we call it case X) to calculate the appropriate number of documents to aggregate. After having obtained such, the documents sets obtained are optionally normalized before merging (we tried not normalizing, and normalizing with both formulae).
Using normalized runs (we call it case Y) to calculate the appropriate number of documents to agregate.
After obtaining such documents sets, merging is done. We also used both types of normalization. The several cases tested are encoded in the run identifier. The first two characters are “ml”, followed by two characters that indicate one of the two cases described above, and the parameters used: following the rest of the run identifier. The full run identifier follows one of the patterns: ml([123456]X)RSFSen([23456])S([nl]?) ml([123456]Y)RSFSen([23456])S([nl]) where usual regular expression patterns are used, but inclosed in “()”. The same comments apply that for the previous naming scheme for multilingual runs.
The following figure resume the performance of our best experiments in the multilingual robust task. The details of all the experiments run and their performance figures can be found in the appendix. We have not used a different system or different types of runs for the robust case, so we just present the results obtained. Please, note that in these tables, geometric mean average precision figures are given instead average precision figures. 1 0.8 0 0
Robust multilingual runs from English
This year we have not changed a lot our previous processing scheme, although some improvements have been incorporated regarding proper nouns and entities detection and indexing. For this reason we think that the obtained results are quite similar to previous ones. We need to work harder in some stages of processing, especially these ones that can improve performance substantially.
It is clear that the quality of the tokenization step is of paramount importance for precise document processing. We still think that a high-quality entity recognition (proper nouns or acronyms for people, companies, countries, locations, and so on) could improve the precision and recall figures of the overall retrieval, as well as a correct recognition and normalization of dates, times, numbers, etc. Although we have introduced some improvements in our processing scheme, a good multilingual entity recognition and normalization tool is still missing. This step is the one in which we are currently devoting more work.
We are also mproving the architecture of our indexing and retrieval trie-based engine in order to get even better performance in the indexing and retrieval phases, tuning some data structures and algorithms.
This work has been partially supported by the Spanish R+D National Plan, by means of the project RIMMEL (Multilingual and Multimedia Information Retrieval, and its Evaluation), TIN2004-07588-C03-01; and by the Madrid’s R+D Regional Plan, by means of the project MAVIR (Enhancing the Access and the Visibility of Networked Multilingual Information for Madrid Community), S-0505/TIC/000267.
Special mention to our colleagues of the MIRACLE team should be done (in alphabetical order): Ana María García-Serrano, José Carlos González-Cristóbal, Ana González-Ledesma, José Miguel Goñi-Menoyo, José Mª Guirao-Miras, Sara Lana-Serrano, José Luis Martínez-Fernández, Paloma Martínez-Fernández, Antonio Moreno-Sandoval and César de Pablo-Sánchez.
The results from our experiments follow. For each of the monolingual or bilingual tasks, we show a table with the precision at 0 and 1 points of recall, the average precision, the percentage deviation (in average precision) from best one obtained, the run identifier, and the precedence of the run, when the run was submitted. The results are sorted in average precision ascending order, but an asterisk marks all the best precision values for each column (in average precision, or in precision at 0 or 1 points of recall).
In the case of the robust tasks, in addition to the columns indicated above, the tables include a column with the geometric mean average precision, and the rows are sorted using this figure in ascending order. The percentage deviation in this case refers to the average precision, in order to facilitate the comparison with the ordering using this number.
In all cases a figure that compares the submitted runs and the best one, when it was the case that it was not submitted, is included for each task and language pair. The best run here refers that one with best average precision, not geometric mean average precision.
1
run x bgFWbg2W bgFWbg4W bgFWbg3W bgx153bgFW3FS3 bgx153bgFW4FS5 bgx153bgFW4FS6 bgx153bgFW4FS4 bgx091bgFW4FS6 bgx091bgFW4FS5 bgx091bgFW4FS4 bgFSbg2S 3 bgx091bgFW3FS3 bgybgFS3FW3 bgx021bgFW4FS6 bgx011bgFW4FS6 bgx021bgFW4FS5 bgx011bgFW4FS5 bgx021bgFW3FS3 bgx021bgFW4FS4 bgx011bgFW4FS4 1 bgx011bgFW3FS3 bgFSbg6S bgFSbg5S bgx101bgFS3FS6 bgFSbg3S 2 bgybgFS3456 bgx101bgFS3FS4 bgFSbg4S bgx101bgFS4FS5 1 0.8 0.6 0.4 0.2 0 0
Monolingual runs: English
Bilingual runs: English to Bulgarian
run frFWfrAes2W frFWfrSen2W frFWfrSes2W frFWfrSpt2W frFWfrAes4W frFWfrSes4W frFWfrVen2W frFWfrAes3W frFWfrSpt3W frFWfrSpt4W frFWfrSen4W frFWfrSes3W frFWfrVde2W frFSfrAes2S frFWfrSen3W frFSfrSes2S frFWfrVen4W frFSfrSpt2S frFSfrAes6S frFSfrSen2S frFWfrVde4W frFSfrSpt5S frFSfrAes5S frFSfrVde2S frFWfrVde3W frFSfrSes6S frFSfrSpt6S frFSfrVen2S frFSfrSen5S frFSfrSpt4S frFSfrAes4S frFWfrVen3W frFSfrAes3S frFSfrSes5S frFSfrSpt3S frFSfrSes4S frFSfrXes3S frFSfrSes3S frFSfrSen4S frFSfrSen6S frFSfrVde5S frFSfrVde6S frFSfrVen5S frFSfrSen3S frFSfrVen4S frFSfrVen6S frFSfrVde4S frFSfrVde3S frFSfrVen3S ml5XRSFSen2Sl ml5XRSFSen2S ml6XRSFSen2Sl mlRSFSen5Sn ml6XRSFSen2S ml4XRSFSen2S ml3XRSFSen2S mlRSFSen3S ml6YRSFSen5Sl mlRSFSen6Sn ml5YRSFSen5Sl mlRSFSen3Sl mlRSFSen4Sn ml4YRSFSen5Sl ml3YRSFSen5Sl ml6YRSFSen5Sn ml1XRSFSen5Sn ml5YRSFSen5Sn ml4XRSFSen5Sn ml3XRSFSen5Sn ml3YRSFSen5Sn ml6YRSFSen6Sl ml2YRSFSen5Sn ml2YRSFSen5Sl ml1YRSFSen5Sn ml4YRSFSen5Sn ml2XRSFSen5Sn ml6XRSFSen5Sn ml5XRSFSen5Sn ml1YRSFSen5Sl ml6YRSFSen4Sl ml5YRSFSen6Sl ml4YRSFSen6Sl ml5YRSFSen4Sl ml3YRSFSen6Sn ml2YRSFSen6Sn ml6YRSFSen3Sl ml6YRSFSen6Sn ml5YRSFSen6Sn ml3YRSFSen6Sl ml3YRSFSen4Sl ml4YRSFSen4Sl ml1XRSFSen6Sn ml1YRSFSen6Sn ml2YRSFSen6Sl ml5XRSFSen6Sn ml2XRSFSen6Sn ml3XRSFSen6Sn ml4XRSFSen6Sn ml6XRSFSen6Sn ml6XRSFSen5S ml1YRSFSen6Sl ml6YRSFSen4Sn ml2YRSFSen4Sl ml1XRSFSen5S ml4XRSFSen4Sl ml5YRSFSen4Sn ml3XRSFSen5S ml2YRSFSen4Sn ml3YRSFSen4Sn ml4YRSFSen4Sn ml2XRSFSen5S ml1YRSFSen4Sn ml1XRSFSen4Sn ml5XRSFSen5S ml6XRSFSen4Sn ml1YRSFSen4Sl ml2XRSFSen4Sn ml1XRSFSen5Sl ml3XRSFSen4Sn ml5YRSFSen3Sn ml4XRSFSen3Sn ml4XRSFSen4Sn ml5XRSFSen4Sn ml4XRSFSen5Sl ml4YRSFSen6Sn ml5XRSFSen5Sl ml6XRSFSen5Sl mlRSFSen3Sn ml1XRSFSen4S ml3XRSFSen4S ml5XRSFSen4S ml2XRSFSen4S ml1XRSFSen6S ml4XRSFSen4S ml2XRSFSen6S ml6XRSFSen4S ml3XRSFSen6S ml4XRSFSen6S ml5XRSFSen6S ml5YRSFSen3Sl ml6XRSFSen6S ml4YRSFSen3Sl ml1XRSFSen6Sl ml2XRSFSen6Sl ml3XRSFSen6Sl ml3YRSFSen3Sl ml4XRSFSen6Sl ml3XRSFSen4Sl ml2XRSFSen4Sl ml1XRSFSen4Sl ml5XRSFSen6Sl ml2YRSFSen3Sl ml6XRSFSen6Sl ml4XRSFSen5S ml1YRSFSen3Sl ml6XRSFSen4Sl ml5XRSFSen4Sl ml1YRSFSen3Sn ml6XRSFSen3Sn ml3YRSFSen3Sn ml4YRSFSen3Sn ml2YRSFSen3Sn ml5XRSFSen3Sn 0.7463 0.7465 0.7626 0.7627 0.7465 0.7466 0.7903 0.7904* 0.7903 0.7901 0.7902 0.7903 0.7504 0.7505 0.7506 0.7503 0.7497 0.7498 ml6YRSFSen3Sn ml3XRSFSen3Sn ml3XRSFSen5Sl ml2XRSFSen5Sl ml2XRSFSen3Sn ml1XRSFSen3Sn ml1XRSFSen3S ml2XRSFSen3S ml3XRSFSen3S ml6XRSFSen3S ml5XRSFSen3S ml4XRSFSen3S ml3XRSFSen3Sl ml2XRSFSen3Sl ml1XRSFSen3Sl ml4XRSFSen3Sl ml6XRSFSen3Sl ml5XRSFSen3Sl [2] Automatic Trans SL, Spain. Automatic translation server. On line http://www.automatictrans.es [Visited 18/07/2006]. [3] BabelFish translation resources. On line http://babelfish.altavista.com [Visited 18/07/2006]. [4] Babylon.com, Ltd, Israel. On line http://www.babylon.com [Visited 18/07/2006]. [5] de Pablo, C.; González-Ledesma, A.; Martínez-Fernández, J. L.; Guirao, J.M.; Martínez, P.; and Moreno, A. MIRACLE’s Cross-Lingual Question Answering Experiments with Spanish as a Target Language. Accessing Multilingual Information Repositories: 6th Workshop of the Cross Language Evaluation Forum 2005, CLEF 2005, Vienna, Austria, Revised Selected Papers (Peters, C. et al., Eds.). Lecture Notes in Computer Science, vol. 4022, Springer (to appear). [6] de Pablo, C.; González-Ledesma, A.; Martínez-Fernández, J. L.; Guirao, J.M.; Martínez, P.; and Moreno, A. MIRACLE’s 2005 Approach to Cross-Lingual Question Answering. Working Notes for the CLEF 2005 Workshop. Vienna, Austria, 2005. [7] de Pablo, C.; Martínez-Fernández, J. L.; Martínez, P.; and Villena, J. miraQA: Experiments with Learning Answer Context Patterns from the Web. Multilingual Information Access for Text, Speech and Images: 5th Workshop of the Cross-Language Evaluation Forum, CLEF 2004, Bath, UK, September 15-17, 2004, Revised Selected Papers (Carol Peters, Paul Clough, Julio Gonzalo, et al., Eds.). Lecture Notes in Computer Science, vol. 3491, pp. 494-501. Springer, 2005. [8] de Pablo, C.; Martínez-Fernández, J. L.; Martínez, P.; Villena, J.; García-Serrano, A. M.; Goñi, J. M.; and González, J. C. miraQA: Initial experiments in Question Answering. Working Notes for the CLEF 2004 Workshop (Carol Peters and Francesca Borri, Eds.), pp. 371-376. Bath, United Kingdom, 2004. [9] Ergane multilingual translation dictionary. On line http://download.travlang.com [Visited 18/07/2006]. [10] Free2Translation. Free text translator. On line http://www.freetranslation.com [Visited 18/07/2006]. [11] Goñi-Menoyo, J.M.; González-Cristóbal, J.C.; and Villena-Román, J. MIRACLE at Ad-Hoc CLEF 2005: Merging and Combining without Using a Single Approach. Accessing Multilingual Information Repositories: 6th Workshop of the Cross Language Evaluation Forum 2005, CLEF 2005, Vienna, Austria, Revised Selected Papers (Peters, C. et al., Eds.). Lecture Notes in Computer Science, vol. 4022, Springer (to appear). [12] Goñi-Menoyo, J.M.; González, J.C.; and Villena-Román, J. Miracle’s 2005 Approach to Monolingual
Information Retrieval. Working Notes for the CLEF 2005 Workshop. Vienna, Austria, 2005. [13] Goñi-Menoyo, José M; González, José C.; Martínez-Fernández, José L.; and Villena, J. MIRACLE’s Hybrid Approach to Bilingual and Monolingual Information Retrieval. Multilingual Information Access for Text, Speech and Images: 5th Workshop of the Cross-Language Evaluation Forum, CLEF 2004, Bath, UK, September 15-17, 2004, Revised Selected Papers (Carol Peters, Paul Clough, Julio Gonzalo, et al., Eds.). Lecture Notes in Computer Science, vol. 3491, pp. 188-199. Springer, 2005. [14] Goñi-Menoyo, José M.; González, José C.; Martínez-Fernández, José L.; Villena-Román, Julio; GarcíaSerrano, Ana; Martínez-Fernández, Paloma; de Pablo-Sánchez, César; and Alonso-Sánchez, Javier. MIRACLE’s hybrid approach to bilingual and monolingual Information Retrieval. Working Notes for the CLEF 2004 Workshop (Carol Peters and Francesca Borri, Eds.), pp. 141-150. Bath, United Kingdom, 2004. [15] Goñi-Menoyo, José Miguel; González-Cristóbal, José Carlos and Fombella-Mourelle, Jorge. An optimised trie index for natural language processing lexicons. MIRACLE Technical Report. Universidad Politécnica de Madrid, 2004. [16] González, J.C.; Goñi-Menoyo, J.M.; and Villena-Román, J. Miracle’s 2005 Approach to Cross-lingual
Information Retrieval. Working Notes for the CLEF 2005 Workshop. Vienna, Austria, 2005. [17] Google language tools. On line http://www.google.com/language_tools [Visited 18/07/2006]. [18] Lana-Serrano, S.; Goñi-Menoyo, J.M.; and González-Cristóbal, J.C. MIRACLE at GeoCLEF 2005: First Experiments in Geographical IR. Accessing Multilingual Information Repositories: 6th Workshop of the Cross Language Evaluation Forum 2005, CLEF 2005, Vienna, Austria, Revised Selected Papers (Peters, C. et al., Eds.). Lecture Notes in Computer Science, vol. 4022, Springer (to appear). [19] Lana-Serrano, S.; Goñi-Menoyo, J.M.; and González-Cristóbal, J.C. MIRACLE’s 2005 Approach to Geographical Information Retrieval. Working Notes for the CLEF 2005 Workshop. Vienna, Austria, 2005. [20] [21] [22] [23] [24] [25] [26] [27]
Martínez-Fernández, J.L.; Villena-Román, J.; García-Serrano, A.M.; and González-Cristóbal, J.C. Combining Textual and Visual Features for Image Retrieval. Accessing Multilingual Information Repositories: 6th Workshop of the Cross Language Evaluation Forum 2005, CLEF 2005, Vienna, Austria, Revised Selected Papers (Peters, C. et al., Eds.). Lecture Notes in Computer Science, vol. 4022, Springer (to appear).
Martínez-Fernández, José L.; García-Serrano, Ana; Villena, J. and Méndez-Sáez, V.; MIRACLE approach to ImageCLEF 2004: merging textual and content-based Image Retrieval. Multilingual Information Access for Text, Speech and Images: 5th Workshop of the Cross-Language Evaluation Forum, CLEF 2004, Bath, UK, September 15-17, 2004, Revised Selected Papers (Carol Peters, Paul Clough, Julio Gonzalo, et al., Eds.). Lecture Notes in Computer Science, vol. 3491, pp. 699-708. Springer, 2005. Martínez-Fernández, J. L.; García-Serrano, A.; Villena, J.; Méndez-Sáez, V.D.; González-Tortosa, S.; Castagnone, M.; and Alonso, J. MIRACLE at ImageCLEF 2004. Working Notes for the CLEF 2004 Workshop (Carol Peters and Francesca Borri, Eds.), pp. 545-553. Bath, United Kingdom, 2004.
Martínez, J.L.; Villena-Román, J.; Fombella, J.; García-Serrano, A.; Ruiz, A.; Martínez, P.; Goñi, J.M.; and González, J.C. Evaluation of MIRACLE approach results for CLEF 2003. Working Notes for the CLEF 2003 Workshop (Carol Peters, Ed.), pp. 115-124. Trondheim, Norway, 21-22 August 2003. Martínez-González, A.; Martínez-Fernández, J. L.; de Pablo-Sánchez, C.; Villena-Román, J. JiménezCuadrado, L.; Martínez, P.; and González-Cristóbal, J.C. MIRACLE at WebCLEF 2005: Combining Web Specific and Linguistic Information. Accessing Multilingual Information Repositories: 6th Workshop of the Cross Language Evaluation Forum 2005, CLEF 2005, Vienna, Austria, Revised Selected Papers (Peters, C. et al., Eds.). Lecture Notes in Computer Science, vol. 4022, Springer (to appear). Martínez-González, A.; Martínez-Fernández, J. L.; de Pablo-Sánchez, C.; Villena-Román, J. JiménezCuadrado, L.; Martínez, P.; and González-Cristóbal, J.C. MIRACLE’s Approach to Multilingual Web Retrieval. Working Notes for the CLEF 2005 Workshop. Vienna, Austria, 2005.
Morphological, Hungary. MoBiCAT translation resources. On line http://www.morphologic.hu [Visited 18/07/2006]. [28] Porter, Martin. Snowball stemmers and resources page. On line http://www.snowball.tartarus.org [Visited 18/07/2006]. [29] Pro Langs Ltd., Bulgary. BULTRA translation resources. On line http://www.bultra.com [Visited 18/07/2006]. [30] Prompt-Online free automatic translation service. On line http://translation2.paralink.com [Visited 18/07/2006]. [31] Reverso translation resources. On line http://www.reverso.net/text_translation.asp [Visited 18/07/2006]. [32] Robertson, S.E. et al. Okapi at TREC-3. In Overview of the Third Text REtrieval Conference (TREC-3).
D.K. Harman (Ed.). Gaithersburg, MD: NIST, April 1995. [33] Savoy, Jacques. Report on CLEF-2003 Multilingual Tracks. Comparative Evaluation of Multilingual Information Access Systems (Peters, C; Gonzalo, J.; Brascher, M.; and Kluck, M., Eds.). Lecture Notes in Computer Science, vol. 3237, pp. 64-73. Springer, 2004.