For MIRACLE participation on WebClef 2005, a set of independent indexes was constructed for each top level domain of the EuroGOV collection. Each of these indexes contains information extracted from the document, like URL, title, keywords, detected named entities or HTML headers. These indexes are queried to obtain partial document rankings, which are combined with various relative weights to test the value of each index. The trie based indexing and retrieval engine developed by the MIRACLE team is now fully functional and has been adapted to the WebClef environment and employed in this campaign. Other tools, such as the Named Entities Recognizer based on a finite automaton, have also been developed.
or HTML headers. The MIRACLE group has taken part in the two main tasks (Mixed Monolingual and
2
For MIRACLE participation on WebClef 2005 we decided not to follow a full text approach. Instead, a set of independent indexes was constructed for each top level domain of the collection. Partial results are obtained by applying the probabilistic ranking formula BM25 [12] to these indexes. Finally, these partial results are combined to get the final result. In different experiments, different weights are given to each set of partial results, so as to evaluate the relative importance of the different information sources that have been indexed.
‘.’ ,‘/’ or ’–‘ as delimiters.
Consequently, the total number of indexes was 85 (5 indexes/domain * 17 domains). Another index called LINKS was initially planned but finally not included due to lack of time. This index contained the words in the anchor (<A>) elements of documents that point to the indexed document. Note that, unlike the other indexes, LINKS needs two passes over the collection, so that links pointing to a document not in the collection are discarded. Although this index was not included, the tool for building it is available for future participations. 3
MIRACLE toolbox [
• •
documents must be extracted.
HTML parser: based on the El-Kabong HTML processing library [2]. The content or attributes of special tags such as headers, anchors or META tags are extracted. The body is then extracted to plain text format. • • • •
recognizer in current development. Recognition is based on the evaluation of predicates in a Finite
the target language of the query was allowed to be used, so this module tries to guess the target language from the words of the query title. For the multilingual task we have used a list of stop words, while for the mixed monolingual task a list of locations and names of the inhabitants of a country or region. A simple vote algorithm has been used. 4
The MIRACLE team has taken part in the two main tasks (Mixed Monolingual and Multilingual), submitting five runs for each one of them. A baseline run, using no metadata is mandatory. The other four runs (which will be referred as extended in this paper) use supplied metadata (the target domain). In the baseline runs, the language identification tool is employed to guess the target language from the words in the query title. For each query, only the indexes of the top level domain corresponding to the target language and the international INT domain are queried. For example, if the target language of a query is known of has been guessed to be Spanish, only ES and INT domains are considered, even though there are documents in Spanish in the other domains.
• • • • •
indexes are considered more relevant than the rest, so a weight factor with value 2 is applied for these partial lists.
In the Multilingual runs Multibase, MultiExt, MultiExtH1PN, MultieExtUrlKy and MultiExtAH1PN, partial results are mixed in the same way as in the corresponding monolingual runs. All this information is summarized in the table below. The following graphics are based on the evaluation results provided by WebCLEF organizers. The different parameters included in these results will be explained bellow as they appear in a figure.
The average success at n is defined as the portion of topics where the known-entity was found at a rank less than or equal to n. The following figures show this average success rate as a function of n for the monolingual and multilingual tasks.
Average success in monolingual runs 0,45
0,4 0,35
0,3 0,25
0,2 0,15
0,1 0,05 0 5 0,4 0,3 0,2 0,1 Average success in multilingual runs
1 5 10 20 50
find known-items. In Multilingual runs results are worse and the effect of different combinations of results are not so significant.
The Mean Reciprocal Rank (MRR) is 1 divided by the rank given to the known-entity or 0 if the relevant document has not been retrieved. The parameter called DFA is defined as the difference between the MRR score for a given topic and the average MRR score over the submitted runs of all participants. In the figures bellow, DFA values as a function of the topic are given for the best extended monolingual and multilingual runs:
The DFA value averaged over all topics is -0,04 for MonoExtH1PN and +0,02 for MultiExtAH1PN: both of them are very close to 0, the mean value over runs submitted by all participants. Our results in the multilingual task are, although worse in absolute terms than the monolingual results, better if considered relative to the other participants, even though our approach was quite simple, with no query or document translation. Elements without translation such as named entities are less noisy and especially valuable for known-item search. Although not shown in the figures above, our results were rather variable with the target language of the topic. The results in languages such as Greek or Russian were much poorer than other languages, even though the techniques used are language independent (with the partial exception of named entities recognition). This suggests we have had some sort of problem with character sets and encodings, which should be corrected for future participations. 6
Obviously, in this first year of WebCLEF track, there were no previous results available and the selection of experiments was somehow blind. Nevertheless the foundations for future campaigns have been settled and several valuable conclusions have been drawn. We have at our disposal a set of software tools that we plan to use and further improve in future campaign in order to pursue more ambitious aims.
We believe that a full text index, combined appropriately with the more specific indexes would probably improve the results. In the next campaign, we are also planning to introduce some sort of query translation mechanism. Another improvement would be to consider the hyperlink structure of the collection; a voting algorithm could be used to estimate the relative importance of web pages and this way detect home pages.
This work has been partially supported by the Spanish R+D National Plan, by means of the project RIMMEL (Multilingual and Multimedia Information Retrieval, and its Evaluation), TIN2004-07588-C03-01. Special mention to our colleagues of the MIRACLE team should be done (in alphabetical order): Ana María García-Serrano, José Carlos González-Cristóbal, Ana González Ledesma, José Miguel Goñi-Menoyo, José Mª
[2] El-Kabong HTML. A speedy HTML processing library. On line http://www.ekhtml.sourceforge.net/ [visited 28/07/2005].
and González, J.C. (Carol Peters, Ed.): Evaluation of MIRACLE approach results for CLEF 2003.
[9] de Pablo, C.; Martínez-Fernández, J. L.; Martínez, P.; Villena, J.; García-Serrano, A. M.; Goñi, J. M.; and
[11] Ragget, D.; Le Hors A. and Jacobs I. (Ed.). HTML 4.01 Specifications. W3C Recommendation 24
[12] Robertson, S.; Walker, S.; Hancock-Beaulieu, M.M. and Gatford, M. Okapi at trec 3. Text Retrieval
[13] SYSTRAN 5.0 translation resources. On line http://www.systransoft.com. [Visited 13/07/2005] [14] University of Neuchatel. page of resources for CLEF (Stopwords, transliteration, stemmers, …). On line http://www.unine.ch/info/clef/. [Visited 13/07/2005] [15] Villena, Julio; Martínez, José L.; Fombella, Jorge; G. Serrano, Ana; Ruiz, Alberto; Martínez, Paloma;
[Visited 13/07/2005]