After our first participation in the Bilingual task of WebCLEF 2005, we have emigrated to a more challenging task. In this report we are presenting the results obtained after evaluating a set of topics in the Mixed-Monolingual task of WebCLEF 2006. Our efforts were focused on the preprocessing of the EuroGOV corpus which is itself a very challenging task, due to the high variety of errors that must be treated in order to correctly interpret the content of each document to index. Moreover, we have tested a new formula for the ranking of the documents retrieved, which is based on the Jaccard formula but includes a penalization factor. Results are low but encourage to investigate whether they are the result of a bad preprocessing process and/or the malfunction of the search engine components.
Learning to deal with the high volume of data of Internet becomes more a need than a curiosity. Currently, we are witnesses of a big explosion of information available that must be adequately cataloged. Moreover, this information comes from all parts of the world, from very different cultures and different languages which makes this task even more difficult. At the moment, it would seem that only search engines such as Google and Yahoo could have enough resources for this challenge, but new proposals would provide advances from the scientific instead of the comercial viewpoint. Certainly, forums dedicated to the analysis of information search and retrieval, more particularly in a cross-language environment, are needed.
The WebCLEF concern is about the evaluation of information retrieval systems using
crosslingual web pages. The justification of the WebCLEF track is based on the fact that many issues
for which people turn to the web are essensially multilingual. In 2005 [
In 2005, we participated in the bilingual English to Spanish task [
The preprocessing phase of the EuroGOV corpus presents a big challenge, due to the written variants a government web page could have. We have found that a big amount of documents do not present a strict html syntax. We have written two scripts for obtaining the terms to be indexed from each document. The first script uses regular expressions for excluding all the information which is enclosed by the characters < and >. Although this script obtains very good results, it is very slow and therefore we decided to used it only with three domains of the EuroGOV collection, namely Spanish (ES), French (FR), and German (DE).
On the other hand, we wrote a script based in the html syntax for obtaining all the terms considered interesting for indexing, i.e., those different than script codes (javascript, vbscript, style cascade sheet, etc), html codes, etc. This script speeded up our indexing process but it did not took into account that some web pages are incorrectly written and, therefore, we missed important information from those documents.
Another preprocessing problem consists in the charset codification, which leads to a even more difficult analysis. Although the EuroGOV corpus is given in UTF-8, the documents that made up this corpus does not neccesarily keep this charset. We have seen that for some domains, the charset codification is given in the html metadata tag, but also we found that this codification could be wrong, perhaps because it was filled without the supervision of the creator of that page, who may be does not know anything, and evenmore does not matter about charsets codifications. We consider it as the most difficult problem in the preprocessing process.
As usual in the information retrieval systems, we eliminated stopwords for each language
(except Greek). A good repository of resources for this step is suministered by Jacques Savoy from
the Institut interfacultaire d’informatique (see [
Nowadays, different information retrieval models are reported in literature [
Score(Qi, Dj) = |Dj| ∩ |Qi| − 1 − |Dj| ∩ |Qi|
|Dj| ∪ |Qi| |Qi|
As can be seen, the first component of this formula is the typical Jaccard approximation. The evaluation of this formula is quite fast, and allows its implementation in real situations. The results obtained by using this approach are presented in the next section. 4
At the moment of writing this paper, individual results are only known by each team in the competition, and therefore a comparative table with the other teams results is not presented. In Table 1 we show the results obtained with our approximation. We evaluated different runs, varying the use of diacritics and the preprocessing of the corpus.
The WithoutDiac run eliminates all diacritics in both, the corpus and the topics, whereas the WithDiac run only supresses the diacritics in the corpus. We can observe a expected reduction of Mean Reciprocal Rank (MRR), but it is not significatively high with respect to the first run. This is clearly derived from the amount of diacritics introduced in the topics of evaluation, which is not very high. An analysis of the queries in real situations may be interesting in order to determine whether the topics set is realistic. The last run (CDWithoutDiac) eliminates diacritization in both, the topics and corpus, but also tries a charset detection for each document to be indexed. Unfortunately, from the table we can observe that we did not success in our attempt.
Run WithoutDiac WithDiac CDWithoutDiac We have proposed a new approach for the ranking formula in a information retrieval system based on the Jaccard formula, but with a penalization factor. After evaluating this approach in the approximately 75% of queries from the WebCLEF competition, we obtained low results. We observed that our major problem was related to the preprocessing phase. The charset decodification must be improved in order to correctly interpret the data.
An evaluation of the use of diacritization in the task has shown that results are not significatively different, which may be suggesting that the set of queries provided for the evaluation does not have a high number of diacritics. More investigation would determine whether this behaviour is realistic or must be tuned in further evaluations.
Even if a comparison with other results of the competition is still pending in order to determine how low are the results we obtained, we assume that our results are not very good from observing those from the last competition.