Domain-Speci c Cross Language Retrieval: Comparing and Merging Structured and Unstructured Indices

Abstract

This year, we participated in all Monolingual, Bilingual and Multilingual tasks of the DomainSpeci c track. We used a redesigned version of our retrieval system prototype from 2006, which is based on the Lucene API [1]. A plugin to access the online translation services Google Translate [2] and PROMT [3] was implemented for the cross-language experiments. Furthermore, we tried to gure out the di erences between plain and structured indices and also applied a data fusion approach for both index schemes. In comparison to the median of all participants of the Monolingual tasks we achieved average performance for our german and english and strong performance for our russian runs. The results of the cross-language tasks were robust compared to our own monolingual experiments and better than the average of the results submitted by all participants.

Introduction and outline

2.1

English 2.2

German 2.3

Russian

Monolingual con gurations

In our experiments, we tried to improve the basic retrieval performance by combining di erent index schemes in a multi-index. Therefore, we created two di erent indices for each of the three languages of the task. We used the structure of the corresponding corpus for one index. For the creation of the other index we simply threw away the complete structure of the documents from the collection. We submitted one run using the structured index, another one using the plain index and nally a multi-index run combining both of them. That was the main setup for all Monolingual tasks. The data fusion of the multi-index con gurations was realized with the z-score operator, which had been introduced in [ 7 ].

The general con guration of our system was as follows. We used a classic language processing chain for processing the topics, i.e. a stopword lter with the stopword lists provided by [ 8 ] and a stemming algorithm depending on the language (see the following subsections) as well as a standard tokenizer. A standard pseudo-relevance feedback approach has been used to improve retrieval performance. We also used our frequency-based topic pre-processor from last year.

For this task, we merged a number of indices in each run. The main di erence between the con gurations is the structure of the data in the indices, as mentioned above. Additionally, two di erent english stemming approaches were combined with data fusion in each of the submitted runs. The rst was the Porter stemmer from the Snowball Project [ 9 ] and the second the Krovetz stemmer, which is described in [ 10 ]. For the Monolingual German task we only used the German2 stemmer from the Snowball Project [ 9 ]. We did not use any decompounding algorithm or decompounding stemmer as we had done last year, since we run short on time for the adaptation of the code to work within our new retrieval framework. Furthermore, we were not able to use the thesaurus for query expansion for the same reason. We did some additional experiments after the submission deadline. The results are also shown in the section 4. In our experiments for the Monolingual Russian task we used an analyzer and a stemmer, which are part of an outdated version of the Lucene API [ 11 ]. Again, we combined the two di erent index schemes that were mentioned before. 3

Cross-lingual con gurations

We developed a plug-in that is capable to access an online translation service to receive translations for the cross-lingual experiments. Namely, Google Translate [ 2 ] and PROMT [ 3 ] had been used, because they performed best in some preliminary runs. Additionaly, we used the bilingual thesauri that were provided for the tasks. The con guration of the cross-lingual runs is based on the combination of the monolingual runs on the corresponding target collection. 3.1

Bilingual con gurations

The con gurations we submitted for the Bilingual task are summarized in table 1. The merged monolingual run of the corresponding target collection is the basis of each of the con gurations. CUT DS BILI RU2EN MERGED CUT DS BILI DE2EN MERGED CUT DS BILI DE2EN MERGED THES CUT DS BILI RU2DE MERGED CUT DS BILI RU2DE MERGED THES CUT DS BILI EN2DE MERGED CUT DS BILI EN2DE MERGED THES CUT DS BILI EN2RU MERGED CUT DS BILI DE2RU MERGED CUT DS BILI DE2RU MERGED THES

RU-EN DE-EN DE-EN RU-DE RU-DE EN-DE EN-DE EN-RU DE-RU DE-RU

Google Translate Google Translate Google Translate + thesaurus PROMT PROMT + thesaurus Google Translate Google Translate + thesaurus Google Translate PROMT

PROMT + thesaurus

Results

4.1

Monolingual

This section summarizes the results of our experiments according to the corresponding task of the DomainSpeci c track.

The results for the Monolingual English task are shown in table 3. As we expected, the merged run performed best. An interesting observation is that the experiment based on the structured index performs very bad. Whether this is due to an unbalanced weighting scheme or to an inappropriate structure of the collection is currently under investigation. Interestingly the merged run performes only slightly better than the plain one.

Concerning the performance of the structured index, the same conclusions as for the Monolingual English task can be drawn.

We also submitted experiments for the Monolingual Russian task. The results are shown in table 5. Again, the merged run performs best and the experiment with the structured index worst. The general performance of the runs is worse compared to the other monolingual tasks. But the evaluation results of the past years share this observation, which can be seen in [ 12 ] and [ 13 ].

The results show that the russian source topics performed best for the english target collection. The robustness of our cross-lingual retrieval is approved by the small decrease of 12.36% in performance (in comparison to our best monolingual run). One can also see that using the provided bilingual thesaurus enhances the performance. 1not o cially submitted experiment

For the german target collection the english source topics achieved the best results in our experiments. Again, the use of the provided bilingual thesauri improves performance in all cases and the small gap (9.03%) between the best monolingual and bilingual experiments shows the robustness of the cross-language retrieval.

The bilingual experiments on the russian target collection perform not that good, but compared to our monolingual runs the results are acceptable. Again, the gap of 12.96% to the best monolingual run is very small. In contrast to the runs on the other target collections, the utilization of the thesaurus for translation does not improve retrieval performance here. The results of our multilingual experiments are summarized in table 9. It can be seen that the multilingual retrieval is still a hard task for scienti c data collections. We show the performance of an additional experiment, which was not o cially submitted. The performance of this experiment is best, because we changed the data fusion approach from z-score to simple sum-score merging.

Conclusions and future work

In our experiments we achieved fairly robust cross-lingual retrieval results. Nevertheless, our monolingual retrieval experiments did not meet our expectations and performed signi cantly worse than last year. This was mainly due to major changes in the system architecture: Some of the language processing algorithms were not ready this year. Additional experiments next to the o cial runs included thesaurus-based query expansion. Here, a slight increase in performance for the Monolingual tasks could be achieved. In the future we will improve the system and implement some language processing algorithms that we already used last year. Furthermore, we have to investigate our weighting scheme to use the collection structure and implement some kind of adaptive weighting, for example. Besides, we will do some work on our translation scheme to achieve better performance in the cross-lingual tasks. Especially for the Multilingual task, we will focus our research on more e cient data fusion approaches. 2not o cially submitted experiment

References

[1] The Apache Software Foundation ( 1998 - 2007 ). Lucene. Retrieved August 16 , 2007 , from Lucene Web site: http://lucene.apache.org

[2] Google ( 2007 ). Google Translate BETA . Retrieved August 16 , 2007 , from Google Web site: http://www.google.com/translate_t

[3] PROMT , Ltd. ( 2003 - 2007 ). PROMT online-translator . Retrieved August 16 , 2007 , from PROMT Web site: http://www.online-translator.com/text.asp

[4] Kursten, J. & Eibl , M. ( 2006 ). Monolingual Retrieval Experiments with a Domain-Speci c Document Corpus at the Chemnitz Technical University . In Working Notes for the CLEF 2006 Workshop , 20 - 22 September, Alicante, Spain. Retrieved August 16 , 2007 , from CLEF Web site: http://www.clef-campaign.org/2006/working_notes/workingnotes2006/kuerstenCLEF2006.pdf

[5] Overture Services , Inc. ( 2006 ). Babel Fish online-translator . Retrieved August 16 , 2007 , from Altavista Web site: http://babelfish.altavista.com/babelfish/tr

[6] Softissimo ( 2007 ). Reverso online-translator . Retrieved August 16 , 2007 , from Reverso Web site: http://www.reverso.net/text_translation.asp

[7] Savoy , J. ( 2004 ). Data Fusion for E ective European Monolingual Information Retrieval . In Working Notes for the CLEF 2004 Workshop , 15 - 17 September, Bath, UK. Retrieved August 16 , 2007 , from CLEF Web site: http://www.clef-campaign.org/2004/working_notes/WorkingNotes2004/22.pdf

[8] University of Neuch^atel, IIUN - computer science department ( 2007 ). CLEF and Multilingual information retrieval . Retrieved August 16 , 2007 , from IIUN Web site: http://members.unine.ch/jacques.savoy/clef/index.html

[9] Porter , M. ( 2001 - 2007 ). The Snowball Project . Retrieved August 16 , 2007 , from Snowball Web site: http://snowball.tartarus.org

[10] Krovetz , B. ( 1993 ). Viewing morphology as an inference process . In Proceedings of the 16th Annual International ACM SIGIR Conference , pages 191 - 202 .

[11] The Apache Software Foundation ( 1998 - 2007 ). Lucene. Retrieved August 16 , 2007 , from Lucene Web site: http://svn.apache.org/viewvc/lucene/java/branches/lucene_1_ 4 _2_dev/

[12] Di Nunzio , G. M. & Ferro , N. ( 2005 ). Appendix A - Results of the Core Tracks and Domain-Speci c Tracks . In Working Notes for the CLEF 2005 Workshop , 21 - 23 September, Vienna, Austria. Retrieved August 16 , 2007 , from CLEF Web site: http://www.clef-campaign.org/2005/working_notes/workingnotes2005/appendix_a.pdf

[13] Di Nunzio , G. M. & Ferro , N. ( 2006 ). Appendix C - Results of the Domain Speci c Track . In Working Notes for the CLEF 2006 Workshop , 20 - 22 September, Alicante, Spain. Retrieved August 16 , 2007 , from CLEF Web site: http://www.clef-campaign.org/2006/working_notes/workingnotes2006/Appendix_Domain% 20Specific.pdf