This paper reports on our participation in CLEF 2005's domain-specific retrieval track. The experiments were based on previous experiences with the GIRT document corpus and were run in parallel to the multi-lingual experiments for CLEF 2005. We optimized the parameters of the system with one corpus from 2004 and applied these settings to the domain specific task. In that manner, the robustness of our approach over different document collection was assessed.
In previous CLEF campaigns, we tested an adaptive fusion system based on the MIMOR model (Mandl & Womser-Hacker 2004) within the domain specific GIRT track (Hackl et al. 2003). For CLEF 2005, the parameter optimization was based on a French document collection. The parameter settings were applied to the four language document collection of the multilingual task of CLEF 2005 (Hackl et al. 2005). In addition, we applied almost the same settings to the domain specific track in order to test the robustness of our system over different collections.
Robustness has become an issue in information retrieval research recently. It has been noted often, that the variance between queries is worse than the variance between systems. There are often very difficult queries which few systems solve well and which lead to very bad results for most systems (Harman & Buckley 2004). Thorough failure analysis can lead to substantial improvement. For example, the absence of named entities are a factor which can make queries more difficult overall (Mandl & Womser-Hacker 2004). As a consequence, a new evaluation track for robust retrieval has been established at the Text Retrieval Conference (TREC). This track does not only measure the average precision over all queries but also emphasizes the performance of the systems for difficult queries. To perform well in this track is more important for the systems to retrieve at least a few documents for difficult queries than to improve the performance in average (Voorhees 2005). In order to allow a system evaluation based on robustness more queries than for a normal ad-hoc track are necessary. The concept of robustness is extended in TREC 2005. Systems need to perform well over different tracks and tasks (Voorhees 2005).
For multilingual retrieval, robustness would also be an interesting evaluation concept because the performance between queries differs greatly (Mandl & Womser-Hacker 2004). Robustness in multilingual retrieval could be interpreted in three ways:
• Stable performance over all topics instead of high average performance (like at TREC) • Stable performance over different tasks (like at TREC) • Stable performance over different languages (focus of CLEF)
For the participation in the domain specific track in 2005, we tested the stability of our ad-hoc system for the domain specific track.
Our system was optimized with the French collection of CLEF 2004. The optimization procedure is described in detail in Hackl et al. 2005. The GIRT runs were produced with only slightly different settings.
Previous experiences with the GIRT corpus showed that blind relevance feedback does not lead to good results (Kluck 2004). Our test runs confirmed that fact and blind relevance feedback was not applied for the submitted runs. Instead, term expansion was based the thesaurus available for the GIRT data. This thesaurus was developed by the Social Science Information Centre (Kluck 2004). For the query terms, the fields Broader, Narrower and Related term were extracted from the thesaurus and added to the query for the second run. The topic title weights were set to ten, topic description weights to three and the thesaurus terms were weighted with one. This weighting scheme was adopted from the ad-hoc task.
For the second mono-lingual run UHIGIRT2, we added terms from the multilingual European terminology database Eurodicautom1 which was also used for the ad-hoc experiments. However, Eurodicautom contributed terms for very few queries. Most often, it returned "out of vocabulary".
As bilingual GIRT run, we submitted one English-to-German run. The query and the thesaurus terms were translated by ImTranslator2 . In addition, the document field "english-translation" was indexed.
For next year, we intend to implement for multi-lingual runs for the domain specific task. The thesaurus use led to a drop in performance. For the future, we intend to develop a more sophisticated strategy to apply thesaurus terms.