For our first participation in CLEF we chose the domain specific GIRT corpus. We implemented the adaptive fusion model MIMOR (Multiple Indexing and Method-Object Relations) which is based on relevance feedback. The linear combination of several retrieval engines was optimized. As a basic retrieval engine, IRF from NIST was employed. The results are promising. For several topics, our runs achieved a performance above the average. The optimization based on last years' topics and relevance judgements proved to be a fruitful strategy. For the CLEF 2002 campaign we tested an adaptive fusion system based on the MIMOR model [WomserHacker 1997; Mandl & Womser-Hacker 2001b]. The experiments are done fully automatic and deal with the domain specific corpus of social science documents. We choose GIRT for our first participation data because we are especially interested in the challenges of the domain specific task and because it allows mono-lingual retrieval. As basic retrieval engine for our fusion system, we used the IRF package from NIST.
MIMOR (Multiple Indexing and Method-Object Relations) represents a learning approach to the fusion task
which is based on results of information retrieval research which show that the overlap between different
systems is often small [
The GIRT data is part of a digital library for the social sciences. The domain specific task is described in [
Our main tool was the IRF retrieval package available from NIST1. IRF is an object oriented retrieval framework which is programmed in JAVA. The sample IR application proved to meet our retrieval needs sufficiently, which means ranked results, setting of parameters etc. Thus, only few changes were necessary.
In addition we used the Lucene package for topic and data preprocessing. It is written in JAVA, too, and it can be obtained from the Jakarta Apache Project’s homepage2. Lucene comprises both a built-in German analyzer and a German stemmer apart from plenty other features. Both may easily be altered as desired. The indexing application could be run without major changes, so we slightly modified the filter settings and provided a different, more thorough stop word list3.
E N E IRF
M I M O R M E R G E R Diagramm 1. Sequence of operations 4.2
The current MIMOR implementation of the University of Hildesheim could not be employed for all processing steps. Therefore the necessary in-between actions were carried out by AWK4 scripts. We had some formatting 1 http://www.itl.nist.gov/iaui/894.02/projects/irf/irf.html 2 http://jakarta.apache.org/lucene/docs/index.html 3 http://www.unine.ch/info/clef/ 4 http://members.cox.net/dos/txtfrmt.htm done and cut off all information except for but document number, title and text. The most important function however was to simulate MIMOR’s merging algorithm (cf. diagramm 1). Several scripts were necessary to statically simulate this functionality.
Three different indexing parameter settings of the IRF – inverse document frequency (weight title:1, weight text:1), inverse document frequency (weight title:3, weight text:1), keyword (weight title:1, weight text:1) represented the individual retrieval systems.
We also ran the topics from last year with the same setup. Subsequently, we compared the results obtained with the relevant documents assessed for last years campaign. Based on these results we globally optimized the fusion by assigning a higher weight to the system with better results.
For comparison we submitted two runs: • •
Run with equal weights for both individual systems
Run with optimized weights derived from last years results 4.3
As table 1 shows, the average precision for both runs is almost equal. Speaking in absolute figures, run UHi02r2 (where optimized fusion was applied) produced ten more relevant documents.
Run UHi02r1 UHi02r2 Query 67 – where out of five possible relevant documents none was retrieved – is considered an outlier and is left out for the following operations.
Taking the difference between the two precision scores as a method to compare, the fused result was better on nine, equal on two and worse on twelve occasions. However the sum of differences 75 75 ∑ prec(r1) - ∑ prec(r2) i=51
i=51 produces a slight advantage in favor of the optimized fusion: -0.0089 which is less than one percent and results in a fourth decimal point mean deviation of 0.000387 for each query.
Twenty out of 23 topics (=86.96%) could be found within a 0.0060 range and still nine of 23 (=39.13%) were within a 0.0010 range. However, these deviations are not necessarily statistically significant. The precision at five documents is at 0.2667 respectively 0.2750, which we consider not too bad for our first participation. Some of our results were above the average of all participants, others were below. The analysis of the particular documents will presumably reveal IRF’s strengths and weaknesses.
To summarize, we gained valuable experience from the CLEF experiments. The lack of genuinely different information retrieval engines makes a more sophisticated evaluation of fusion versus non fusion unavailing as can be seen from the overall rather narrow interval scales. Consequently, high priority will be given to the integration of further information retrieval engines in the MIMOR architecture. We expect that a learning process over some cycles will eventually lead to an advantageous system performance. 5
Several improvements need to be implemented for next years participation in the GIRT experiments: • MIMOR needs to be further automated • •
The optimization process needs to be refined
More basic retrieval systems need to be integrated Furthermore, we plan to further extend our work on the CLEF challenge and participate also in the multi-lingual track with our fusion system.
We would like to thank NIST and especially Paul Over and Darrin Dimmick for providing the source code of the IRF package.
We also thank the Jakarta and Apache projects’ teams for sharing Lucene with a wide community. Furthermore, we acknowledge the work of several students from the University of Hildesheim who implemented MIMOR as part of their course work.