This paper describes the participation of RICOH in the Monolingual Information Retrieval tasks of the Cross-Language Evaluation Forum (CLEF) 2003. We used our system with same kind of stemmer, same options and different parameters for 5 European languages to compare each result. Total performance of the system was reasonable. For French, German and Italian, we found some problems. For CLEF 2003 monolingual information retrieval task, RICOH submitted runs for French, German, Italian, Spanish and Dutch. We have worked on English and Japanese text retrieval in past few years [2,3,4,5]. CLEF 2003 experiments were our first trials for European languages. Our main focuses at the experiments were: 1) to test our approach based on the probabilistic model in European languages 2) to find language-specific problems This paper is organized as follows: Section 2 introduces outline of our system, section 3 shows modifications for the experiments, section 4 describes the results, and section 5 reports some conclusions.
We used “title” and “description” fields of each topic. Input topic string is transformed into a sequence of stemmed tokens using the tokenizer and the stemmer. Stop words are eliminated using a stopword dictionary. Two kinds of terms are extracted from stemmed tokens for initial retrieval. “single term” is each stemmed token and “phrasal term” consists of two adjacent tokens in the stemmed query string.
Each query term is assigned a weight wt, and documents are ranked according to the score sq,d as follows: 2.2 Initial retrieval
K = k1 (1 − b) + b ld
lave where N is the number of documents in the collection, nt is the document frequency of the term t, ft,d is the in-document frequency of the term, ld is the document length, lave is the average document length, and k'4, k1 and b are parameters.
Weights for phrasal terms are set lower than those for single terms.
As a result of initial retrieval, top 10 documents are assumed to be relevant (pseudo-relevant) to the query and selected as
a “seed” of query expansion. Candidates of expansion terms are extracted from the seed documents by the same way as in
the query term extraction mentioned above. Phrasal terms are not used for query expansion. The candidates are ranked on
the Robertson's Selection Value [
The weight of initial query term is re-calculated with the same formula as above, but with a different α value and an additional adjustment to make the weight higher than expansion terms.
Using the initial query terms and expansion terms, the ranking module performs second retrieval to produce the final result.
There are four items in the system that need adjustments depending on the language, 1) tokenizer, 2) stemmer, 3) stopword dictionary and 4) training data. We used the same tokenizer originally developed for English for all target languages. The others are as follows.
We used Snowball stemmers [
We trained the system by selecting the best parameter-set from 500 candidate parameter-sets for each language to get the highest average precision score.
There was a bug in our training scripts. The system was trained using CLEF 2002 queries and CLEF 2003 data collections, instead of CLEF 2002 data collections. This mismatch resulted in extra noise documents in retrieved documents and made tuning performance rather worse.
Table 2 shows the results with and without training. without training with training using 2002 data with training using 2003 data
we expect that the main causes are that we have no German compound splitter. For the French result, we need time to analyze it.