This paper describes the participation of RICOH in the Monolingual Information Retrieval tasks of the Cross-Language Evaluation Forum (CLEF) 2003. We used our system with same kind of stemmer, same options and different parameters for 5 European languages to compare each result. Total performance of the system was reasonable. For French, German and Italian, we found some problems.
For CLEF 2003 monolingual information retrieval task, RICOH submitted runs for French, German, Italian, Spanish and Dutch. We have worked on English and Japanese text retrieval in past few years [2,3,4,5]. CLEF 2003 experiments were our first trials for European languages. Our main focuses at the experiments were:
1) to test our approach based on the probabilistic model in European languages 2) to find language-specific problems This paper is organized as follows: Section 2 introduces outline of our system, section 3 shows modifications for the experiments, section 4 describes the results, and section 5 reports some conclusions.
Before describing our approach to European languages, we give the system description as background. The basic features of the system are as follows:
Effective document ranking based on the probabilistic model [8] with query expansion using pseudo-relevance feedback [2] Scalable and efficient indexing and search based on the inverted file module [4] This system was also used for TREC and NTCIR experiments and showed its effectiveness.
In the following sections, we explain the processing flow of the system [5].
We used "title" and "description" fields of each topic. Input topic string is transformed into a sequence of stemmed tokens using the tokenizer and the stemmer. Stop words are eliminated using a stopword dictionary. Two kinds of terms are extracted from stemmed tokens for initial retrieval. "single term" is each stemmed token and "phrasal term" consists of two adjacent tokens in the stemmed query string.
Each query term is assigned a weight w t , and documents are ranked according to the score s q,d as follows:
where N is the number of documents in the collection, n t is the document frequency of the term t, f t,d is the in-document frequency of the term, l d is the document length, l ave is the average document length, and k ' 4 , k 1 and b are parameters. Weights for phrasal terms are set lower than those for single terms.
As a result of initial retrieval, top 10 documents are assumed to be relevant (pseudo-relevant) to the query and selected as a "seed" of expansion. Candidates of expansion terms are extracted from the seed documents by the same way as in the query term extraction mentioned above. Phrasal terms are not used for query expansion. The candidates are ranked on the Robertson's Selection Value [6], or RSV t and top ranked terms are selected as expansion terms. The weight is recalculated as w2 t with the Robertson/Sparck-Jones formula [7]
where R is the number of relevant documents, r t is the number of relevant documents containing the term t and α is a parameter.
The weight of initial query term is re-calculated with the same formula as above, but with a different α value and an additional adjustment to make the weight higher than expansion terms.
Using the initial query terms and expansion terms, the ranking module performs second retrieval to produce the final result.
There are four items in the system that need adjustments depending on the language, 1) tokenizer, 2) stemmer, 3) stopword dictionary and 4) training data. We used the same tokenizer originally developed for English for all target languages. The others are as follows.
We used Snowball stemmers [1] for all target languages because 1) we didn't have stemmers for European languages except for English 2) we aren't familiar these languages to develop stemmers and 3) unlike the earlier result [9], Snowball stemmers showed their reasonable efficiencies for preparatory experiments. Table 1 shows the results using CLEF 2002 data with and without stemming.
We didn't use stopword dictionaries because we didn't have them.
We trained the system by selecting the best parameter-set from 500 candidate parameter-sets for each language to get the highest average precision score.
There was a bug in our training scripts.
Table 3 shows summarize of our official results for CLEF 2003. Table 4 shows summarize of our additional results using parameters trained with CLEF 2002 (correct) data collections. The additional result for Dutch is same as formal one because data collection is same. The additional results for French and German are same as formal ones because new parameters selected by correct training scripts were unchanged from formal runs.
Our approach was tested and its results were reasonable. According to "comparison to median by topic", the Spanish result may be good, but the German and French results may not.
We compared results for each language under same conditions. The comparison brought us questions for each language. Why query expansion is not effective for French and Italian? Why retrieval of some queries failed badly in French and German? It is likely that our query expansion doesn't work well with few relevant documents. There is a strong correlation between the effectiveness of our expansion and the number of relevant documents for each language. This correlation should be checked with each query.
We think that there are different kind of problems about failure of queries in French and German. For the German result, we expect that the main causes are that we have no German compound splitter. For the French result, we need time to analyze it.