The paper describes our participation in monolingual tasks at CLEF 2006. We have submitted results for the following languages: English, French, Portuguese and Hungarian. We focused on studying different weighting schemes (okapi and dfr) and retrieval strategies (passage retrieval and document retrieval) to improve retrieval performance. After an analysis of our experiments and of the official results at CLEF, we find that our different configurations (French, Portuguese and Hungarian) achieve considerably improved scores.
In our sixth participation at CLEF, we focused on evaluating: a new weighting model (dfr), retrieval based on passages/documents and setting the best configuration to each language. Specifically, we participated in the following languages: English, French, Portuguese and Hungarian.
IR-n system [
Futhermore, our team participated in other tasks at CLEF-2006 as GeoCLEF, CL-SR... and
we also applied PR systems in other tasks, such as Question Answering (QA) [
This paper is organized as follows: next section describes IR-n system and its new changes. Following, we describe the task developed at CLEF 2006 by our system and the training. And finally, we present the achieved results and the conclusions. 2
In this section the main characteristics of the IR-n system are presented and details are given on the resources and the techniques of the system used in CLEF 2006. 2.1
Resources: stemmers and stopword lists We used the stemmers and stopwords lists available on the web http://www.unine.ch/info/clef. We highlight that Hungarian collections are encoded in UTF-8. 2.2
IR-n system uses several similarity measures. This year, the weighting model dfr [
The document ranking produced by each weighting model is represented using the same general expression, namely as the product of a document-based term weight by a query-based term weight: sim(q, d) = X
wt,p · wt,q t∈q∧p List of variables Here is described the list of variables used in the following formulas. 2, 3: • ft,p is the frequency of the term t on the passage p, • ft,q is the frequency of the term t on the query q, • n is the number of documents in the collection, • nt is the number of documents which t appears, • c, k1, b and k3 are constant values, • ld is the length of the document, • avgld is the average of the length of the documents Okapi Using the okapi model, the relevance score of a passage p for query q is given by: wt,q = wt,p = (k1 + 1) · ft,p
K · ft,p (k3 + 1) · ft,q k3 · ft,q · wt ld K = (1 − b) + b · avrld wt = log2 n − nt + 0.5 nt + 0.5 (1) (2)
Using this model, the weight of a passage p for query q is given by: wt,q = ft,q wt,p = (log2(1 + wt) + wt0,p · log2( 1 + wt )) · nt · (wt0,p + 1)
ft + 1 wt wt0,p = ft,p · log2(1 + c · avrld )
ld wt = ft n (3) 2.3
Most IR systems use query expansion techniques [
This section describes the training process which has been carried out in order to obtain the best features to improve the performance of the system. Firstly, the collections and resources are described. The following section explains the specific experiments which we have carried out. 3.1
This year our system has participated in the following monolingual tasks: English, French, Portuguese and Hungarian. Table 1 shows the characteristics of the language collections.
English
French Portuguese Hungarian
The Angeles Times 94 Glasgow Herald 95 Le Monde 94/95 SDA French 94/95 Pu´blico 94/95
Folha 94/95 Magyar Hirlap 02
169477
579 MB
25
529
WSAvg 20 177452 • SDAvg is the average of sentences in each document. • WDAvg is the average of words in each document. • WSAvg is the average of words in each sentence. 3.2
The aim of the experiment phase is set up the optimum value of the input parameters for each collection. For training has been used the collections CLEF-2005 (English, French, Portuguese and Hungarian). Query expansion techniques have also been used in all languages. In addition, we describe the input parameter of the system: • Size passage (sp): We established two size passage: 8 (normal passage) or 30 (big passage). • Weighting model (wm): We use two weighting models: okapi and dfr. • Opaki parameters: these are k1, b and avgld (k3 is fixed as 1000). • Dfr parameters: these are c and avgld. • Query expansion parameters: If exp has value 1, this denotes we use relevance feedback based on passages in this experiment. But, if exp has value 2, the relevance feedback is based on documents. Moreover, np and nd denote the k terms extracted from the best ranked passages (np) or documents (nd) from the original query. • Evaluation measure: Mean average precision (avgP) is the evaluation measure used in order to evaluate the experiments. 3.2.1
As we can see at table 2, the best weighting scheme is dfr. Therefore, the passage size 8 obtains 0.5403 as average precision.
k1 b exp np
nd For French language, the best weighting scheme is okapi with 9 as passage size. This configuration has obtained 0.3701 as average precision. The best weighting scheme is dfr to Hungarian language, whereas the passage size is 30 to Hungarian. The best average precision obtained by this configuration is 0.3644. wm dfr dfr dfr dfr dfr okapi okapi wm dfr dfr dfr dfr dfr dfr okapi okapi okapi avgP 0.3119 0.3333 0.3334 0.3534 0.3644 0.2930 0.3264 avgP 0.3362 0.3484 0.3457 0.3474 0.3733 0.3948 0.3283 0.3676 0.3793 b exp np
nd exp np
nd The best configuration by Portuguese language is the same as Hungarian language (dfr as weighting scheme and 30 as size passage). The average precision obtained with this configuration is 0.3948. In conclusion, the table 6 shows the best configuration for each language. These configurations were used at CLEF 2006.
language
8-dfr-exp 9-okapi-exp 30-dfr-exp 30-dfr-exp We submitted four runs for each language in our participation (except for English that we have submitted 1 run) in CLEF 2006. The best parameters, i.e. those that gave the best results in system training, were used in all cases.
This is the description of the runs that we submitted at CLEF 2006: – yy is the passage size – xx is the weighting model used (dfr or okapi) – z is the expansion query (not used ’nexp’)
The official results for each run are showed in Table 7. Like other systems which use query expansion techniques, these models also improve performance with respect to the base system. Our results are appreciably above average in all languages, except for English where they are sensible below the average. This results present that the percentage of improvement in Portuguese is 15.43% in avgP. In this seventh CLEF evaluation campaign, we proposed a different configuration for the English, French, Portuguese and Hungarian languages (see table 6). In order to enhance retrieval performance, we have evaluated different weighting models using also a query expansion approach based on passages and documents.
The results of this evaluation indicate that for the French, Portuguese and Hungarian languages proved to be effective (see table 7) because the results are above average. However, the English language has obtained results sensible below average.
For Portuguese language, the best results are obtained by okapi weighting model. For other languages (English, French and Hungarian), the best results are obtained by dfr (see table 7).
The best passage size for French was 9, although for other languages (English, Portuguese and Hungarian) was 30 (this passage size is comparable to IR based on the complete document).
As in previous evaluation campaigns, pseudo-relevance feedback based on passages improves mean average precision statistics for all languages, even though this improvement is not always statistically significant.
Lastly, we outline the future directions that we plan to undertake are evaluate languages as Bulgarian or Spanish. Therefore, as future work we also consider to research into different ways of providing Natural Language information to basic IR and evaluating the impact of each approach.
This research has been partially funded by the Spanish Government under project CICyT number TIC2003-07158-C04-01 and by the Valencia Government under project number GV06-161.