This paper describes our participation in monolingual tasks at CLEF-2005. In this research we have worked in the following languages: English, French, Portuguese, Bulgarian and Hungarian. Our task has been focused on using combined different size passages to improve the Information Retrieval process. Once we have studied the experiments which have been carried out and the official results at CLEF, we have realized that this combining model gets better the achieved scores considerably.
Information Retrieval systems based on passages (PR) [
PR systems are classified according to how the passages are determined in each document. IR-n system is a PR system which defines the passages based on a fixed number of sentences. This provides the passages with some syntactical content. Last years our researches with IR-n system are based on detecting the suitable size for each collection ( to experiment with test colletion ), but determining the similarity of a document based on the passage with more similarity. This year the score which is given to each document is based on the similarity of several size passages.
This paper is organized as follows: next section describes IR-n system and its new changes.
Following, we describe the task developed at CLEF 2005 by our system and the training. And
finally, we present the archieved results and the conclusions.
IR-n system [
The system works in Linux system without excesive requeriments, nevertheless, in view of the fact that the search process is carried out with structures load in memory, it is convenient that the computer has the enough memory.
In this section is presented the main charateristics of IR-n system and it is detailed the combined passages method used in this edition. 2.1
This was the first year that we have worked with Bulgarian and Hungarian languages. It has been used the stemmers and stopwords lists available on the web http://www.unine.ch/info/clef. We can highlight that Hungarian and Bulgarian collections are encoded in UTF-8. In addition Bulgarian stemmer is developed in perl to support UTF-8. The rest of the stemmers are developed in C. 2.2
IR-n system is ready for using several similarity measures: cosine [
On the whole, experiments carried out by okapy measure show us that we could obtain the best results. Furthermore, we have contrasted normalization concept with size passage again. Previous versions of IR-n system does not use size passage in the similarity measures because all the passages had the same size. Last edition we could check that results improved if it was considered size passages. 2.3
Most of IR systems use query expansion techniques [
The present year, technique called ’combined passages’ has been developed. The model consists of applying similar techniques for merging relevant document lists in multilingual task but using relevant passage lists of different size.
This model consists of using different size passages in order to get relevant document lists. The list which have been obtained are combined sequently. Table 1 shows different methods used to obtain the ranking of scores.
We have used four methods: MAX merges the n list and if a document is in several lists it will provide the highest score. SUM carries out the average of the scores. The methods 3 and 4 are as the previous ones but using normalization. This normalization is carried out subtracting the score of each document RSVk from the minimum score of the list and dividing by max(RSVK ) − min(RSVk).
Obviously, this model improves and involves speed process. However in IR-n system architecture, this trouble does not increase the speed process of the system. This happens because IR-n system produces a segmentacion of the documents in passages in the search time and the calculation of similarity is carried out on structures load in memory. 3
This section describes the training process which has been carried out in order to obtain the best features to improve the performance of the system. Firstly, the collections and resources are described. The following section explains the specific experiments which we have carried out. 3.1
This year our system has participated in the following tasks: Monolingual: English, French, Portuguese, Bulgarian y Hungarian. Table 2 shows the characteristics of the collection which we have worked.
• SDAvg is the average of sentences in each document. • WDAvg is the average of words in each document.
• WSAvg is the average of words in each sentence. 3.2
This year several tests have been carried out in order to establish the best similarity measure for each language and to provide the value of input parameters in the system. We have evaluated the following languages: English, French and Portuguese. We could not evaluated Bulgarian and Hungarian languages because we did not have data of last years, as result of this, we have choosed the similarity measures and the parameters for these languages comparing them with the rest ones.
The aim of the experiments phase is set up the optimum value of the input parameters for each collection. For training has been used the collections CLEF-2003 (English and French) and CLEF-2004 (Portuguese). Query expansion techniques have also been used in all languages. avgP 0.5083 0.5240 0.4741 expd 5 5 10 It has been performed experiments for setting the size passages and the values of parameters in okapi system which allow us to obtain the best results. As the table shows 3 the size passage is the same for all languages ( 8 sentences), however in French is 9 sentences. 3.2.2
On the one hand, experiments which are carried out with query expansion tried to fix the number terms to add in the original query and the number of documents (passages) to take into account. Furthermore, we have evaluated the use of different size passages. We have got the best results with 10 terms in every test and it has been used the 5 or 10 passages more relevants depending on the specific language. On the other hand, it can be appreciated that the size passage is 8 in this case also.
As we check in the table 5 query expansion in fixed system allows us to improve scores between 3.6% and 7.2% according to the different languages. 3.2.3
Combined passages method consist on using the similarity values which are provided from different size passages of the same document to obtain the document similarity. Because of that, it has been defined three types of passages: small, medium and big passage. The number of sentences which composes each passage is the following:
P 1 = (3, 4, 5, 6) P 2 = (7, 8, 9, 10) P 3 = (11, 12, 13, 14)
Experiments have been carried out by means of using one of each type. In this way is obtained the similarity of each passage. Document similarity is got using one of the four method described previously (see table 1).
English 0.5083
French 0.5240 Portuguese (2004) 0.4741 avgP with exp 0.5267 0.5488 0.5084
Dif +3.6% +4.7% +7.2%
The tests which get better results are showed in table 6. ’Model’ belongs to the combined method used and the columns P1, P2 and P3 are size passages which are provided by the best combined method. The columns P1, P2 and P3 are a small, medium and big passages respectively.
As we show the combined method, which provides the best results, is the method 2 (SUM without normatization) for all languages.
We have proved that the results increases comparing with fixed system in all languages except in French. 3.2.4
We have carried out the same tests with query expansion and the results improves in all languages, although the increase is not meaningful in Portuguese.
The best combined method for English and Portuguese carries on being the method 2 (SUM), but in French is 1 (MAX) (see table 7).
As we check in table 5 combined passages system improves the results between 3.1% and 7.7% according to each language.
Tables 9 and 10 compare both methods showing the best results obtained in each test.
• IRn-xx-fexp is based on using the fixed size passages system which obtains the best results in training. Query expansion techniques have been used in these runs. • IRn-xx-vnexp Combined passages system has been used without applying query expansion techniques. • IRn-xx-vexp Combined passages system has been used with query expansion techniques.
Officials results for each run are showed in table 11. The model IRn-xx-vnexp is taken as a reference. As other models which use query expansion techniques, our model also increases the performance on the base system.
In this table 11 the two models with query expansion are compared. This one presents that the percentage of improvement in the combined model is around 4% of increase avgP in every language (except for Bulgarian).
As shown table 11, our results are above average in all languages appreciably, except for Bulgarian that the results are below average. 5
This paper has described PR model which uses similarity values of three different size passages for each document in order to obtain the similarity of document regarding the question. This model has allowed us to improve the results around 4% according to models which used only a fixed size passage.
French Portuguese Hungarian Bulgarian
Run IRn-en-vexp IRn-en-fexp IRn-en-vnexp CLEF Average
IRn-fr-vexp IRn-fr-fexp IRn-fr-vnexp CLEF Average
IRn-pt-vexp IRn-pt-fexp IRn-pt-vnexp CLEF Average
IRn-hu-vexp IRn-hu-fexp IRn-hu-vnexp CLEF Average
IRn-bu-vexp IRn-bu-fexp IRn-bu-vnexp
This research has been partially funded by the Spanish Government under project CICyT number TIC2003-07158-C04-01 and by the Valencia Government under project numbers GV04B-276 and GV04B-268