This paper describes our participation in monolingual tasks at CLEF-2005. In this research we have worked in the following languages: English, French, Portuguese, Bulgarian and Hungarian. Our task has been focused on using combined different size passages to improve the Information Retrieval process. Once we have studied the experiments which have been carried out and the official results at CLEF, we have realized that this combining model gets better the achieved scores considerably.
Information Retrieval systems based on passages (PR) [2] determine the relevance of a document regarding to a question. This relevance is obtained from the similarity of different fragments in this document regarding to the same question. This models not only let to improve the location of relevant documents, but also let us to find the most relevant part of the document accurately. This last advantage allows us that these systems which are used in other tasks as Question Answering (QA).
PR systems are classified according to how the passages are determined in each document. IR-n system is a PR system which defines the passages based on a fixed number of sentences. This provides the passages with some syntactical content. Last years our researches with IR-n system are based on detecting the suitable size for each collection ( to experiment with test colletion ), but determining the similarity of a document based on the passage with more similarity. This year the score which is given to each document is based on the similarity of several size passages.
This paper is organized as follows: next section describes IR-n system and its new changes. Following, we describe the task developed at CLEF 2005 by our system and the training. And finally, we present the archieved results and the conclusions.
IR-n system [3] was developed in 2001. It was written in C++ entirely, although it has been used external tools (stemmers) in few occasions. Last year [4] the system was designed and developed again with the aim of getting more information (size of passages in words) and to improve the process speed.
The system works in Linux system without excesive requeriments, nevertheless, in view of the fact that the search process is carried out with structures load in memory, it is convenient that the computer has the enough memory.
In this section is presented the main charateristics of IR-n system and it is detailed the combined passages method used in this edition.
This was the first year that we have worked with Bulgarian and Hungarian languages. It has been used the stemmers and stopwords lists available on the web http://www.unine.ch/info/clef. We can highlight that Hungarian and Bulgarian collections are encoded in UTF-8. In addition Bulgarian stemmer is developed in perl to support UTF-8. The rest of the stemmers are developed in C.
IR-n system is ready for using several similarity measures: cosine [5], pivoted cosine [7] and okapi [6]. To the last one the values of parameters (k1,b,avg) could be update in a easy way, in order to get the best results.
On the whole, experiments carried out by okapy measure show us that we could obtain the best results. Furthermore, we have contrasted normalization concept with size passage again. Previous versions of IR-n system does not use size passage in the similarity measures because all the passages had the same size. Last edition we could check that results improved if it was considered size passages.
Most of IR systems use query expansion techniques [1] based on adding the most frequent terms contain in the more relevant documents to the original query. Architecture IR-n allows us to use query expansion based on more relevant passages or documents. In fact, last edition we got better results using the more relevant passages.
The present year, technique called 'combined passages' has been developed. The model consists of applying similar techniques for merging relevant document lists in multilingual task but using relevant passage lists of different size.
This model consists of using different size passages in order to get relevant document lists. The list which have been obtained are combined sequently. Table 1 shows different methods used to obtain the ranking of scores.
We have used four methods: MAX merges the n list and if a document is in several lists it will provide the highest score. SUM carries out the average of the scores. The methods 3 and 4 are as the previous ones but using normalization. This normalization is carried out subtracting the score of each document RSV k from the minimum score of the list and dividing by max(RSV K ) − min(RSV k ).
Obviously, this model improves and involves speed process. However in IR-n system architecture, this trouble does not increase the speed process of the system. This happens because
This section describes the training process which has been carried out in order to obtain the best features to improve the performance of the system. Firstly, the collections and resources are described. The following section explains the specific experiments which we have carried out.
This year our system has participated in the following tasks: Monolingual: English, French, Portuguese, Bulgarian y Hungarian. Table 2 shows the characteristics of the collection which we have worked.
• SDAvg is the average of sentences in each document.
• WDAvg is the average of words in each document.
• WSAvg is the average of words in each sentence.
This year several tests have been carried out in order to establish the best similarity measure for each language and to provide the value of input parameters in the system. We have evaluated the following languages: English, French and Portuguese. We could not evaluated Bulgarian and Hungarian languages because we did not have data of last years, as result of this, we have choosed the similarity measures and the parameters for these languages comparing them with the rest ones.
The aim of the experiments phase is set up the optimum value of the input parameters for each collection. For training has been used the collections CLEF-2003 (English and French) and CLEF-2004 (Portuguese). Query expansion techniques have also been used in all languages.
It has been performed experiments for setting the size passages and the values of parameters in okapi system which allow us to obtain the best results. As the table shows 3 the size passage is the same for all languages ( 8 sentences), however in French is 9 sentences.
On the one hand, experiments which are carried out with query expansion tried to fix the number terms to add in the original query and the number of documents (passages) to take into account. Furthermore, we have evaluated the use of different size passages. We have got the best results with 10 terms in every test and it has been used the 5 or 10 passages more relevants depending on the specific language. On the other hand, it can be appreciated that the size passage is 8 in this case also. As we check in the table 5 query expansion in fixed system allows us to improve scores between 3.6% and 7.2% according to the different languages.
Combined passages method consist on using the similarity values which are provided from different size passages of the same document to obtain the document similarity. Because of that, it has been defined three types of passages: small, medium and big passage. The number of sentences which composes each passage is the following: P 1 = (3, 4, 5, 6) P 2 = (7, 8, 9, 10) P 3 = (11, 12, 13, 14) Experiments have been carried out by means of using one of each type. In this way is obtained the similarity of each passage. Document similarity is got using one of the four method described previously (see table 1). The tests which get better results are showed in table 6. 'Model' belongs to the combined method used and the columns P1, P2 and P3 are size passages which are provided by the best combined method. The columns P1, P2 and P3 are a small, medium and big passages respectively.
As we show the combined method, which provides the best results, is the method 2 (SUM without normatization) for all languages.
We have proved that the results increases comparing with fixed system in all languages except in French.
We have carried out the same tests with query expansion and the results improves in all languages, although the increase is not meaningful in Portuguese.
The best combined method for English and Portuguese carries on being the method 2 (SUM), but in French is 1 (MAX) (see table 7).
As we check in table 5 combined passages system improves the results between 3.1% and 7.7% according to each language.
Tables 9 and 10 compare both methods showing the best results obtained in each test. We have submitted three runs for each language in our paticipation at the CLEF-2005. The best parameters, which provided the best results in the training, have been used in all cases. We did not have any training data about Bulgarian and Hungarian languages, therefore we have used the parameters of English language.
• IRn-xx-fexp is based on using the fixed size passages system which obtains the best results in training. Query expansion techniques have been used in these runs.
• IRn-xx-vnexp Combined passages system has been used without applying query expansion techniques.
• IRn-xx-vexp Combined passages system has been used with query expansion techniques.
Officials results for each run are showed in table 11. The model IRn-xx-vnexp is taken as a reference. As other models which use query expansion techniques, our model also increases the performance on the base system.
In this table 11 the two models with query expansion are compared. This one presents that the percentage of improvement in the combined model is around 4% of increase avgP in every language (except for Bulgarian).
As shown table 11, our results are above average in all languages appreciably, except for Bulgarian that the results are below average.
This paper has described PR model which uses similarity values of three different size passages for each document in order to obtain the similarity of document regarding the question. This model has allowed us to improve the results around 4% according to models which used only a fixed size passage. Lastly, we outline the future directions that we plan to undertake not only to improve this model, but also to be applied it in Question Answering task.
This research has been partially funded by the Spanish Government under project CICyT number TIC2003-07158-C04-01 and by the Valencia Government under project numbers GV04B-276 and GV04B-268