This paper describes the third participation of IR-n system at CLEF-2003. Two previous participation are focused on Spanish monolingual task. This year, we participated in three different tasks: multilingual task (four languages), bilingual task (Italian-Spanish) and monolingual task (Spanish, German, French, Italian). This paper describes the experiments carried out as training process in order to set up the main system features and shows the results obtained. These results shows that IR-n system obtains good scores for the three tasks improving the average of the CLEF 2003 systems.
Information Retrieval (IR) systems have to find the relevant documents to a user’s query from a document collection. We can find different kind of IR system at the literature. On the one hand, if the document collection and the user’s question are written in the same language then the IR system is a monolingual system. On the other hand, if the document collection and the user’s question are written in different languages then the IR system is a bilingual (two different languages) or multilingual (more than two languages) system. Obviously, the document collection for multilingual system is written in two different languages at least.
This paper presents the adaptation of IR-n system [
– Italian-Spanish ² multilingual tasks:
– Spanish-German-French-Italian
IR-n system is an IR system based on passages instead of traditional systems based on full
document. Every passage is made up of a fragment or piece of text [
This proposal adds the following advantages: ² To consider the proximity of appearance of query terms into the documents. ² To define a new information transmission unit, more adequate both users than further treatment.
² To avoid normalization problems of documents. 2
This section presents the conceptual modelling of IR-n. The following main features are presented in the next subsections:
2. Similarity measure between the user’s question and the documents collection 3. Similarity measure between the user’s question and the documents collection based on similarity passages
2.1
First Passage Retrieval systems (PR) used the paragraph as passage size. The use of paragraph as passage unit caused the built of an heterogeneous collection of passages due to the different size of paragraphs. Moreover, this segmentation do not guarantee the fact that every passage is related to different subjects. For this reason, further proposals of PR systems used more than one paragraph as passage.
A different trend proposes the use of a number of words as passage [
Between the word and the paragraph exists an unit with structure that it is the sentence. The main aspect of the sentence is the self-contained of meaning. This aspect is very important in Information Retrieval because the answer of system can be understood by the user. Obviously, an only sentence does not have an enough identity to determine if a document that contains it is relevant in relation to certain topic. Although it establishes some limits and helps to value the fact that terms of the user’s query appear in the same sentence. Since the sentence does not have an entity the sufficiently complete to define a passage, the passages are defined as a set of consecutive sentences. The system IR-n uses the sentence as basic information unit to define the passages. The size of passage can be adapted to improve the efficiency of the IR-n system. The size of passage is measured in number of sentences. The use of sentence to define passages presents advantages against the use of paragraph or word.
The use of paragraph as unit to define the passage has two main problems: - It is possible that the documents collection does not have information about the paragraphs marks in the document. - The paragraphs can be used for visual reasons instead of structural reasons of the document. The use of a number of words as unit to define the passage presents two problems: - The number of words to be considered as a passage depends on the writing style used. The same event is described using less words in a document of news agency that in a newspaper.
If the same event is also written in a novel the number of words will be bigger. - If the system uses words to define the passage, it can happen that the lack of structure of the considered text fragment can cause the does not understanding of the text recovered.
This is due to the fact that the passage can start and end in any part of document.
Finally, the use of sentences to define the passage presents the following advantages:
- A sentence usually expresses an idea in the document.
- Usually documents use the punctuation signs to separate ideas. There are algorithms to
obtain each sentence from a document using their superficial structure with a precision of
100% [
At the beginning, the system IR-n used the traditional measure of the cosine [
X (wQ;t ¢ wP;t) t2Q^P wQ;t = f reqq;t ¢ loge(
N ¡ f reqt )
f reqt wP;t = 1 + loge(1 + loge(f reqp;t + 1)) (1) (2) (3) where f reqY;t is the number of appearances or the frequency of the term t in the passage or the question Y . N is the total number of documents in the collection and f reqt is the number of different documents that contain the term t.
All systems of PR calculate the similarity measure of the document in function of the similarity
measure of their passages using the sum of similarity measure for each passage or using the best
similarity measure of passage for each document. The experiments carried out in [
Our proposal is based on the fact that if a passage is relevant then the document is also relevant. In fact, if a PR system uses the sum of every similarity measure of passage the the system has the same behavior like IR system based on document adding concepts of proximity.
Moreover, the use of the best similarity measure of passage allows to obtain the best passage improving further search process.
The system IR-n calculates the similarity measure of the document based on the best similarity measure of their passages in the following way: sim(Q; D) = max sim(Q; Pi) 8i:Pi2D (4) 2.4
The techniques of query expansion allow to locate relevant documents that do not contain the exactly words of the user’s query.
Different studies have been carried out in order to add these techniques to the system
IRn. These studies were two: the former, in the CLEF-2001 [
This year, the pattern proposed in [
This section describes the experiments carried out in order to obtain and to optimize some system’s features to improve the performance of the system. The training corpus used in these experiments was the Clef-2002 document collection. Moreover, all experiments have only been carried out using short questions, that is the system only used the title and description from the query. The following subsections explain the specific experiments carried out to every CLEF task. 3.1
First experiments are focused on establishing the adequate number of sentence (N ) to make up the passage for each language (Spanish, Italian, German, French and English). The performance of the system was measured using the standard average interpolated precision (AvgP). For every language, the stemmers and the stop-word lists used were provided by the organization of the clef1.
Table 1 shows the scores achieved for each language without query expansion. The German scores are also obtained without splitting the compound nouns. The obtained figures show the best results for German, French and English using 14 sentences, for Spanish using 9 sentences and for Italian using 8 sentences. The bigger size for German, English and French is due to the kind of document collections used for each language. The three collections are made up of documents with a bigger number of sentences that Spanish and Italian documents. Moreover, the lowest scores achieved for German language (0,3364) show the influence of splitting the compound nouns. The
5 0,4839 0,4073 0,3236 0,4260 0,4675 lack of an algorithm to split compound nouns should us to use a list of more frequently compound noun made up of 200000 terms. The scores achieved for German using the compound list were better as show the table 2.
Once, the size of passage was established for each language, the following experiment was
carried out in order to study the influence of query expansion. The IR-n system uses a feedback
technique to apply the query expansi´on. The IR-n system adds to the query the T more important
term from the P more relevant passage according to [
We use three different translator in order to obtain an automatic translation of queries. The
three used translator were PowerTranslator, FreeTranslator2 and Google3. In multilingual task (4
languages), the queries written in English were translating to French, Spanish and German. Once,
we obtained the query translation, four different experiments were carried out in order to choose
the best translation. The three first ones only used a translation and the last one used the merge
of all translations as query. Table 3 shows the scores achieved in the four experiments developed
using every document collection as the same way as monolingual task. Best scores were achieved
using the merge of translations. The IR-n system was run obtained three different rank document
collections in multilingual task. According to [
T
P Spanish Italian German French English
No expansion 0,5042 0,4207 0,4027 0,4731 0,5057 in which rsvj is the original retrieval status value, and rsvmin and rsvmax are the minimum and maximum document scores values that a collection could achieve for the current request. Table 5 shows the scores achieved using both merging method. These scores show that best results are obtained using M2 merging method.
In bilingual task, an additional problem was found. We do not have a direct translator ItalianSpanish and Spanish-Italian. We had to translate Italian to English and late English to Spanish. This process carries out more errors than a directly translation. Table 6 shows the scores achieved in the bilingual task. At the same way as multilingual task, the best score was obtained using the merge of translations. 4
The IR-n system used in order to participate in CLEF’2003 was the best IR-n configuration obtained in the training process using the CLEF’2002 collection.
The following subsections show the runs carried out and the scores achieved in the monolingual, bilingual and multilingual tasks. 4.1
Two different runs have been submitted for each Spanish, French and Italian monolingual tasks. The first run does not use query expansion and the last one using it (IRn-xx -noexp and IRn-xx -exp, where xx are the language –es, fr or it –). Four different runs were submitted for German. The first and second runs follows the same strategies as previous languages but without splitting the compound nouns (IRn-al -noexp-nsp and IRn-al -exp-nsp). The third and fourth experiments used the splitting of compound nouns with and without expansion (IRn-al -noexp-sp and IRn-al -exp-sp)
Tables 7, 8 , 9 and 10 show the scores achieved for each run in the monolingual task. IRn system using query expansion obtained better results than the average scores of CLEF 2003 systems for Spanish, French and German and lower scores in Italian.
Traslation Italian-Spanish 0,4207
Free 0.3367
Power 0.3490
Babel 0.3480
Power+Free+Babel 0.3663 +8.75% -1.44% Two different runs have been submitted for bilingual tasks (Italian-Spanish). The first run does not use query expansion and the last one using it (IRn-ites-noexp and IRn-ites-exp). The English was used as intermediate language due to the lack of a direct translator Italian to Spanish. Table 11 shows that IR-n system using query expansion for bilingual task increase around a 26% the average scores of CLEF’2003 bilingual system. Five runs were submitted to Multilingual task made up four languages. The first runs (IRnm-noexp-nsp) shows the scores achieved by IR-n system without query expansion and without splitting the German compound nouns. The second one (IRn-m-exp-nsp) presents the performance of IR-n system using query expansion and without splitting the compound nouns. The third and fourth runs (IRn-m-noexp-sp and IRn-m-exp-sp, respectively) are the same experiments but using the splitting of German compound noun. Finally, an additional experiment (IRn-mi-exp-sp) was carried out using the same passage’s size for all languages (10 sentences), and using the query expansion and the splitting of compound nouns. This size was obtained experimentally in the training task.
Table 12 shows that IR-n system improve the average scores of CLEF’2003 around a 23% using the AvgP measure. Moreover, IR-n system also obtains around a 23% of improvement using he same size of passages. 5
General conclusions are positive. On the one hand, IR-n system has obtained better results than the average of CLEF’2003, excluding for Italian monolingual task. Moreover, we want to remark that all runs submitted are carried out only using short queries (title and description) and the average provided by CLEF organization is made up for all system (systems using both short or long queries). On the other hand, the achieved improvement using a list of the most frequent compound nouns in German conduct us to develop for next participation an algorithm to split the compound nouns.
Also, we want to emphasize the good performance of IR-n system in our first participation in bilingual and multilingual tasks. However, we planned to use a new method but time problem stopped us to submitted a new run. We hope to participate with the new method in the next conference.
Moreover, we want to underline the good scores achieved using the same size of passages for all languages. Finally, we want to emphasize that IR-n system is an information retrieval system based on passages and independent of languages according to the scores obtained in our participation at CLEF 2003.