In this paper we will present the result of the interactive CLEF experiment at the University of Alicante. Our aim was to compare two interactive approaches: one based on passages (presented at the iCLEF 2002 [5]), and a new interactive approach based on syntactic semantic patterns. These patterns are composed by the main verb of a sentence plus its arguments, and they are extracted automatically from the passages. With this, these patterns show only the basic information of each sentence. The objective was to know which of these approaches is most useful and fast in the selection of relevant documents by the user in a language different than one of the query (and of the user). The results show that both approach are useful, but the approach based on syntactic semantic patterns is, in the majority of cases, more fast. Finally, with these approaches we avoid the use of Machine Translation systems, due to the problems that they have in Interactive Cross-Language Information Access tasks.
One of the most important aspects of the Interactive Multilingual Information Access is the way in which the system shows the retrieved documents to the user; mainly, the way in which the system shows the relevant information. Only with this information the user must decide if the retrieved documents are relevant or not. This is a key point in order to ensure the correct selection of documents, and a key point for future refinements of the query.
The main problem is the multilingualism: the user formulates the query in one language, but the relevant documents are written in a different language. To deal with this situation, there are two main solutions: to show the relevant documents to the user in his/her language, or to show the relevant documents in the language of the documents. In the first case, a translation of the document with a Machine Translation system is necessary. However, there are many problems related to Machine Translation. In the second case –to show the information in the language of the document–, it is possible that the user does not be able to understand the information, and does not be able to decide which documents are the relevant ones.
At iCLEF 2002, the University of Alicante proposed the use of passage for the interaction with the user. In this approach, the system selects the most important passage of the retrieved documents. Each passage is translated to the language of the user with a Machine Translation system and, then, the translated passages are shown to the user.
The experiment concluded that this approach based on passages is more fast and has more
precision than the approaches based on the whole document. The user only read the relevant
passage, not the whole document. This is enough information to decide if the document is relevant
or not with high precision. However, there is an important problem with this approach: a lot of
passages was unreadable for the user due to problems with the machine translation from English
to Spanish [
This year we want to improve this approach in two aspects: first, we want to improve the interaction speed –that is, the time consuming by the user between the uploading of the passage to the decision about its relevance–; and second, we want to improve the recall and precision in the selection of relevant documents. On other hand, we want to solve the problem with the Machine Translation systems.
To do this, we have defined an interactive approach based on syntactic semantic patterns [
To conclude, the objectives of our experiment at iCLEF 2003 are: ² to know if it is possible that a searcher decide if a document is relevant or not only with the syntactic semantic patterns extracted automatically from the passage; ² to know if the approach based on syntactic semantic patterns is better than the approach based on passages only; ² to know if the approach based on syntactic semantic patterns is better than the approach based on a machine translation of the passage.
In the next section, we will present these two methods of interaction with the user, the method based on passages and the method based on patterns. Then we will describe briefly the experiment design and the results. Finally, we will show the conclusions and future works. 2 2.1
The first interaction method is based on passage. A passage is the most relevant pieces of text of
a document. The main idea of this approach is that it is better to show only the relevant passage
to the user, instead of the whole document. This approach was proved at iCLEF 2002 with good
results (for more information, see [
The second interaction method is based on syntactic-semantic patterns. These patterns are automatically extracted from the passage selected by the Information Retrieval system. The difference between both methods is only how to show the relevant information to the user in a language different from the one of the query: the passage in English only –method one–, or the syntactic semantic patterns extracted from these passages in English too –method two–.
From a theoretical point of view, basically, a syntactic semantic pattern is a linguistic pattern formed by three fundamental components:
2. The subcategorization frame of the sense.
For the establishment of this kind of pattern we have take into account several works about
subcategorization frame and subcategorization acquisition ([
For the automatic extraction of these patterns, we have use the syntactic parser Minipar [
For example, from this passage: “Primakov suggested that the Administration was using the Ames arrest to score domestic political points, to punish Russia for its independent stance on the conflict in Bosnia-Herzegovina and to provide convenient excuse for cutting American aid to Russia, according to journalists who attended.” the system extracts patterns like these: ² Primakov suggest Administration ² administration use Ames arrest ² administration score domestic point ² Primakov punish Russia for its stance ² Primakov provide convenient excuse for ² Primakov cut American aid to Russia according to journalist ² journalist attend
With these syntactic semantic patterns, only the most important information of each sentence is shown to the user: the most important words of each sentence –the verb and the subcategorizated nouns– and the syntactic and semantic relation between them.
Due to the searchers have not fluency nor deep knowledge about the foreign language (English in our experiment), we think that it is better not to process the sentences completely. In order to decide about the relevance of a passage, it is more easy to put the attention on the main words of the document only, that is, to put the attention on the syntactic semantic patterns only.
With this patters, to understand a text written in a foreign language completely is difficult. However, this is not our objective. Our objective is to know the topic of a text or passage and to decide if it is relevant or not. 3
We have focused our experiment in the cross-language document selection with a searcher group which has passive language abilities in the foreign language. The language of the user group is Spanish, and the foreign language is English.
The Information Retrieval system used is IR-n system, developed at the University of Alicante
[
Each retrieved document are shown to the user following the order of the experiment design. The first system shows only the passages, and the second system shows the syntactic-semantic patterns extracted from these passages.
Each searcher must decided, showing the passage or the patterns, if the document extracted is relevant or not. As we said before, passages and patterns are written in English, the foreign language. Together with the relevance judgment, we save information about time consuming for each document. Finally, we have developed an HTML interface in order to facilitate the decision of the user about the relevance of each document. 4
These results show that it is possible to decide if a text is relevant or not only with the interaction with the syntactic semantic patterns. The result obtained for each system is very similar (only a difference of 0.0179371875). The first hypothesis is correct.
The time consuming by each searcher during the decision about the relevance of a document is shown in the Table 2. Five of the eight searches consume less time with patterns than with passages. Only in one case, the time consuming with the patterns and with the passages is very similar (searcher 3). Finally, searcher 5 and searcher 6 use much more time with the patters than with the passage. However, this is an abnormal time consuming, because the different time consuming between passages and patterns is very large. Probably, due to problems during the experiment. Finally, the searcher that has obtain the better result (searcher 4) consumed more time with the passages than with the patterns.
With this data, we can conclude that the use of pattern in the interaction process improve the time consuming in the majority of cases. So the second hypothesis is, in some way, correct.
Finally, with the syntactic semantic pattern it is possible to avoid the use of Machine Translation systems. The results obtained this year at the iCLEF 2003 are better than the results obtained at iCLEF 2002, in which a Machine Translation system was used. 5
In this experiment, we have compare two method of interaction with a IR system, the first one based on passages and the second one based on syntactic semantic patterns. The results show that it is possible to decide if a document is relevant or not with the syntactic semantic patterns only. On other hand, the time consuming is less with the patterns than with the passages in the majority of cases. Finally, with these patterns it is possible to avoid the use of Machine Translation systems.
These syntactic semantic patterns are a simplification of the language: each pattern contains the main concepts and linguistic relations of a sentence. Due to this simplification, it is possible the use of these patterns in other cross-linguistic information access task as, for example, the indexation and search of documents by patterns, the alignment of pattern extracted from different languages (through the verb), or the refinement of the query with the patterns contained in the documents selected by the user. System Passages Patterns Searcher 5 1350 829 Searcher 6 1707 5046 Searcher 7 1835 7957 Searcher 8 5287 2555
We would like to thank the users (Bel´en, Raquel, Irene, Julio, Rafa, A´ ngel, Sonia and Yenori) and Rub´en, who implemented the pattern extraction system.