The iCLEF 2004 experiment at the University of Alicante has focused on how to assist users in the localization of the correct answer in passages written in a language different from the one of the query. The language of the users is Spanish and the language of the documents/passages English. In order to help users, a first system shows, together with the passage in English, the relevant domains of the passage and the relevant domains of the query. These relevant domains were extracted automatically from WordNet Domains. A second system shows, together with the passage in English, the syntactic-semantic patterns (SSP) of each passage and the SSP of the query. The SSP are formed by the verb and the main nouns of a sentence (that is, the head nouns of the main complements). For users without deep knowledge or with low competence in English, our hypothesis is that to know the relevant domain and/or the SSP will be useful in order to find the correct answer in the passage. The results show that the SSP are few more better in the interaction with the users. However, some users said that it is more easy to find the answer knowing the relevant domains than through the SSP.
The iCLEF 2004 experiment at the University of Alicante has focused on how to assist users in the localization of the correct answer in passages written in a language different from the one of the query. To achieve this objective, we have thought in two important questions: 1. What information must be shown to the user: It must be enough for the efficient localization of the correct answer. The user do not know the correct answer previously. He/she must infer the correctness of the answer from the context where it appears. So it is important to show, not only the correct answer, but enough context that clearly shows that a possible answer is the correct one (or not). 2. How the information is shown to the user: Specifically, in what language is shown the information to the user. If users are not mastered in the language of the passage, it is necessary to help them in order to identify the correct answer.
In this experiment we have focused on the assistance users when they have
low fluency or no linguistic competence in the language of the passages. This is
the most common case for Spanish people with English language. Most of them
know English, but it is very common that they can not formulate a correct query
or understand correctly a possible answers. On other hand, we are looking for
alternative method to deal with large multilingual collection of documents, but
avoiding the use of Machine Translations systems (due to the computational cost
of the machine translations of the collection completely) [
As we said before, the objective of the experiment is how to assist users in the localization of the correct answer. For this propose, the experiment has followed the next steps:
We have taken the queries in Spanish, and they have been translated with a machine translation system to English.
For the localization of the relevant passages in the collection of English
documents, we have used an Information Retrieval system: IR-n system [
The queries (in Spanish) and the passages (in English) are shown users through a web page. The users check the passage of each query until to find a passage with a (possible) correct answer. Then they select the answer (the string of characters) and the passage where it appears, and check the next query.
The problem is the language: as we said before, the users do not have deep knowledge of English. They need assistance for the correct localization of the passage and the answer. Ruling out machine translation, two interaction methods have been used in this task. The first one is based on relevant domains: the system shows user the passage in English and the relevant domains of the passage and the query. Our hypothesis is that with the relevant domains, the user can decide previously if a correct answer is contained in a passage or not. The second method is based on syntactic semantic patterns (SSP): the system shows user the passage in English and the SSP of each passage, formed by the main verbs and the main nouns of the passage (that is, the verbs and their subcategorization frame). Our hypothesis is that knowing the SSP, the user can decide if the passage contains the correct answer and, finally, he/she can locate it. In the next section, both methods will be explained deeply.
Figure 1 and Figure 2 are the web page used in the experiment. They show the query, the passage and the relevant domains of the query and the passage (Figure 1); or the query, the passage and the SSP of the passage (Figure 2). If the answer is this passage, the user selects it, and the system stores the answer, the passage and the time spent. If the answer is not in this passage, the user checks the next passage up to the last one: 50 passages have been extracted from each query.
The fist method uses relevant domains to assist the localization of the correct
answer. The relevant domains of a word are the more relevant and representative
ontological domain of this word. They are extracted from WordNet Domains
(WND) [
As we said before, in this interaction method, the system shows user the passage in English and the relevant domains of the passage and the query. Theoretically, the relevant domains of both must agree: the passage with the correct answer must contain the same relevant domains (or very similar relevant domains) than the query. So if users know previously the relevant domains of each one, they can decide previously if the answer will be contained in the passage or not.
WordNet Domains [
We extract the relevant domains of the query and the relevant domains of each passages. These relevant domains are extracted through a vector context in which the relevant domains of the words of the query/passage are represented. From this, we take only the common relevant domains to specify the relevant domains of the whole query/passage. With this, the relevant domains of the passage or query are the common relevant domain to most of the words.
Furthermore, the passages order has been recalculated according to the similarity between the relevant domains of the the query and the relevant domains of the passage. So the system shows first the passage with high similarity between its relevant domains and the relevant domains of query, and at the end the passage with low similarity.
The second method is based on syntactic semantic patterns. As we said before, with this method the system shows user the passages in English and the SSP of each passage, formed by the main verbs and the main nouns (that is, the verbs ant their subcategorization frame). Our hypothesis is that knowing this information, the user can decide if the passage contains the correct answer and locate it. The intuitive idea is that, when the user is looking for an answer in a text, he/she looks at the main nouns and verbs, trying to locate the same or similar nouns/verbs than in the query. With the SSP, the main nouns and verbs have been previously extracted, so maybe they facilitate the task.
From a theoretical point of view, a syntactic semantic pattern is a linguistic
pattern formed by three fundamental components [
2. The subcategorization frame of the sense. 3. The selectional preferences of each argument.
However, this theoretical SSP is difficult to process automatically: it is difficult to extract patterns like these and to use them in iCL-QA. From this model of SSP, we have developed a new model more easy to deal with from a computational point of view. In this “lite” model, the verb is represented by the word and its sense (or senses) represented in EuroWordNet; the subcategorization frame is represented by the head noun of each arguments1; and finally the selectional preferences of each argument are represented by the sense or senses of the head nouns.
With these syntactic semantic patterns, only the most important information of each sentence is shown to the user: the most important words of each sentence –the verb and the subcategorizated nouns– and the syntactic and semantic relation between them. Due to users have not fluency nor deep knowledge about the foreign language (English in our experiment), we think that it is better not 1 If the argument is a clause, the head will be a verb, not a noun. These verbs are, at the same time, a new SSP. to process the sentences completely looking for a possible answer. In order to decide if a passage could contain a correct answer, to know the main words of the document only (that is, the syntactic semantic patterns) will facilitate this task. With this patters, to understand completely a text written in a foreign language is difficult. However, this is not our objective. Our objective is to find a specific answer for a specific question: first, to decide if the answer is contained in the passage, and then look for it and find it. 5
the answer (“strict”). In this table, four users locate the correct answer with the correct passage with the SSP method, two users achieve the same results with both methods, and two users achieve better results with the method based on relevant domain.
The second table (Figure 5) shows the correct answers located by each user, independently of the correctness of the passage (“lenient”). In this cases, five users have obtained better results with the interaction method based on SSP, one the same results with both methods, and two users have obtained better results with relevant domain method. 5.3
The results of the questionnaire (that users complete during the experiment) do not indicate preference for any method: five users said that there are no differences between both interaction methods; two users prefer SSP method, and one relevant domain method.
About the really help of each method (according with the personal opinion of the users), most of them do not prefer one method or other. One user clearly prefers relevant domains method and other user clearly prefers SSP method. However, some users said that the really help of the system in the localization of the answer is low with both systems. Finally, the Figure 6 shows the time consuming by each user. The time consuming with both interactive method is similar. Two users spent more time with SSP method, and the other six with relevant domains method. 6
– The results are low, maybe because we have not used any kind of translation.
In this sense, it is necessary some kind of translation (at least, superficial translation) to really help the localization of the answer. – The order in which the passage are shown to the user in the SSP method (the output order of the IR-n system) seems to be correct. – The order of passage based on the similarity of relevant domains do not seem the most correct one. – According to the results and the personal opinion of the users, relevant domains method really help the localization of the answer. However, an error in the extraction of the relevant domain will confuse users. For these cases, it is necessary to improve the extraction of relevant domains.
– About the syntactic semantic patterns, we are working now in a method to translate the patterns from one language to another based on the alignment of the verbs. – About the relevant domains, we are improving their automatic extraction.
The idea is to improve the Information Retrieval system with the information about the relevant domains of the passages. 7
We want thanks all users for they work and opinions in the development of the experiment.