This paper describes AliQAn, a monolingual open-domain Question Answering (QA) System developed in the Department of Language Processing and Information Systems at the University of Alicante for CLEF-2005 Spanish monolingual QA evaluation task. Our approach is based fundamentally on the use of syntactic pattern recognition in order to identify possible answers. Beside, Word Sense Disambiguation (WSD) is applied to improve the system. The results achieved (overall accuracy of 33.00%) are shown and discussed in the paper.
Question Answering (QA) is not a simple task of Information Retrieval (IR). A QA system must provide concise answers to questions stated by the user in natural language.
The research in open domain QA has mainly focused around English due to the advances in IR and Natural Language Processing (NLP). However, the Cross-Language Evaluation Forum Campaigns (CLEF) provide a multilingual forum for evaluation of QA systems in languages other than English. Multilingual open domain QA systems have been recognized as an important issue for the future of information search.
Nowadays, there are several types of implementations of Spanish QA systems. Generally, most
of the systems are based on NLP tools [
The systems based on NLP tools are complex because of the number of di®erent NLP tools that they use. Moreover, a good integration between them is needed. Our system have been developed during the last two years in the Departament of Language Processing and Information Systems at the University of Alicante. It is based on complex pattern matching using NLP tools. Beside, Word Sense Disambiguation (WSD) is applied to improve the system.
As usual, in our approach, three task have been de¯ned: question analysis, selection of relevant passages and extraction of the answer.
The rest of this paper is organized as follows: section two describes the structure and functionality of the system. Afterwards, the achieved results are shown and discussed in section three and ¯nally, section four details our conclusions and future work. 2 2.1
In this section, the structure and functionality of our approach to open domain Spanish QA system are detailed. The next paragraph presents the phases of the QA process.
Our approach is based fundamentally on syntactic analysis of the questions and the Spanish
documents (the EFE collection in this case), where the system tries to localize the answer. In
order to make the syntactic analysis, SUPAR [
Using the output of SUPAR we are going to identify three types of SB: verb phrase (VP), simple nominal phrase (NP) and simple prepositional phrase (PP). For example in the sentence: Hillary Clinton was in Jerusalen, the obtained list of SB is: [NP, hillary*clinton] [VP, to be] [PP, in: jerusalen].
The overall architecture of our system (Figure 1) is divided in two main phases: Indexation phase and Search phase.
² Indexation phase. Indexation phase consists of arranging the data where the system tries to ¯nd the answer of the questions. This process is a main step to accelerate the process. Two di®erent indexation are carried out: IR-n and QA indexation. The ¯rst one is carried out by IR-n system and it is independent from the second one, in which more syntactic and semantic information is stored. For example, the QA indexation stores the NP, VP and PP obtained from the parsing, and it also stores the results of the WSD process. ² Search phase. This phase follows the most commonly used schema. The three main modules of our approach are:
These modules are described below. Previously, the used annotation is commented. The symbols \[ ]" delimit a SB (NP, VP and PP), \sp" is a preposition of a PP, the term \ap" indicates that PP is an apposition of the previous nominal head, SOL is the place where the answer can be found and the symbols \[: : :]" indicate some irrelevant SB for the search. 2.2
In this step the system carries out two tasks: ² To detect the type of information that the answer has to satisfy to be a candidate of answer (proper name, quantity, date : : :). ² To select the question terms (keywords) that make possible to locate those documents that can contain the answer.
We have based on WordNet Based-Types and EuroWordNet Top-Concepts in order to develop our taxonomy that consists of the next categories: person, group, object, place, place city, place capital, place country, abbreviation, event, numerical quantity, numerical economic, numerical age, numerical measure, numerical period, numerical percentage, temporary year, temporary month, temporary date and de¯nition.
The expected answer type is achieved using a set of syntactic patterns. The question posed to the system is compared with all the patterns of all the categories. For each category a score is assigned that measures its probability of being the correct type. We choose the category having the highest probability.
We have 173 syntactic patterns for the determination of the di®erent semantic category of our ontology. The system compares the SB of the patterns with the SB of the question, the result of the comparison determines the category of the question.
The next example shows the behavior of question analysis: ² Question: ² Syntactic Block ¡ [IP qui¶en](who) ¡ [VP ser ](to be) { Qui¶en es el Secretario General de la ONU? (Who is the General Secretary of the
ONU?) ¡ [NP secretario general [PP, de: onu]] (General Secretary of the ONU)
We have a pronoun or interrogative particle qui¶en (who) followed by two syntactic blocks: a verb phrase and a nominal phrase. This example matches with the next pattern: [IP, qui¶en j qui¶enes] (who) [VP, ser ] (to be) [NP, hipo¶nimo persona] (hyponim person) therefore, the category of the question is person.
For each SB of the pattern, we keep a °ag in order to determine whether the SB of the question is considered for the next stage of the QA process or not. 2.3
This second module of the QA process creates and retrieves passages using IR-n system [
The inputs of IR-n are the detected keywords in question analysis, IR-n returns a list of passages where we apply the extraction of the answer process. Beside, the objective of this task is reducing complexity of the process of searching the solution by means of reducing the amount of text in which the system searches for the answer. 2.4
The ¯nal step of QA is the extraction of the answer. In this module, the system takes the set of retrieved passages by IR-n and tries to extract a concise answer to the question.
Moreover, the type of question, SB of the question and a set of syntactic patterns with lexical, syntactic and semantic information are used in order to ¯nd a possible answer.
As shown in the next list, the system use the following NLP techniques. ² Lexical level. Grammatical category of answer must be checked according to the type of the question.
For example, if we are searching for a person, the proposed SB as possible answer has to contain at least a noun. ² Syntactic level. Syntactic patterns have been de¯ned. Those let us to look for the answer inside the recovered passages. ² Semantic level. Semantic restrictions must be checked. For example, if the type of the question is city the possible answer must contain a hyponym of city in EuroWordNet. Semantic restrictions are applied according to the type of the questions. Some types are not associated with semantic restrictions, such as quantity.
In order to design and group the patterns in several sets, the cases of the question are used. The patterns are classi¯ed in the followings three cases: ² Case 1. In the question, one SB of type NP or PP is only detected. For example: { Question: Who is the president of Yugoslavia?
We only have a SB, the verb to be that is not used to ¯nd the answer because it is a copulative verb.
{ SB: [NP, president [PP, of: Yugoslavia]] ² Case 2. A VP is detected in the question. This verb expresses an action that must be used in order to search the answer. For example: { Question: Who did write Star Trek? { SB: [VP, to write] [NP, star¤trek] { Question: Which team did win the NBA tournament? { SB: [NP, team] [VP, to win] [NP, NBA¤tournament] ² Case 3. VP is preceded by a NP or PP. In this case we used three sections to ¯nd out the possible answer.
When the system tries to ¯nd a possible answer in a sentence, ¯rst, the SB of the question are localized in the text, secondly the system attempts to match the pattern in the sentence. If this has been possible, then a possible answer has been founded that must be appraised using lexical and semantic restrictions according to the type of the question. Spanish QA system has about 60 patterns, the number of patterns that is processed in each sentence depends on the type of the question. Therefore, a question of case 1 and type \person" processes di®erent patterns than a question of case 1 and type \place city " .
The next example shows the used pattern and the behavior the extraction of the answer: ² [SOL[PP, sp: NP1]] [: : :] [VP][: : :] [NP2]
First, NP2 (or PP2) and VP are searched by the system, afterward the NP1 with the answer must be found. Next example shows the process: ² Question: >Qu¶e presidente de Corea del Norte murio¶ a los 80 an~os de edad? (What North Korea's president died at the age of 80? ) ² Type: person ² List of SB: [NP, north¤korea¤president] [VP, to death] [PP, at: age [PP, of: 80] ² Text: [: : :] Kim Il Sung, presidente de Corea del Norte, murio¶ ayer a los 82 an~os [: : :] ([: : :] Kim Il
Sung, president of North Korea, died yesterday at the age of 80 [: : :]) ² List of SB of sentence: [: : :] [NP, kim¤il¤ sung [PP, apposition: president [PP, of: north¤korea]]] [VP, to death] [PP, at: age [PP, of: 80] [: : :] ² Answer: Kim Il Sung 2.4.1
Value of the solution In order to select the answer from a set of candidates, each possible answer is scored. The calculation of the value of the solution on each pattern is described in this subsection of the paper.
The score of a candidate is structured in three phases: comparison of the terms inside a nominal head of a SB with the terms of the nominal head of another SB, comparison of a SB of the question with a SB of the text and weighting of a pattern according to the di®erent SB. Comparison of the terms of a nominal head. When the system is comparing two terms, the system does not only contemplate the literal value of terms, also checks the relations between these terms in EuroWordNet. So, weighting of terms is calculated using the equation 1, where N is the number of terms inside nominal head and pti is the value of the terms that is calculated using EuroWordNet (1 same lemma, 0.8 synonym and 0.6 hyponim ).
Comparison of the SB. In our approach, the comparison of the SB occurs in two kinds of circumstances. When the SB of the question is localized in the text in order to apply a pattern and when the system is analizing a SB to ¯nd the answer.
The ¯rst type of comparison is called \value of terms ", this measure can be a®ected by ¯xed circumstances, such as: ² Depth of appearance. The terms of the SB of the question may not appear as nominal heads in a
SB of the text. ² Excess or missing of modi¯ers. If the nominal head of the question has more or less modi¯ers its value is penalized. ² Appearance of terms, but some complements are missing. When the system detects only the term of the question in the text, them it continues the searching until it is able to ¯nd the complements.
Second type of comparison of SB is the calculation of the value of solution, this value is calculated when it is searching for a possible answer. It takes into account a set of evaluation rules according to the type of the question, such as: ² Lexical restrictions. Grammatical category of the answer depends on the type of the question. For example, a question of type \persona (person)" the answer must have at least a proper noun or common noun. ² Semantic restrictions. The system leaks the answer according to semantic relations such as hyponimy. For example, a question of type \ciudad (city)" the answer must be a hyponim of \ciudad (city)" in EuroWordNet. ² Ad-hoc restrictions. An example of this kind of restrictions is founded in the questions of type \fecha (date)", when the system penalizes the value of solution if the answer does not contain day, month and year.
Comparison of the patterns. When the system is evaluating a pattern in the text, a set of circumstances are considerate in order to provide the value of solution. The total value of an answer is de¯ned by the equation 3, where N is the number of retrieved SB of the question, vti is the value of terms of each SB, d is the distance between the localized SB in the text and vs is the value of solution. As shown in the equation 3, vs is 30% of total and the remaining ones is the 70%.
V r = (
PN i=1 vti ¡ d ¤ 0:1) ¤ 0:7 + vs ¤ 0:3 N (3) Final evaluation of patterns. The system generates a list of candidate solutions, where each solution has been obtained in a passage. If two solutions have the same value for a question, the system chooses one considering the proposed order by IR-n.
Spanish QA system must determine when a question has answer or not. In order to do that we suggest an umbral that indicates if an answer is solution or not. A question has answer if its V r is higher than 0:5.
Next, an example (question 114, In Workshop of Cross-Language Evaluation Forum (CLEF 2003)) of resolution of one question, where system chooses the correct solution since the V r is higher than 0:5.
² Question: A qu¶e primer ministro abri¶o la Fiscal¶³a de Mil¶an un sumario por corrupci¶on? ( To whom prime minister the O±ce of the public prosecutor of Milan opened a summary for corruption?) ² Type: person 0 ¤ 0:1) ¤ 0:7 + 1(Silvio ¡ Berlusconi) ¤ 0:3 = 0:93 ² Text where is a correct solution: \[: : :] la Fiscal¶³a de Mil¶an abri¶o, hoy martes, un sumario al primer ministro, Silvio Berslusconi, por un supuesto delito de corrupci¶on [: : :]" ² Value of the solution:
V r = ( 1([NP;fiscalia[P P;de:milan]])+0:65([NP;sumario[P P;por:corrupcion]])+1([NP;primer¤ministro])+1([V P;abrir]) 4 ¡ This section describes some tables related with the results and the evaluation of our system in CLEF-2005. The proposed system was applied to the set of 200 questions, all of them was supported by our system.
For the development of our system we used as training set the questions developed for CLEF2003 and CLEF-2004 questions.
During this test process many faults were detected in the tools used in the lexical and morphological phases. The analysis of question 145 of CLEF-2003 shows one of these errors: ² Qui¶en es el ministro de econom¶³a alem¶an? (Who's the German Minister of Finance?) The term Alem¶an is not in the prepositional phrase where the term econom¶³a is, because of econom¶³a is tagged as feminine and aleman is tagged as masculine. So, when searching for SB in the corpus to ¯nd an answer for the questions, it gives wrong answers.
We submitted two runs. The ¯rst run was obtained applying the system after repairing the lexical and morphological errors that we have detected (alia051eses) while the second run (alia52eses) performed QA process without repairing theses faults. Table 1 shows the results for each run and how theses errors lowered our system performance giving wrong answers.
Inexact answers also lower the system performance and in our system, these are due to errors in parsing process. An answer was judged inexact when the answer string contained more or less than just the correct answer, ie. the system ¯nds this answer in the text but it does not extract the part of the information needed to return it as an answer. Our system returned 24 inexact answers (see Table 1). We may obtain a higher level of performance (45%) if we take into account that these inexact answers include the expected answer.
Finally, Table 2 shows that the accuracy over temporal questions was 34.38%, ie. we have obtained 11 right answers over 32. This is considered a good score because no special mechanism was developed. 4
For our ¯rst participation in the QA@CLEF track, we proposed a QA system designed to search Spanish documents in response to Spanish queries. To do so we used a Spanish syntactic analyzer in order to assist in identifying the expected answers and the solution of the question.
All questions given in this track have been supported by our system. The results showed overall accuracy levels of 33%.
As previously mentioned, the used tools reduce exactitude to our system (alia52eses). These are encountering results that show the potential of the proposed approach, taking into account that the use of patterns is a less expensive recourse compared with other proposals.
Ongoing work on the system is focused on multilingual task, temporal question treatment and the incorporation of knowledge to those phases that can be useful to increase the our system performance. 5
This research has been partially funded by the Spanish Government under project CICyT number TIC2003-07158-C04-01 and by the Valencia Government under project number GV04B-268.