This paper describes the novelties introduced in the Question Answering system developed in the Natural Language Processing and Information Systems Group at the University of Alicante for QA@CLEF 2005 campaign with respect to our previous participations. Thinking of future developments, this year we have designed a modular framework based on XML that will easily let us integrate, combine and test system components based on di®erent approaches. In this context, several modi¯cations have been done. Mainly, a new machine learning based question classi¯cation module has been added and tested, the document retrieval engine has been changed and several new heuristics for answer extraction have been applied. We took part in the monolingual Spanish task.
Most of the Question Answering systems are based on pipeline architecture comprising three main stages: question analysis, document retrieval and answer extraction. These three tasks can be isolated in di®erent modules, so that the development of each one could be set apart and afterward integrated as a whole system. In order to achieve this goal, we have developed an XML framework that facilitates the communication between the di®erent components of the system, so that we can easily substitute and test new modules into the general framework for further development.
Besides that, the system has su®ered several modi¯cations with respect to previous
competitions [
This paper is organized as follows: in section 2 we describe the system architecture detailing the novelties included this year; section 3 outlines the XML framework that allows the communication between modules; section 4 presents and analyses the results obtained at QA@CLEF 2005 Spanish monolingual task, paying special attention at the question classi¯cation module performance; ¯nally, in section 5 we discuss the main challenges for future work. 2
Our approach has evolved from the system developed in our research group [
The system follows the classical three-stages pipeline architecture mentioned above. In the question analysis stage, queries proposed to the system are analyzed and useful information for next modules is extracted, like keywords or question type. Document retrieval module takes the keywords extracted from the query in the previous stage and returns a set of relevant documents related to these terms. The documents obtained are the input for the ¯nal step, the answer extraction stage, which processes these texts in order to locate and present the ¯nal answer. Figure 1 shows the system architecture. Next paragraphs describe each module in detail.
Question
Query Analysis
Question Keyword Classification Extraction
Document Retrieval Xapian
Google Answer Extraction
XML file <QAN>
… </QAN> <DR> … </DR> <AE> … </AE>
Answers This stage carries out two processes: question classi¯cation and keyword selection. The ¯rst one detects the sort of information claimed by the query, mapping the question into a previously de¯ned taxonomy. Otherwise, keyword selection chooses meaningful terms from the query that helps to locate the documents that are likely to contain the answer.
This year we have replaced the former question classi¯cation module, based on hand made
lexical patterns, with a new one based on machine learning [
On the other hand, the keyword extraction module remains the same for this year competition
[
To accomplish this task we use two di®erent search engines: Xapian and Google3. Xapian performs document retrieval over the entire EFE Spanish document collection. The lemmas of the keywords detected in the question analysis stage are used to retrieve the 50 topmost relevant documents from the EFE collection.
In parallel, the same keyword list (not lemmatized this time) is sent to Google search engine through its Web API4, selecting the 50 top ranked short summaries returned. We store this information for later use as a statistical indicator of answer correctness.
As a novelty, we introduced last year the use of English search to improve the retrieval task. The original Spanish question is translated into English via SysTran5 online translation service. Keywords are extracted and sent to Google, selecting again the 50 top ranked short English summaries returned that are later used to weight possible answers in the extraction module. This special search is only performed if the question is mapped to type NUMBER, DATE, PERSON or ORGANIZATION, the classes that are likely to have a language independent answer: numbers, dates, people and company names tend to keep unchanged through languages. 2.3
In this stage a single answer is selected from the list of relevant documents retrieved from the EFE Spanish corpus. At this point we have the following information: keywords and de¯nition terms from the query, the set of relevant Spanish documents retrieved from the EFE corpus, the set of relevant summaries in Spanish retrieved from Google and, depending on the question class, a set of summaries in English also retrieved from the Web. The system uses all this information to select the correct answer. The set of possible answers is formed extracting all the n-grams (unigrams, bigrams and trigrams in our experiments) from the relevant documents in the EFE collection.
Although the general process is similar to the one we used in previous competitions and already
explained in detail in [
The following ¯lters are applied to the set of possible answers so that we can narrow down the solution space: ² Part of speech and query class: depending on the query class, we reject the answers that do not contain a proper part of speech. For instance, if the query class is PERSON we expect the answer to be a proper noun, not an adjective or a verb. ² Query terms: we reject all the possible answers that contain any keyword or that only contain de¯nition terms. 2Text REtrieval Conference, http://trec.nist.gov 3http://www.google.com 4http://www.google.com/api 5http://www.systransoft.com ² Stopwords: all the answers that start or end with a stopword are rejected.
Once the ¯ltering process is done, remaining candidate answers are scored taking into account the following information: ² The sentence where the answer appears is scored depending on the number of keywords and de¯nition terms that co-occur. We refer to this weighting value in the formula below as ws. ² The frequency (wf ) of the answer in the documents and summaries obtained in the retrieval stage. Here we have three di®erent values: the frequency in the documents retrieved by Xapian, the frequency in the Spanish web summaries and, depending on the query type, the frequency in the English web summaries. The value wf is computed like this: wf = log( ddooccFMraexc ) + log( ggooooFMraexc ) + log( ggooooEEnnFMraexc )
n where numerators inside logarithm functions represent the word frequency in the Spanish corpus, in the Spanish web summaries and in the English web summaries respectively. Denominators represent the frequency of the most frequent word in the corresponding collections, being used to normalize the ¯nal values. The global frequency wf is normalized through n, that represents the number of IR processes taking place: 2 or 3, depending on whether we use Google English summaries or not. ² The distance (wd) or number of words between the possible answer and the keywords and de¯nition terms co-occurring in the same sentence. ² The size (number of words) of the answer (wn). For instance, DEFINITION questions are more likely to have long answers, while NUMBER answers tend to be shorter.
All the weights obtained are normalized in order to obtain a ¯nal value between 0 and 1, determining the con¯dence of the system in the answer selected. The ¯nal answer score (as) is computed as follows: (1) (2) as = 0:2 ¢ ws + 0:3 ¢ wf + 0:2 ¢ wd + 0:3 ¢ wn 3
Once detailed the di®erent stages of the Question Answering system, we are describing the XML framework where all the process takes place. The eXtensible Markup Language (XML) is a generalpurpose markup language that has become a standard de facto in inter-system communication, being widely used to facilitate data sharing between applications. We have used it to exchange information between the di®erent modules in our system, building a framework were individual components can be easily interchanged. Thus, new modules can be developed separately and later used in place of old ones in the framework for testing purpose. In order to change a module, we only have to make sure that it ¯ts de XML speci¯cation for that process.
We have associated an XML tagset for each stage of the process. Every module adds the XML fragment generated to a common ¯le where the following modules can extract the information required to perform. So, what we ¯nally get is a sort of log ¯le that stores the complete question answering process in XML format. This ¯le can be used to save time testing individual modules, as we have the information needed already stored in the ¯le. For instance, if we just want to test the answer extraction module, we wouldn't need to execute the previous processes as the information might be already stored in the ¯le because of a previous run.
Although our run was limited to Spanish monolingual task, the framework is prepared to store information in di®erent languages together for multilingual purpose.
In the question analysis stage, we must store information about query terms and question type detected. For instance, the question \>En qu¶e provincia est¶a Atapuerca?" (\Which province is Atapuerca in?") generates the following XML fragment: <QAN> <CLASS type="LOCATION"/> <TERMS lang="en"> <TERM type="DT" wf="Which" lemma="which" pos="WDT"/> <TERM type="DT" wf="province" lemma="province" pos="NN"/> <TERM type="SW" wf="is" lemma="be" pos="VBZ"/> <TERM type="KW" wf="Atapuerca" lemma="Atapuerca" pos="NP"/> <TERM type="SW" wf="in" lemma="in" pos="IN"/> </TERMS> <TERMS lang="sp"> <TERM type="SW" wf="En" lemma="en" pos="SPS00"/> <TERM type="DT" wf="qu¶e" lemma="qu¶e" pos="DT0CN0"/> <TERM type="DT" wf="provincia" lemma="provincia" pos="NCFS000"/> <TERM type="SW" wf="est¶a" lemma="estar" pos="VMIP3S0"/> <TERM type="KW" wf="Atapuerca" lemma="Atapuerca" pos="NP00000"/> </TERMS> </QAN>
QAN tag indicates the kind of process taking place (question analysis). CLASS tag indicates the query type detected by the question classi¯cation module. TERMS stores the information on question terms, telling us if they are keywords (KW), stopwords (SW) or de¯nition terms (DT). It also stores the word itself, the lemma and the part of speech detected by Freeling. The attribute lang indicates the language of the query, so that we can store multilingual information for every question.
This snippet is included in the common XML ¯le and passed to the document retrieval module, which reads the information required in that process. As a result we obtain the relevant documents retrieved by the search engine (in this this case Xapian and Google), which are stored as an XML fragment in the common ¯le. In the example above: DR tag stands for the name of the process (document retrieval). Each ENGINE tag indicates the name of the search engine and the language of the documents returned. Inside this tag the di®erent documents are separated in DOC tags, and these are subdivided in PARAGRAPH tags, that evidently represent paragraphs.
Finally, the answer extraction module gets information from both QAN and DR fragments to select the candidate answers. Therefore another XML snippet is generated and added to the common ¯le. In the example above: <AE> <ANSWER lang="sp" doc="EFE19950620-13256" cert="0.895063">Burgos</ANSWER> <ANSWER lang="sp" doc="EFE19950907-04306" cert="0.432019">Le¶on</ANSWER> <ANSWER lang="sp" doc="EFE19950907-04306" cert="0.311245">Granada</ANSWER> ... </AE>
AE tag represents the kind of process (answer extraction). Each ANSWER tag stores a possible answer, indicating its language, the document where it was found and the con¯dence of the system.
Another bene¯t of this XML framework is that additional tags could be added on demand if extra information storing is required for new modules, having not to change the old modules performance as the original structure remains the same. 4
This year we submitted one run for the Spanish monolingual task. Table 1 shows the overall and detailed results obtained for each CLEF question type.
These results are very similar to the ones obtained in last year competition [
Table 3 shows the detailed performance of the question classi¯cation process, but not the impact in the whole Question Answering system. DEFINITION questions are not included in this comparison as this type of query was identi¯ed by CLEF organization before question answering process took place. Thus we present the results for the classi¯cation of the 150 factoid questions in the set. Table 2 reveals that almost 77% of the factoid questions were correctly classi¯ed (up to 82.5% if we also consider DEFINITION questions), quite promising for a system trained on surface text features aiming to reach language independence. In this paper we have described the novelties introduced in our Question Answering system for QA@CLEF 2005 competition. Mainly, a new XML framework has been introduced laying the foundations for future developments. In this framework we can easily introduce new modules and substitute old ones for testing purpose. This year we have introduced a new question classi¯cation module that can be trained with di®erent languages, proving to be as competitive as other state-ofthe-art systems. We also introduced a new IR engine that can be easily adapted to new languages.
Therefore, the main goal is to continue the gradual development and integration of new multilingual modules in order to have a system that can deal with many di®erent languages at the same time. To sum up, this can be considered the ¯rst step of a full multilingual framework for QA. 6
This work has been developed in the framework of the project CICYT R2D2 (TIC2003-07158-C04) and it has been partially funded by the Spanish Government through the grant BES-2004-3935.