This paper describes the system developed by MIRACLE group to participate in the Spanish monolingual question answering task at QA@CLEF 2007. A basic subsystem, similar to our last year participation, was used separately for EFE and Wikipedia collection. Answers from the two subsystems are combined using temporal information from the questions and the collections. The system is also enhanced with a correference module that processes question series based on a few simple heuristics that constraint the structure of the dialogue. The analysis of the results show that the reuse of strategies for factoids is feasible but definitions would benefit from adaptation. Regarding questions series, our heuristics have good coverage but we should find alternatives to avoid error chaining from previous questions.
QA@CLEF 2007 has introduced two innovations over last year evaluation. The first change consists
on topic-related questions, a series of ordered questions about a common topic that simulates
the dialogue between a user and the system to obtain information related to the topic. In this
dialogue the user could introduce anaphoric expressions to refer to mentioned entities or events
that appear in previous answers or questions. The second innovation is related to the inclusion of
the November 2006 dump of the Wikipedia [
MIRACLE submitted a run for the Spanish monolingual task using a system that was enhanced
to answer questions from Wikipedia and EFE collections. Each subsystem results were combined
in a unified ranked list. The system also included a module that handles topic related questions.
It identifies the topic and use it to enhance the representation of the following questions. The
basic QA system was based on the architecture of our last year submission [
This paper is structured as follow, the next section decribes the system architecture with special attention to the new modules. Section 3 introduces the results and a preliminary analysis of the kind of errors that the system made. Conclusions and directions of future work to solve the main problems follow in Section 4. 2
The architecture of the system used this year is presented in figure 1 and it is composed of two
streams for each of the sources similar to [
The basic system follows the classic pipeline architecture used by many other QA systems [
These operations are performed during the preparation of the collection to speed up online document retrieval and answer extraction.
• Collection indexing. Collections are indexed at the word level using Lucene [
With the inclusion of topic related questions, the system needed a method to solve referential
expressions that appear between questions and answers in the same series. The system processes
the first question and generates a set of candidates with the topic, the focus and the future answer.
We have implemented a few rules that we believed covered the most common cases to select the
best topic for the whole group. The rules use the information available after question analysis
and simplified assumptions about the syntactic structure of the questions. The rules to locate the
topic for the question series are only applied to the first question:
• Answers of subtypes NUMEX (numbers and quantities) are ignored as topics for questions
series. We believe it is improbable for a number, if not representing any other type of entity,
to be a natural topic. We use the subject, the topic of this question, as the topic of the
question series.
• Answers of subtypes TIMEX (dates, years, etc...) are also ignored as topics, but in this case
it is because we believed that they fall outside the guidelines for this year. We use the same
rule than for NUMEX so far. Using a temporal expresion as a topic is very natural. In fact,
most temporally restricted questions and specially those that have an event as restrictions
are naturally split into two questions. This strategy has already been used by [
Once the topic for the group is identified, the rest of the questions use it as an additional relevant term in order to locate documents and filter relevant sentences. This is obviously a problem when the topic was the answer but the system was not able to find the right one.
These rules are based on the structure of the information seeking dialogue and how we introduce new topics in conversation. Our rules select the most promising candidate using the firt question and ignoring the rest. Nevertheless, it is posible to shift the topic by introducing a longer referential expresion like the examples mentioned for TIMEX. We plan to investigate how to modify our procedure to work in two steps, generating a ranking list of cantidates and selecting the best candidate depending on constraints expressed in the referential expression. 2.2.2
As already mentioned, this year there are two possible collections to find the right answer to a question, one is Wikipedia and the other is the EFE newswire collection for years 1994 and 1995. This means that there should be some automatic method to decide which of these sources should be more relevant to extract candidate answers to a given question. This is the role of the Source Mixer component, based on very simple heuristics: • If the verb of the question appears in present tense preference is given to answers appearing in the Wikipedia collection. • If the verb is in past tense and the question makes reference to the period covered by the EFE news collection, i.e., 1994 or 1995 years appearing in the question, then preference is given to answers from the EFE news collection.
The preference given to each answer is measured by two parameters, one referring to the verb tense factor and the other referring to the time period factor. In this way, no answer is really dropped from the candidates list but the list is reordered according to this clues. At the output of the Source Mixer component there is a list of candidates ordered according to their source and the information present in the question. 3 3.1
Using the system described above we have submitted one monolingual run for the Spanish subtask of this edition. Evaluation results and combined measures are outlined in tables 1 and 2.
Results are disappointing in general, as they are lower than previous years results for almost all types, despite the inclusion of new sources like Wikipedia. Even though almost all modules have been improved, the overall accuracy is not improving. We believe that whether the question set is more difficult or the system was not tuned as much as needed. We do not show results for the ten list questions because we did not implement an specific strategy. The case of definitions is analyzed with more detail below. We believe that question series introduce additional complexity in the task and this is reflected in the results. If we ignore question series we obtain an accuracy around 18% for the rest of the questions, which supports this idea but reflect that even base results are not good. 3.2
We are analysing our results in order to estimate which parts of the system need further improvement. For the moment, we are only able to present a preliminary analysis of the contents of our submission that uses one answer per question with their supporting sentence and document. The document returned is correct in 44% of the results, which is a lower bound to measure the performance of the document retrieval system. It also includes errors caused by an incorrect selection of a document collection. Given a correct document, 31% of the sentences selected with the first answer do not really contain a correct answer. Even if the sentence is correct, the error selecting a correct string is about 47%. These kind of error accumulates incorrect selection of a candidate, incorrect extraction and incorrect identification of the expected answer type. Finally, at least 8 questions have found an unsupported answer, which signals that in those cases where the same answer string has been extracted from several sentences, the score to select the most representative answer is not working as well as expected. A more detailed analysis isolating the contribution of the main modules will be presented in the final version of this article.
Analyzing the behaviour of the system for different types we have detected that the accuracy of definition questions has dropped dramatically. We believe that the heuristics that we have defined to extract definitions in EFE do not work for Wikipedia. The system used to signal appositions, nominal phrases before Named Entities and expansion of acronyms as valid definitions of persons and organizations. In contrast, definitional sentences in Wikipedia usually are copulative, they have a longer distance between the defined object and a valid definition and they usually are placed in the beginning of the document. Unfortunately, we did not implement any special strategy for these questions. If we consider the rest of the types, the behaviour is similar to previous years, most of the factual questions with well defined Named Entities achieve a reasonable accuracy. Questions with EAT OTHER are in contrast much harder.
There were 20 topic related groups of questions that made a total of 50 questions. The system correctly answered 5 questions of this kind, from three different groups. This behaviour was expected as errors usually chain, specially in the case when the answer to the first question is not correct and this happens to be the topic. We have analyze the main source of errors in order to evaluate the correference component. Rules for topic detection are the source of errors only for three of the cases, and therefore seems to work reasonably well although there is room for some simple improvements. In one of these cases, the error is due to a question not correctly identified as a NUMEX type. The rest of the errors are due to incorrect identification of the first answer and the chaining of errors. 4
We have presented the architecture of the MIRACLE QA system and the main modifications introduced to cope with new challenges in the CLEF@QA task. Using this system we have submitted one run for the monolingual Spanish subtask where results were lower than expected.
The analysis of the performance across types have signalled that for some type of questions
the style of the text is an important issue. This is specially accute in the case of definitions.
We have employed the same subsystem and strategies for the EFE and the Wikipedia collections
with dissapointing results. The analysis of the errors have shown that the methods for document
retrieval and candidate extraction could be adapted to improve the accuracy. We plan to use type
speficic approaches like the ones used by [
We have found that the module for correference resolution is effective even if it uses a limited amount of knowledge. In contrast, the greater contribution of errors is due to the low accuracy at the first answer. This is even more accute as a great deal of the questions that set the initial topic are definitional. Besides type specific approaches, we need to improve the correference method to consider more than one candidate answer and cope with uncertainty.
Another question that we have not evaluated thoroughly yet is the way we combine results. We
use a semantic kind of combination that exploits the different time spans of the two collections.
It is still unclear if these method is appropriate or whether techniques adapted from information
retrieval from heterogeneus collections as in [