This paper summarizes the participation of the MIRACLE team in the Multilingual Question Answering Track at CLEF 2009. In this campaign, we took part in the monolingual Spanish task at ResPubliQA@CLEF 2009 and submitted two runs. We have adapted our QA system which has been evaluated in EFE and Wikipedia to the new JRC-Acquis collection and the legal domain. We tested the use of answer filtering and ranking techniques to a base system using passage retrieval with no success. Our run using question analysis and passage retrieval obtained a global accuracy of 0.33 while the addition of an answer filtering step obtained 0.29. We provide an initial analysis of the results across the different questions types while we research the reason why it is difficult to leverage previous QA techniques. A different focus of our work has been on temporal reasoning applied to question answering and also detailed discussion of this issue in the new collection and analysis of the questions is provided.
The rest of the paper is structured as follows, the second section describes the system architecture with special attention paid to the novelties introduced this year, Section 3 introduces the submitted runs and analyzes the results. Finally, conclusions and future work are presented in Section 4. 2
The system architecture is similar to our previous system [
Document Collection
Question Analysis
Linguistic Analysis Timex Analyzer
Question Classification
Offline Operations
Collection Indexer Timex Analyzer Linguistic Analysis
Information Retrieval
Query Generation Information Retrieval
Document
Index
Answer Selection
Answer Filter Passage Fallback Strategy
Answe Indexes are really important for QA as obtaining a good retrieval subsystem can considerably improve the final results of the QA system. Due to the change in the document collection, all IR indexes have been newly created using Lucene as the retrieval engine. To accomplish the task of storing the relevant information as appropriately as needed, we have designed two different document types or indexing units: • Document, where all the information related to title, note and the text of a collection file is stored. • Paragraph, which store each paragraph, the title and the notes in a different document. Lucene uses a length document normalization term in the retrieval score which was arguably of no help in the case of paragraph scoring because paragraphs are expected to have more uniform lengths. Both types of indexes, with length normalization and without were tested.
In all our experiments previous to the submission the paragraph or passage index worked better than the document index. Besides, we also created different index types regarding the analysis, characterized by the linguistic analyzer used in each case: • •
Simple Index, where the text analyzer used is a simple analyzer adapted for Spanish. It makes grammar based parsing, stems words using a snowball-generated stemmer, removes stop words, replaces accented characters in the ISO Latin 1 character set and converts text into lower case. All the texts are stored in the same field: text.
Temporal Index, which adds a recognition and normalization of time expressions. These time expressions are normalized and included in the index. Texts are also stored also in the field text. Finally, other modifications required the query generation process to be changed to use the same analyzer that was used to create the index.
The idea of a rule engine, was initially considered for classifying question types; later, it has also been used not
only in the Question Classification module, but also in the Answer Filter, Timex Analyzer and Topic Detection
ones [
EXISTENTIAL QUESTION TYPE ("DEFINITION") AND WORD_I(N, OBTAIN_FOCUS()) AND (WORD_I(N+1, ":") OR
WORD_I(N+1, "\"") AND WORD_I(N-1, "\"") OR WORD_I(N+1, "\"") AND WORD_I(N+2, ":") AND
WOD_I(N-1, "\"")) THEN
ANSWER_EXTRACTION(0,POS_LAST_TOKEN()); This rule has been created to detect the topic in definition questions. In most of them, the topic in the answer paragraph was written in quotation marks and/or followed by colon. This rule locates the topic of the question and looks for it in the source documents.
Some authors have defined the temporal question answering (TQA) as the specialization of the QA task in which
questions have some features that denote temporality [
Question analysis, indexes generation and answer selection modules have been considered potentially more influential for achieving better results by means of the application of temporal management. They have been slightly adapted to the requirements of this year’s competition, keeping the essence of their functionality. z During question analysis process, queries, including those with temporal features, are classified, distinguishing between TR and TA queries. If a TA query is detected, it determines the granularity of the expected answer (complete date, only year, month, etc.). z
The answer selector is involved in two directions: in the case of TA queries, the module must favour a temporal answer, whereas if it manages TR queries, it applies extraction rules based on the temporal inference mechanism and demotes the candidates not fulfilling the temporal restrictions.
As a novelty, this year we have created more sophisticated indexes according to the paragraph retrieval approach of the competition. In some configurations, the normalized resolution of temporal expressions is included in the index instead of the expression itself. The main objective is to assess the behaviour of the QA system using different index configurations, mainly focusing on the temporal queries of the collection.
Due to the nature of the collection, a large number of questions were expected to be expansion of acronyms, especially about organizations. On the other hand, the recall of the information retrieval step could be improved by including the acronym and their expansion in the query.
We implemented a simple offline procedure to mine acronyms by scanning the collection and searching for a pattern which introduces a new entity and provides their acronym between parentheses. Then, results are filtered in order to increase their precision. First, only those associations that occur at least twice in the corpus are considered. As parentheses often convey other relations like persons and their country of origin, another filter removed countries (Spain) and their acronyms (ES) from the list. Finally, some few frequent mistakes were manually removed and acronyms with more than one expansion were also checked.
Once we have cleaned the file, we index the acronyms and their expansions separately to be able to search by acronym or by expansion.
The index is used in two different places in the QA system: • Query Generation, where it analyzes the question and adds searching terms to the query that is sent to the document collection index. • Answer Filter, where it analyzes the text extracted from the paragraph to determine if that paragraph contains the acronym (or the expansion) and if so, identifies the paragraph as correct answer.
This module, previously called Answer Extractor, process the result list from the information retrieval module and selected chunks to form a possible candidate answer. In previous years, this module was designed to extract answers selected from the document. In this campaign, the answer must be the complete text of a paragraph therefore, this year the module works as a filter which removes passages with no answers. The kind of linguistic rules used last year to perform answer extraction has been adapted and new rules to detect acronyms, definitions as expressed in the new corpora and new rules for temporal questions have been developed. The possibility of getting no answer from the answer filter led to the development of a module that simply creates answers from the retrieved documents. This module is called Passage Fallback Strategy. It takes the documents returned by the information retrieval module and generates an answer from every document. The way of generating the indexes (concretely the paragraph index) makes possible the functionality of this module.
Evaluation is a paramount part of the development process of the QA system. In order to develop and test the system the English development test provided by CLEF organizers was translated to Spanish and a small goldstandard with answers was developed. Mean Reciprocal Rank (MRR) and Confidence Weighted Score (CWS) were consistently used to compare the outputs of the different configurations with the development gold standard. Periodically, the output and the XML logs of different executions were manually inspected to complete the gold standard and to detect integration problems. 3
We submitted two runs for the monolingual Spanish task. They correspond to the configurations of the system that yielded best results during our development using the translated question set. Paradoxically, both runs match with the simplest configurations that we have tested.
• Baseline (mira091eses): The system is based on passage retrieval using the simple index. Question analysis is performed to generate queries and the acronym expansion is used. • Baseline + Answer Filter (mira092eses): Adds answer filtering and the passage fallback strategy after the previous passage retrieval.
A number of additional configurations were also tested but no improvements over the baseline were found consistently. In fact, most of the additions seem to produce worse results on our development test. We considered different functions for Answer Ranking and Passage Re-ranking which we have tested for previous participations and some new ones. Different passage length normalization strategies were also applied to the indexes. Finally, a great deal of effort was devoted to the treatment of temporal expressions in question analysis, indexing and extraction and more detailed experiments are presented below.
Document Collection
Question Analysis
Linguistic Analysis
Question Classification
Offline Operations
Collection Indexer Linguistic Analysis
Information Retrieval
Query Generation Information Retrieval
Document
Index
Linguistic Analysis
Question Classification Evaluation figures are detailed in Table 1. Answer accuracy has been calculated as the ratio of questions correctly answered to the total number of questions. Only the first candidate answer is considered, rejecting the rest of possibilities. 0.32 0.29 The results on the CLEF09 test set show similar conclusions to those we obtained during our development process, the baseline system using passage retrieval is hard to beat and in fact our second run provide lower accuracy. As in the case of our development experiments there are changes for individual answers of a number of questions but the overall effect is not positive.
After the evaluation, and using the larger test set of 500 questions we have decided to carry a class based analysis in order to understand the causes behind our unfruitful efforts. We have manually annotated the questions and grouped them in 6 main question types. In contrast with our expectations, the performance of the second submitted run is also worse for the factual and definition questions. As we have considered these questions types in previous evaluations we expected to have better coverage in the Answer Filter and therefore an improvement. Similar behaviour has been observed across answer types for factual questions, being the class of TIMEX questions the only where the more complex configuration really improves.
Our analysis of the errors show that further work is needed to be able to cope with the complexities of the domain. For example, questions are in general more complex and include a large number of domain specific terminologies that our question analysis rules do not handle correctly. The process of finding the focus of the question which is crucial for question classification is specially error prone. Answer Extraction needs also further adaptation to the domain for factual questions as the typology of NE and generalized NE has not wide coverage. Problems with definitions are rooted more deeply and probably require the use of different specialized retrieval strategies. This year evidence along with previous experiments seems to support that definitions depend deeply on the stylistics of the domain. Finally, new question types would require further study of techniques that help to improve the classification of passages as bearing procedures, objectives, etc.
FACTUAL PROCEDURE
CAUSE REQUIREMENT
DEFINITION OBJECTIVE
ALL ALL - FACTUAL
BL 54 22 43 5 16 21 161 107
BL-AF 48 15 44 5 12 23 147 99 With the aim of evaluating the temporal management capabilities of the QA system, we decided to extract the temporal questions from the whole corpus. 46 out of 500 queries denote temporal information, that means a 9,20% over the total. 24 of them are TR questions, whereas TA queries are 22 (4,80% and 4,40% out of the total, respectively). This subset has been studied, evaluating the correctness of the returned answers by two different configurations of the QA system. The results are presented in Table 3.
BL 0.44
BL-AF
0.39
As we can observe, better figures are obtained by the set of TQ in both runs. There is no significant difference
between TA and TR queries in the first run, while in the second one they achieve a difference of 22%. In our
opinion, the second configuration, with answer filtering and answer creation, enhances precision for TA queries,
whereas for TR queries, temporal restrictions introduce noise that the system is not able to solve.
Non-submitted runs present similar configurations to the submitted ones, but adopting a different index
generation and question analysis strategies. The approach consisted on the inclusion of normalized temporal
expressions into the index, as well as in the question analysis process, aiming to increase recall. We tested the
performance over the total corpus of questions, but worse results were achieved even if the study is restricted to
temporal questions. Results are also presented in Table 3, which show no improvement regarding the submitted
runs. Performance difference between TA and TR queries remains stable, since the system has a better response
to questions without temporal restrictions. The lost of accuracy can be due to the lack of a more sophisticated
inference mechanism at the time of retrieval, capable of reasoning with different granularities in normalized
dates format [
From our point of view, the new ResPubliQA exercise is a challenge for QA systems in two main facets of the
problem domain adaptation and multilinguality. This year our efforts have focused on the first problem where we
have ported the system and the techniques developed for EFE and Wikipedia to the new legal collection
JRCAcquis. However, our experiments, which are exemplified with the submitted runs, show that a system mainly
based on passage retrieval performs quite well. Baseline passage retrieval results provided by the organizers [
Finally, further work in the general architecture of the QA is expected to help in at least three areas: separation of domain knowledge from general techniques, adding different languages to the system and effective evaluation.
This work has been partially supported by the Regional Government of Madrid by means of the Research Network MAVIR (S-0505/TIC/000267) and by the Spanish Ministry of Education by means of the project BRAVO (TIN2007-67407-C3-01) 5th International Conference on Language Resources and Evaluation (LREC'2006). Genoa, Italy, 24-26 May 2006.