In the 2012 edition of CLEF, the DI@UE team has signed up for Question Answering for Machine Reading Evaluation (QA4MRE) main task. For each question, our system tries to guess which of the five hypotheses is the more plausible response, taking into account the reading test content and the documents from the background collection on the question topic. For each question, the system applies Named Entity Recognition, Question Classification, Document and Passage Retrieval. The criteria used in the first run is to choose the answer with the smallest distance between question and answer key elements. The system applies a specific treatment for certain factual questions, with the categories Quantity, When, Where, What, and Who, whose responses are usually short and likely to be detected in the text. For the second run, the system tries to solve each question according to its category. Textual patterns used for answer validation and Web answer projection are defined according to the question category. The system answered to all 160 questions, having found 50 right candidate answers.
this year there are 160 questions, 40 more than last year [
The system architecture and the employed resources are described in the next section. The third section presents the methodology used for question processing, including some examples. Section 4 lists the results of the system, and the last section is devoted to the analysis of results and considerations on the work done. 2
The system maintains the architecture defined for the previous year participation. In Figure 1 we can see the different system modules. The XML Layer receives the input, does the parsing and organization of the questions with their multiple choice answers, while maintaining the connection to the topic to which they relate and to their particular reading test document. When all questions are processed, this component generates the XML output and makes sure the syntax is correct and conforming to the DTD.
The Question Classifier module makes a linguistic analysis to the question text
to determine its category. The system uses the parser CandC with the Boxer2
semantic tool, that can produce Discourse Representation Structures (DRS) [
The Libs Module contains the Background Collections (BC), which include the
English version of the four 2012 topics (AIDS; Climate Change; Music and
Society; Alzheimer ) documents. This corresponds to almost 3 gigabytes of text.
For text retrieval we keep using the Lucene3 search engine. Wordnet [
3 Apache Lucene is an open source project with advanced indexing and searching features. http://lucene.apache.org/ 4 Java API for WordNet Searching: http://lyle.smu.edu/˜ tspell/jaws/index.html The Answer Analyzer is responsible for assess each answer choice for a question. This includes a linguistic analysis of the text of the response and a textual search process. The search can be simple or defined according to the category of question, through textual patterns for answer projection over the BC or over the Web. With the information collected for each candidate answer to a question, the Answer Selector module applies a criteria to choose the most plausible answer. These criteria may be more general, if the question was not classified in any specific category, or more directed and concrete, as we shall see in the next section. For each question, our system tries to guess which of the five hypotheses is more plausible response, taking into account the reading test content and the documents from the background collection on the question topic (Aids, Climate change, Music and Society, and Alzheimer). In this edition, we submitted two runs for QA4MRE evaluation. The first run applies a generic strategy of surface text analysis on the reading test and documents retrieved from background collections, with no other external resources. For each question, the process begins with Named Entity Recognition, for prior identification of any entity names, dates, quantities or other expressions that influence question interpretation. Then, processing continues with question classification, document retrieval and passage retrieval. The system performs a search for documents in the BC that can support one of the possible answers to the question.
Each candidate answer has a set of retrieved passages, which are text segments. For each of the multiple choices we intend to verify if both question and answer key elements are present in the text segments, and what is the distance between them. The criteria used in the first run is to choose the answer with the smallest distance between question and answer key elements. The question key element to find in the text segments is the question focus, the entity or object that the question refers to. In both cases, the stop words are filtered from key elements. Because the textual justification was no longer needed, we decided to answer all questions using the broader heuristic whenever a specialized strategy was not applicable.
Questions with different nature are processed differently. The system applies a specific treatment for certain factual questions, whose class and focus are identified. Such specialized strategy is applied to factoid questions with the categories Quantity, When, Where, What, and Who, whose responses are usually short and likely to be detected in the text. Question classification requires a question text analysis, in particular the identification of the interrogative term and the question focus. As an example, the question ’How many degrees did Burney receive from Oxford? ’ has Quantity category. Its focus is Burney, and what we need about it is the number of degrees received from Oxford. In another example, the interrogative used in question ’Where was Burney working when he first conceived the idea of writing a music history? ’ determines its classification in the type Where, guiding further processing to a location.
The use of textual patterns is very common in question answering mechanisms,
as in [
For the second run, the system tries to solve each question according to its category, adopting a specialized strategy of resolution. The textual patterns used for answer validation are defined according to the category, representing common cases for that type of question. The system checks the presence of question and answer key elements on a text segment based on term exact match as done for the first run, firstly, but also through semantic compatibility (synonym, hyperonym, base form). This is a semantic query expansion of the search terms. Without the need to associate with each answer a supporting text, it becomes possible to use also background collections independent heuristics, such as Web answer projection to validate the options for the question. This technique is only used when the question is classified as one of the factoid categories supported by the system. An example where the Web answer projection helped to identify the correct choice is ’What is the population of Brazil? ’. The correct answer appears on the reading test, but away from the question focus terms. Projecting the answer choices on the web, the answer 180 millions emerged as the right one. The decision to choose the answer to a question is based on the following: 1. The criteria used in the first run is to choose the answer with the smallest distance between question and answer key elements. 2. The second run aims for a more informed choice process. If the system finds some question category answer pattern in a retrieved document, or on the Web, for a single answer option, then that option is the chosen. Even if other options have results for the general text surface search, the answer patterns under the question category have priority. 3. If there is a tie, in any case, the choice falls on the option having more support text segments, which are the retrieved text passages where answer patterns or key search terms were found. 4. If the surface term search, for the general case, or the answer pattern search does not return any results, for any of the five hypotheses, then the question remains unanswered.
In the following section we discriminate the results obtained in each run submitted to QA4MRE. 4
In the first approach, the system answered to 156 questions using the surface based technique. Figure 2 illustrates the proportion between the amount unanswered questions, and also the correct and the mistaken results. The system gave 45 correct answers and the remaining 111 were wrong. The questions referred to in the previous section have been answered, but the choice was not right5 in any of them.
In the first run 4 questions were unanswered due to an unexpected parse error. That problem was fixed for the second run. In this last run, which is the most complete, the system changed the response in 20 of the 156 questions previously answered with the surface text search approach.
The system kept the answer to both Burney questions mentioned in section 3, keeping the erroneous choice. But for ’Which pupil of Dr John Blow taught Charles Burney? ’, a What class question, the system changed from a wrong answer to the right answer (Edmund Baker ), in the second run.
In other examples, such as ’How many countries have acted effectively against AIDS? ’ and ’Where is the epicenter of the AIDS pandemic? ’, were answered correctly in both runs.
The chart in Figure 3 summarizes the evaluation results for our system. The number of hits increased to 50. At the same time, 4 more questions have been processed, leaving no question unanswered, and the number of wrong answers was even reduced by one.
Of the four unanswered questions in the first run, three were answered incorrectly. Only the first of these, ’Name two styles which have contributed to pop music:’, got the right answer on the second run.
Table 1 shows a more detailed assessment with the breakdown of values for each
run. The system first attempts resulted in a 0.28 accuracy and 0.29 C@1 values.
C@1 is a balanced measure rewarding systems that, for the same number of
correct answers, decrease the number of incorrect results by leaving some questions
unanswered [
The accuracy rose to 0.31 in the second run, and the overall C@1 measure was also 0.31.
Table 2 shows a comparison between the result of this year’s better run and the best result achieved by our system in 2011, where there were 120 questions 5 According to the solutions disclosed in July 2012: QA4MRE-2012-EN GS.xml. for processing. The last section has some thoughts on these results and on our participation in this Lab.
Run In 2011, our system answered to 73 of 120 questions, finding correct 18 answers. In this edition, our system answered correctly to 50 questions out of 160. This represents a substantial improvement in accuracy, from 0.15 to 0.31. Compared with the previous year, the question classifier has improved, being more effective in assigning the category of factoid questions. We believe that this update in the question classifier was the key to improving outcomes, especially for allowing the application of specific procedures for each category of question. This year we also improved the text analysis performed on the question and answer hypotheses, using the CandC and Boxer tools.
Looking at the charts in figures 2 and 3, the wrong answers slice is significantly higher. Answer all questions may not have been a good decision. In future we can introduce a confidence factor or more appropriate criteria to decide between responding and non-responding. With such a procedure, the system can improve the C@1 measure.
Errors in question classification not always determine a wrong answer. Question ’What is the external debt of all African countries? ’ asks for a monetary value and should have been classified as Quantity. The system classified it as a What question, whose resolution process is not optimized for numeric values. Yet, both the surface search and the answer projection succeed, and the option chosen by the system was the correct in both runs.
Despite the improved accuracy results, we consider that the number of wrong answers is very high. This may reflect a need to use more semantic based techniques, and perhaps to apply an intensive linguistic analysis to BC documents. This second participation in QA4MRE helped us to adjust our system to English, focusing primarily on factual answer questions and specific categories, but with an alternative methodology for the general case.