This paper describes the Question Answering for Machine Reading (QA4MRE) Main Task at the 2013 Cross Language Evaluation Forum. In the main task, systems answered multiple-choice questions on documents concerned with four different topics. There were also two pilot tasks, Machine Reading on Biomedical Texts about Alzheimer's disease, and Japanese Entrance Exams. This paper describes the preparation of the data sets, the definition of the background collections, the metric used for the evaluation of the systems' submissions, and the results. We introduced two novelties this year: auxiliary questions to evaluate systems level of inference, and a portion of questions where none of the options were correct. Nineteen groups participated in the task submitting a total of 77 runs in five languages.
1. 2.
Linguistic inferences such as co-reference, deictic references (like “then” and “here”), etc.); Simple ontological inferences such as considering part-of relations or obtaining direct super-concepts for common objects; 3. Inferences considering causal relations or procedural steps in “life scripts” like visiting a restaurant or attending a concert; 4. Inferences that require composing several answers, in particular answering one part of the question using the background collection and then, with its answer, answering the other part of the initial question (e.g., “Who is the wife of the person who won the Nobel Peace Prize in 1992?”). 2
In 2013, we had three exercises.
Main Task. This remained the same for participants. Background collections, test documents and reading tests were available in Arabic, Bulgarian, English, Romanian, and Spanish. There were four topics: AIDS, Alzheimer's Disease, Climate Change and Music and Society. As was the case last year, there is also a pilot task on Alzheimer's disease. The difference is that the reference collection for the main task is built from general public sources and for the pilot the source is the PubMed repository. Following the pilot task last year on Processing Modality and Negation, these aspects were incorporated into questions within the main task.
Machine Reading on Biomedical Texts about Alzheimer's disease. This exercise is aimed at setting questions in the Biomedical domain with a special focus on one disease, namely Alzheimer's. This pilot task explored the ability of a system to answer questions using scientific language. Texts were taken from PubMed Central related to Alzheimer's and from 66,222 Medline abstracts. In order to keep the task reasonably simple for systems, participants were given the background collection already processed with Tok, Lem, POS, NER, and dependency parsing. Entrance Exams. University Entrance Exams include questions formulated at various levels of complexity and test a wide range of capabilities. The challenge of "Entrance Exams" aims at evaluating systems under the same conditions humans are evaluated to enter the University. In this first campaign we will reduce the challenge to Reading Comprehension exercises contained in the English exams. More types of exercises will be included in subsequent campaigns (2014–2016) in coordination with the "Entrance Exams" task at NTCIR. Exams are created by the Japanese National Center for University Admissions Tests. The "Entrance Exams" corpus is provided by NII's Todai Robot Project and NTCIR.
In this paper we describe the Main task. The two other tasks are described in detail in dedicated papers in these proceedings. 2.1
Tests were divided into: ‐ 4 topics, namely “Aids”, “Alzheimer”, “Climate change” and “Music and Society”; ‐ Each topic had four reading tests; ‐ Each reading test consisted of one single document, with 15 Main questions (six having no answer in the text) and a set of five choices per question. The last of the five choices was always “None of the above”. In addition, one or more Auxiliary questions could be asked, each of which was a simplification of a Main question (see discussion later).
Overall, the following evaluation setting was proposed: - 16 test documents (4 documents for each of the four topics), - 240 Main questions (15 questions for each document), - 1200 choices/options (5 for each question).
Test documents and questions were made available in Arabic, Bulgarian, English, Romanian and Spanish. These materials were exactly the same in all languages, created using parallel translations. 2.2
We introduced two novelties this year: (i) auxiliary questions to evaluate systems level of inference, and (ii) a portion of questions where none of the options were correct.
With respect to auxiliary questions, they correspond to Main questions where a deliberate simplification is done by removing one inference step. The idea was that if a system answered a Main question incorrectly but the corresponding Auxiliary question correctly, it suggests that the system was near to answering the question but could not perform the inference step. In a similar way, if a system answers the main question but not the simplified one, this indicates a lack in the inference process. Hence this approach could be used to pinpoint the exact shortcomings of a system.
With respect to questions without correct answers among candidates, the idea is to test the ability to reject candidate answers when they are incorrect. We implemented this change by introducing in our tests a portion of questions where none of the options are correct and including a new last option in all questions: “None of the answers above is correct”. 3
This is a very important element of the evaluation setting. It connects the task also with the research in Information Retrieval. The goal of reference/background collections is to contextualize the reading of a single document related to the topic by collecting and fleshing out additional pertinent information. In the future this step may be done on the fly as a retrieval process once a single test text is provided. However, for now, we provide a carefully constructed background corpus for two main reasons: to allow more comparison among participant systems, and to focus on the Reading Comprehension problem. We believe it is important to develop a good methodology for building background collections for the evaluation task.
We define background knowledge in terms of the relation between the testing questions and answers, and the background collection. To determine the potential kinds of uses of the prior knowledge, we distinguish at least four main types of background knowledge (although in fact it’s a continuum): 1. Very specific facts related to the document under study. For example, the relevant relation between two concrete people involved in a specific event. 2. General facts not specific to any particular event. For example, geographical knowledge, main players in international affairs, movie stars, world wars, etc. Also acronyms, transformations between quantities and measures, etc. 3. General abstractions that humans use to interpret language, to generate hypotheses or to fill missing or implicit information. For example, abstractions such as the result of observing the same event with different players (e.g. petroleum companies drill wells, quarterbacks throw passes, etc.) 4. Linguistic knowledge. For example, synonyms, hypernyms, transformations such as active/passive or nominalizations. Also transformations from words to numbers, meronymy, and metonymy.
Obviously this is not an exhaustive list. For example, we do not include ontological relations that enable temporal and spatial reasoning, or reasoning on quantities, which are also all relevant.
Ideally, the background collection should cover completely the corresponding topic. This is feasible sometimes and unrealistic at others. For example, in the case of the pilot on Biomedical documents about Alzheimer's disease, a set of experts built a query (a set of conjunctions and disjunctions over 18 terms) that approximates very much the retrieval of all relevant documents (more than 66,000) without introducing much noise. However, this is not so easy in more open domains (e.g., Climate Change) or cases with non-specialized sources of information. In these cases, we crawl the web using, for each language and topic a list of keywords and a list of sources. Keywords are translated into English and then translated into the rest of the languages. Documents may be crawled from a variety of sources: newspapers, blogs, Wikipedia, journals, magazines, etc. The web sources are obviously language dependent, and each language also requires a list of possible web sites with documents related to the topic.
We realized in 2011, since we organizers knew the test set, we used that information to select the keywords, and ensure the coverage of the questions. The effect is not only that background collections don’t cover completely the topic, but also that the collections have some bias with respect to the real distribution of concepts. In this year's campaign, the assumption that the ideal background collection should include all relevant documents for the topic (and only them) is explicit, and we organizers bear it in mind. Thus, we face the same problem as traditional Information Retrieval: we want all relevant documents (and only them), and we use queries (keywords) to retrieve them
The first strategy with the aim of ensuring the coverage of the topic as much as possible is to make the topic specific enough (e.g., AIDS medicaments rather than AIDS). The second strategy is to try to cover (at least partially) each of the possible “dimensions/aspects” of that topic. How? First, by detecting a good central overview text, such as a Wikipedia article that “defines” the topic, “suggests” its principal aspects, and provides links to additional good material. Then, organizers enumerate these dimensions and prepare a set of queries for each dimension. They document this process with three benefits: (i) to know what organizers and participants can expect or not from the collection; (ii) to give another dimension of re-usability; and (iii) to explore how Machine Reading will connect to Information Retrieval in the future.
TEST SET PREPARATION history of violin technique music patronage rock and roll history of song electric musical instrument classical recording industry economics of classical music classical crossover music
This year the datasets was created for the following five languages: Arabic, Bulgarian, English, Romanian and Spanish. The dataset was created following the methodology developed in previous years and consisting of the following steps:
Four English documents were selected for each of the four topics (Aids, Alzheimer's, Climate Change, Music and Society). These were selected from various sources (see Table 4) and comprised the test documents against which questions were asked. The documents for the first three topics were chosen from copyright-free sources. The documents for Music and Society were selected from Grove Music Online (http;//www.oxfordmusiconline.com) by kind permission of the Editor in Chief, Editor and Oxford University Press. This source was chosen because of its exceptional scholarly quality, as well as the very large choice of articles available on music of all kinds.
In order to have a set of identical questions for the five languages above, we needed to have the selected test documents translated. For this purpose, expert translators were recruited form the Translation for Progress1 platform for all languages.
To ensure that translations were faithful to the original document in both meaning and style and of good quality, all the documents were manually checked and corrected when necessary. We wanted to avoid a situation where portions of the original English text were left out of the translation in a particular target language, or perhaps modified or interpreted in a particular manner which would have made the question impossible to answer in that language.
Fifteen multiple-choice questions were then devised for each test document (the ‘Main’ questions). A question always had five candidate answers from which to choose, with one clearly correct answer and four clearly incorrect answers. In all cases the fifth candidate answer was “None of the above”. Six of the fifteen questions were composed so as to have no answer in the text. The correct response to each of these six questions was thus “None of the above”.
In addition to the fifteen Main questions, one or more Auxiliary questions could also be devised. Each Auxiliary question was a simplified version of an existing Main question. The format of these questions was identical to that of Main questions, i.e. a question followed by five multiple-choice answers. In most cases, the Auxiliary question required less inference to answer. The idea was that if a system was able to answer the Auxiliary question but not the corresponding Main question, the problem could be its ability to perform the missing inference. This is discussed more below.
Once the questions had been composed in the language of the original author, each was then translated into English. The English versions of the questions and candidate answers were carefully checked by a referee to verify that they were clear, that the intended answer was clearly correct, that the intended answer was in the test document, and that the other candidate answers were clearly incorrect. Questions were modified accordingly. 1 http://www.translationsforprogress.org/main.php A Translation Exchange site linking volunteer translators (e.g., linguistics students or professionals in foreign languages interested in building experience as translators can link up with low-budget organizations who are in need of translation work, but without the budget to pay for it. There are currently over 1450 registered volunteer translator members (for 13 language combinations) and over 160 organization members. Translation for Progress database is open for viewing for the general public, but if you wish to post your profile or contact a volunteer translator, a registration is required. http://blog.kylebarlow.com/2012/ 04/of-mice-and-men-alzheimerscure-for-our.html http://www.insight.mrc.ac.uk/201 2/10/19/fighting-alzheimersdisease-get-the-immune-systemon-board/ http://www.ted.com/talks/alanna_ shaikh_how_i_m_preparing_to_g et_alzheimer_s.html http://www.alz.co.uk/icaniwill/lib rary/people-with-dementia/livingwith-early-memoryloss/financial-challenges Grove Music Online at http://www.oxfordmusiconline.co m
http://www.oxfordmusiconline.co m Grove Music Online at http://www.oxfordmusiconline.co m http://www.fpif.org/articles/latin_ america_climate_change_swing_s tates http://www.fpif.org/articles/globa l_warming_its_all_about_energy
Graue, Thomas Milligan Geeta Dayal, Emily Ferrigno
The English versions were then used to translate each question into each of the five languages of the task. The same process was used to translate each candidate answer (five per query) into the five languages. The result of this process was a set of 240 Main questions and 44 Auxiliary questions in five languages, each with five multiple-choice answers, also in those five languages. The final step was to check that the answer to each question was in fact present in the test document for all the languages of the task.
3 4 5 6 7 8 9 10 11
and men: an Alzheimer’s cure for our murine brethren Fighting Alzheimer’s disease? Get the immune system on board How I’m preparing to get Alzheimer’s
Musicology - Analytic Traditions "Latin America: Climate Change Swing States" (Washington, DC: Foreign Policy In Focus, July 22, 2010) "Global Warming: It's All About Energy" (Washington, DC: Foreign Policy In Focus, February 15, 2007) "Ozone Depletion & Global Warming" (Washington, DC: Foreign Policy In Focus, October 12, 2005)
AttributionNonCommercial 3.0
For each text in the test set 10 multiple choice questions were created. Each question had five answer options. The fifth option was always ‘None of the above’. The questions covered five different question types: purpose, method, causal, factoid, and which-is-true. Factoid questions were divided into the following sub-types: Location, Number, Person, List, Time and Unknown. Examples of the basic question types are given below. We took care to spread the question types evenly for a given test document, aiming for two questions per type. The exact breakdown of the number of questions per type in the test collection is provided in Table 5 below. Example questions:
PURPOSE: What is the aim of protecting protein deposits in the brain? METHOD: How can the impact of Arctic drillings be reduced? CAUSAL: Name one reason why electronic dance music owes a debt to Kraftwerk.
FACTOID (number): What is the approximate number of TB patients? WHICH-IS-TRUE: Which problem is similar in nature to global warming?
PURPOSE METHOD CAUSAL FACTOID* WHICH-IS-TRUE
For all questions, the direct answer was contained in the test document; however answering the questions typically required some background knowledge and some form of inference. The required knowledge could be linguistic or could involve basic world knowledge. Linguistic knowledge concerns, for example, the ability to perform co-reference resolution or detect paraphrases on the lexical or syntactic level. World knowledge has to be inferred from the background collection. For instance, the text might mention Barack Obama while the question might refer to the first African American President. The fact that Barack Obama is the first African American President needs to be learnt from the background collection in order to be able to answer the question.
Typical types of world knowledge involve, for instance, knowledge about the basic referents in a text, e.g., being aware that Yucca Mountain is in Nevada. Another type of world knowledge involves knowledge of “life scripts” such as “visiting a restaurant”. Finally, the inference required can also be complex, involving several steps. For example, answering a question might require combining knowledge from the background collection with knowledge from the test document itself. For instance, the question “Who is the wife of the person who won the Nobel Peace Prize in 1992?” contains two facts P and Q, where P=“wife of Y=?” and Q=“winner of Nobel Peace Prize in 1992=Y”. The latter information can be gleaned from the background collection whereas the former is contained within the test document itself.
For each test document, we aimed for a combination of simple, medium, and difficult questions. At most six questions per document did not require knowledge from the background collection. Two of these were simple questions, i.e., the answer and the fact questioned could be found in the same sentence in the test document. Four questions were of intermediate difficulty in that the answer and the fact questioned were not in the same sentence and could, in fact, be several sentences apart. Finally, the remaining four questions did require utilizing information from the background collection. While not all question types require inference based on the background collection, all of them required some form of textual and linguistic knowledge, such as the ability to detect paraphrases, as we made an effort to re-formulate questions in such a way that the answers could not be found by simple word overlap detection. For each question, we kept track of the inference required to answer it. This made it easier to ensure that that inference could in fact be drawn on the basis of the background collection, i.e., that the background collection did indeed contain the relevant fact. It also makes it possible to carry out further analyses regarding which questions or types of questions were difficult for the systems and why.
When creating the questions, we took care not to introduce any artificial patterns that would help finding the correct answer. Thus we ensured that all answer choices for a question were approximately the same length and consistent with respect to formulation and content, that all of the wrong answers were plausible, and that the placement of the correct answers was random and balanced. Background knowledge required Information needs to be gathered from difference sentences of paragraphs #of questions 119 45 120
AUXILIARY QUESTIONS NO CORRECT ANSWER 44 39 0.21 0.30 0.22
Table 7 below shows a breakdown of questions which are auxiliary (see below), have no correct answer, or contain modality or negation in either the question or the answer. 4.2
In the first two years of the QA4MRE task, questions required a deep understanding of the text. However, since they were multiple-choice, the answer was simply judged as correct or incorrect. In the case of a correct question, it was impossible to judge whether the answer had been chosen at random or derived from a valid process of deduction. Similarly, if the answer was incorrect, it was impossible to judge why. To address this latter issue, an experiment was conducted this year in which Auxiliary questions were posed in addition to Main questions. Each Auxiliary question corresponded to a Main question and was a deliberate simplification of it which removed one inference step. The idea was that if a system answered a Main question incorrectly but the corresponding Auxiliary question correctly, it suggests that the system was near to answering the question but could not perform the inference step. Hence this approach could be used to pinpoint the exact shortcomings of a system.
In the main, three forms of simplification were used, hypernym replacement, noun phrase synonymy, and verbal entailment. Moreover, simplification could be made to the question or to the correct answer. Here are some examples.
In this example, the simplification is made to the question: Q (main): What has been offered to the President of the United States if he signs the Kyoto Protocol? Q (aux): What has been offered to Obama if he signs the Kyoto Protocol? Supporting text:
Perhaps most surprising was Stern's stop in Quito, Ecuador. The United States slashed $2.5 million of support when Ecuador submitted a letter that it would not join the accord. In response, Ecuadorian Foreign Minister Ricardo Patiño offered the United States $2.5 million if Obama signed the Kyoto Protocol.
Here, the hypernym "President of the United States" has been replaced by its hyponym "Obama" in the Auxiliary question. The supporting text refers to Obama and not President of the United States. Thus in order to answer the Main question, a system must infer that Obama is the President. This inference is not needed for the corresponding Auxiliary question since Obama is actually mentioned in both the question and the document.
Q: What sort of music was written for Hollywood films in the Golden Age? Supporting text:
The conventions of the "classical" Hollywood film score in the Golden Age - essentially a leitmotifbased symphonic romanticism with narrative orientation, the music almost always subordinated to the primacy of the visual image and dialogue - prevailed in scores by other expatriate musicians.
Here, the simplification is made to the answer, while the wording of the Main question and Auxiliary question remains the same:
A (main): music for orchestra with strong melodies A (aux): music embodying leitmotif-based symphonic romanticism
The Main question can only be answered by deducing that "music for orchestra with strong melodies" is largely synonymous with "leitmotif-based symphonic romanticism"; i.e., "symphonic" implies that the music is for full orchestra while "leitmotif-based" implies the use of strong easily-recognised melodies, associated with ideas or characters in the film.
Q: What was Cramer's attitude towards the music of Bach? Supporting text (with added italics):
He may have been introduced to Das wohltemperirte Clavier as early as 1787, and he developed a lifelong fascination for Bach.
Again, the simplification is made to the answer: A (main): he admired Bach all his life A (aux): he developed a lifelong fascination for Bach
Here, the Main question can only be answered by deducing that "he admired Bach all his life" is entailed by the supporting text "he developed a lifelong fascination for Bach". In the Auxiliary question, the answer is a substring of the supporting text, so no entailment is needed.
In all, 44 Auxiliary questions were composed, seventeen containing a simplification of the question and 27 containing a simplification of the correct answer. Analysis of the results concerning these questions can be found in Section 6.1. 5
This task has the aim of promoting a change in QA architectures giving more importance to the validation step over the IR component in order to improve results. This is why we have been proposing from 2009 to evaluate system confidence by introducing the possibility of leaving questions unanswered [1]. Thus, systems might reduce the amount of incorrect answers while keeping the proportion of correct ones.
However, the analysis of last editions has shown how systems rely more on ranking than in validation of candidate answers. These systems calculate the similarity of each candidate answer with a combination of the question and certain snippets of the document and return the most similar answer. Hence, systems have not shown nor developed their ability discarding incorrect answers. Besides, it is not clear the behavior in case of not providing the candidate answers.
This is why we introduce in this edition an explicit assessment focus on testing the ability to reject candidate answers when they are incorrect. We implemented this change by introducing in our tests a portion of questions where none of the options are correct and including a new last option in all questions: “None of the answers above is correct”.
This modification does not affect the output of participants since given a question with its corresponding candidate answers, a participant system can return two kinds of responses:
An answer selected from the set of candidate ones for that question, taking into account that one candidate is “None of the answers above is correct” A NoA answer. This response should be given if the system considers it is not able to find enough evidences about the correctness of candidate answers and it prefers not to answer the question instead of giving an incorrect answer. Moreover, the system can return as a hypothetical answer the candidate one that it would have been selected, which allows to give some feedback about its validation performance.
The assessments of system’s responses are given automatically by comparing them against the gold standard collection. Therefore, no manual assessment was required, which reduces the effort of the evaluation once the collections have been created and makes easier the future development of systems. Each system’s response to a question receives one and only one of the following three possible assessments:
Right if the system has selected the correct answer among the set of candidate ones of the given question; Wrong if the system has selected one of the wrong answers; NoA if the system has decided not to answer the question. Where the system returned a hypothetical answer, this answer was assessed as NoA_R in the case of it being correct or NoA_W if it was wrong.
It is important to remark that a NoA answer is different to a “None of the answers above is correct” answer. The former means that the system does not return any candidate answer because it is not confident about giving the correct answer, while the latter means that the system rejects the other candidate answers but returns a response that will be assessed as Right or Wrong.
Evaluation of systems given was given from two perspectives following the format of last editions:
A question-answering approach, as in the traditional evaluation performed in past campaigns, where we just evaluate the ability of systems answering a set of questions and rank systems according to the final value given by a measure.
A reading-test evaluation, obtaining figures for each particular reading test and topics. This perspective permits us to evaluate whether a system was able to understand a document and to what degree. More in detail, we evaluate if the system is able to pass each test, in a similar way to humans with RC tests. This is a kind of evaluation studied with more detail in the pilot Entrance Exams task. 5.1
We keep c@1 as the main evaluation in this edition. c@1 was introduced in ResPubliQA 2009 [1] and is fully described in [2]. The formulation of c@1 is given in Formula (1). 1 n ( nR +nU nnR ) (1) where nR: number of questions correctly answered. nU: number of questions unanswered.
n: total number of questions
The main feature of c@1 is its consideration of unanswered questions. c@1 acknowledges unanswered questions in the proportion that a system answers questions correctly, which is measured using the traditional accuracy (the proportion of questions correctly answered). Thus, a higher accuracy over answered questions, which might be associated to a better validation, would give more value to unanswered questions, and therefore, a higher final c@1 value. By selecting this measure we wanted to encourage the development of systems able to check the correctness of their responses because NoA answers add value to the final value, while incorrect answers do not.
As a secondary measure, we also provided scores according to accuracy (see Formula (
n: total number of questions where 5.2
The Question Answering perspective is focused on measuring systems’ performance over a set of questions without considering the ability of a system to pass tests associated with documents. This is an approach similar to the one applied in QA@CLEF campaigns before 2010.
Then, the information considered for each system at this level is:
Total number of questions ANSWERED. This number is divided into: o total number of questions ANSWERED with a RIGHT answer, o total number of questions ANSWERED with a WRONG answer.
Total number of questions UNANSWERED (a NoA response was given). This number is divided into: o total number of questions UNANSWERED with a RIGHT candidate answer, o total number of questions UNANSWERED with a WRONG candidate answer, o total number of questions UNANSWERED with an EMPTY candidate answer.
The following scores are calculated from this information:
An overall c@1 score over the whole collection (the set with 160 questions), A c@1 score for each topic (40 questions for each topic), An overall accuracy score (over the 160 questions of the test collection, considering also the candidate answers given to unanswered questions as it has been explained above), The proportion of answers correctly discarded (see Formula (3)) in order to evaluate the validation performance. correctlydiscarded =
nUW + nUE nUR + nUW + nUE (3) where: nUR: number of unanswered questions whose candidate answer was correct nUW: number of unanswered questions whose candidate answer was incorrect nUE: number of unanswered questions whose candidate answer was empty
The objective of the reading perspective evaluation is to offer information about the performance of a system “understanding” the meaning of each single document. This understanding is evaluated by means of multiple-choice tests with ten questions per document. That is, each system has to pass a test about a given document similar to the evaluation of RC of new language learners. As we said above, this kind of evaluation is studied more in detail in a pilot task of this edition.
This evaluation is performed taking as reference the c@1 scores achieved for each test (one document with its ten questions). Then, these c@1 scores can be aggregated at topic and global levels in order to obtain the following values:
Median, average and standard deviation of c@1 scores at test level, grouped by topic,
Overall median, average and standard deviation of c@1 values at test level.
The median c@1 is provided under the consideration that it can be sometimes more informative at reading level than average values. This is because median is less affected by outliers than average, and therefore it provides more information about the ability of a system to understand a text.
We consider that a system passes a test according to this evaluation perspective if it achieves a score equal or higher than 0.5. 5.4
This baseline randomly selects an answer from the set of candidate answers. Since there is one correct option among five, the overall result of this random baseline is 0.2 (both for accuracy and for c@1). Systems applying a reasonable kind of processing and reasoning should be able to outperform this baseline. 6
From an initial amount of 39 groups that registered for the main task and signed the license agreement to download the background collections, 19 of them finally submitted at least one run, resulting in 77 monolingual runs in four languages (Bulgarian, English, Spanish, and Romanian). There were no Arabic runs this year and neither were there any cross-lingual runs. Tables 8-10 show a characterization of runs.
However, the mean values for all the tests were still under 0.5, which is the score needed to pass the evaluation from the reading perspective. This is the same result as last year and suggests that systems are still far away from obtaining satisfactory results according to this perspective.
Apendix I and II show these results for al submitted runs lims1303enen onto1306bgbg uaic1303roro buap1302enen buap1310enen evma1301enen vens1301enen uaic1302roro uaic1306roro onto1301bgbg onto1302bgbg onto1303bgbg onto1304bgbg
Baseline 1 btbn1301bgbg evma1302enen btbn1302bgbg uaic1304roro uaic1305roro
Baseline 2 # with EMPTY candidate answer 59 19 0 0 0 0
The best results were obtained in English, where the highest score was obtained by jucs1302enen with 0.59. This value is 24 percentage points higher than the next system (kule1304enen at 0.35). In 2012 the jucs group also submitted the best run, jucs12013enen with a score 0.65. So, for a second year running, jucs was the only system able to pass the evaluation according to the reading perspective. Moreover, their individual scores were well over 0.5 in every topic except Topic 3, Climate Change (where their score was 0.3). 0.59 0.39 0.35 0.35 0.34 0.33 0.33
Total 225 265 284 284 264 242 nara1301enen 0.33 buap1304enen 0.32 buap1305enen 0.32 kule1308enen 0.32 buap1303enen 0.31 buap1309enen 0.31 kule1309enen 0.31 buap1306eses 0.30 csui1302enen 0.30 csui1304enen 0.30 csui1305enen 0.30 csui1307enen 0.30 kule1307enen 0.30 nara1302enen 0.30 csui1303enen 0.29 csui1306enen 0.29 csui1308enen 0.29 csui1309enen 0.29 buap1307eses 0.28 buap1308eses 0.28 csui1310enen 0.28 lims1304enen 0.28 lims1305enen 0.28 onto1305bgbg 0.28 onto1307bgbg 0.27 onto1308bgbg 0.27 evma1303enen 0.26 lims1302enen 0.26 lims1303enen 0.25 onto1306bgbg 0.25 uaic1303roro 0.25 buap1302enen 0.24 buap1310enen 0.24 evma1301enen 0.24 vens1301enen 0.24 vens1302enen _LATE_RUN 0.24 uaic1302roro 0.23 uaic1306roro 0.23 onto1301bgbg 0.22 onto1302bgbg 0.22 onto1303bgbg 0.22 onto1304bgbg 0.22 btbn1301bgbg 0.19 evma1302enen 0.19 270 284 240 257 284 284 284 274 245 246 222 230 238 267 244 230 234 236 282 282 233 284 284 240 277 284 249 284 284 284 270 284 284 224 274
Table 15 shows the difference in performance for each type of question. Notice that consistently, all systems perform better on the auxiliary questions that require less inference than main questions. Also results over questions with modality and negation are lower for all runs than the score obtain for all questions.
However, the most important result is that scores show how systems can’t decide whether there is a correct answer or not among candidates. This is a very important challenge that we have to continue addressing in future. 0 0 0 0 0 0 0 0 0 0 0
Finally, Table 16 compares the performance of systems in the three editions of QA4MRE. Results show how introducing NCA questions together with modality and negation made the task more difficult this year.
Table 17 below shows the percentage of correct and NoA answers for different question types over the last three years. Percentages of correct answers are lowest for Causal questions at 22.56% with Purpose following at 24.19%. Which-istrue and Factoid are similar at 25.44% and 25.92%, while the highest is Method at 30.64%. Similar to last year, the Causal questions are the hardest to answer. This corresponds to the intuition that the need for inference in such questions can cause difficulties for systems. However, while Method questions were the most difficult questions after Causal ones last year, this time around the Method questions seems to be the easiest. It may be that easier Method questions were set this year. NoA scores are similar across question types although, interestingly, the number is lowest at 6.32% for Causal questions even though these were the hardest. It seems therefore that for Causal questions, systems were less inclined to withhold their answers than for other types, but then in answering such questions they were less successful than for other types.
As stated above, various Auxiliary questions were added to the test set, each such question being a simplification of a particular Main question. Simplifications took three main forms: hypernym replacement, noun phrase synonymy and verbal entailment: 16 were hyponym replacement (HYP), 18 were noun phrase synonymy (NPS), and 10 were verbal entailment (VEN).
In total there were 44 Auxiliary questions, seventeen being simplifications of Main questions themselves and 27 being simplifications of the correct answers to Main questions. The simplifications were designed to identify the (in)ability of a system to perform specific inferences; essentially, the main question required the inference while the auxiliary one did not. In consequence, we were looking for instances where systems found auxiliary questions easier to answer than their main counterparts. We identified two means of studying the data. First, we looked at how many systems answered main questions correctly as against how many answered the corresponding auxiliary ones correctly (Tables 15 and 17). We expected to see more systems answering auxiliary questions correctly, if indeed they were easier to answer. Secondly we looked at how many main-auxiliary question pairs had the property that more systems answered the auxiliary question correctly than answered the main question correctly. These results are shown in Tables 16 and 18. We were expecting most pairs to have this property.
Overall, our expectations were fulfilled since Auxiliary question simplification led to a score increase in 36 out of 44 cases (81.82%): 13 out of 17 for question simplification (Table 16) and 23 out of 27 for answer simplification (Table 18). Where scores increased, they did so strongly, by 244.21%: in other words there were about two-and-a-half times as many correct responses on average for Auxiliary questions relative to their corresponding Main questions, in cases where there was any increase. This appears to support our hypothesis that certain key inferences were causing systems to get answers wrong. There was a big score difference between simplifying the Question and simplifying the Answer. Question simplisimplification led to a 58.96% increase, while Answer simplification gave 348.91%. It appears that answer simplification makes a question much easier to answer than question simplification.
Question Simplification. The breakdown of this by simplification type is shown in Tables 18 and 19. These tables only consider main questions that have an Auxiliary counterpart. Table 18 shows the counts of Main questions correct, the equivalent Auxiliary questions correct and the percentage difference. These figures are then broken down by simplification type (HYP, NPS or VEN) in the last nine columns. Recall that question type and domain are two different ways of breaking down the same set of question pairs.
For operational reasons there no auxiliary questions for the Alzheimer’s topic. Table 18 shows that the overall percentage difference between main and auxiliary in correct answers was 25%. Concerning topic, Climate was the lowest (9%) and Aids the highest (39%). Concerning question type, Cause was the highest at 52% while Purpose, Fact and True fall in the range 20-30%. This suggests that Cause questions require the most complex reasoning. Interestingly, Method questions were worse in the auxiliary case. Problems in formulating the auxiliary questions could be the reason here.
Aux HYP main aux
Aux NPS main aux
Aux VEN main aux
Overall main aux total total total % 20 0 0 0 0 20 0 0 20 0 20
Answer Simplification. The breakdown of scores by answer simplification type is shown in Tables 20 and 21 which are analogous to Tables 18 and 19. Table 20 shows that the overall percentage difference between main and auxiliary in correct answers was 122%, considerably more than for question simplification. Concerning topic, Climate was once again the lowest (18%) and Aids the highest (179%), with Music close behind (167%). Concerning question type, Cause was now the lowest at 41% while the highest was True (187%). True (i.e. which-is-true) questions often ask for a difficult choice between statements about the text, statements that can take many different forms. So it is reasonable to expect a big improvement here. Concerning Table 21, this once again shows a trend of increasing scores for the auxiliary questions.
Aux HYP main aux
Aux NPS main aux
Aux VEN main aux aux total
To summarize, the aim was to see if simplifications of a question would increase a system's performance. The indications are that this actually occurred. The implication is that the Auxiliary question approach could be used to dig deeper into the exact workings of a system and in particular the performance of individual components within that system, while keeping with the multiple choice answer format which allows complex questions but still permits automatic evaluation. However, this was a pilot study only and reservations should be noted: Firstly, there were only 44 Auxiliary questions out of 240 Main questions which is only 18.33%, and 44 is not a big enough number to comprise a representative sample; Secondly the distribution of simplification types was not controlled; Thirdly, the exact nature of simplifications was not that closely specified or validated, as it is a very complex matter and this was a small part of the project; Finally, in some cases, the simplification substituted a direct substring of the text, which systems could then match using string processing and hence answer correctly. Such a substitution might possibly not be pinning down the lack of an inference at all but simply turning QA into string comparison.
Subject to the above remarks, this pilot study did seem to identify strong effects and to set out a framework which could be refined in future evaluation frameworks. This suggests a more systematic study and analysis of Question and 0 46 20 16 33 115 0 23 37 55 115 total total 0 3 3 0 3 9 0 0 0 9 9 % 0 170 60 0 636 172 0 0 0 172 172 Answer simplification in future years, using additional simplification operations, and hence allowing system builders to pinpoint the strengths and weaknesses of their systems.
This task tries also to promote the use and combination of external sources of knowledge in order to help answering questions as it has been said above. This year participants were allowed to submit a maximum of 10 runs. Run 01 had to be produced using only the Background Collection provided by the organizations—no external resources were allowed. Participants that did not use the Background Collections submitted their runs starting from number 02. Runs 02 to 10 were permitted to make use of any additional resources. Out of 11 groups, 6 submitted also the first run. These runs can be seen below in Table 22 (extracted from Table 13 above). RUN_NAME (...1 Runs) nara1301enen buap1301enen evma1301enen vens1301enen btbn1301bgbg onto1301bgbg Average:
RUN_NAME (other runs) nara1302enen buap1309enen evma1303enen vens1302enen btbn1302bgbg ontol1302bgbg
On the left pair of columns are the submitted ‘1’ runs that were only permitted to use the Background Collection and no other source of knowledge. On the right pair of columns are the best non-‘1’ runs submitted by the same groups. It is clear that the average c@1 scores for the ‘1’ runs (0.23) and the non-‘1’ runs (0.22) are very similar. Viewed individually, nara, vens, btbn and ontol were all better when using just the Background Collection, with the biggest difference being nara which scored 0.24 with non-‘1’ and 0.4 more (0.28) with ‘1’. nara was also the best scoring run in this group (but not the best overall as some groups did not submit ‘1’ runs) and did seem to gain some benefit from the Background Collection. On the other hand, buap and evma were worse in ‘1’ than non-‘1’ though the difference was only 0.1 in each case. The differences, whether increases or decreases, are small except for nara, so it is hard to decide whether the Background Collections are beneficial to systems or not.
Generally the use of the Background Collections on the one hand, and how to measure such use on the other hand, remain unanswered questions. Systems could be asked to ‘prove’ that they have used a background document by for example quoting a supporting passage from it, but it is hard to prevent such use from being reverse engineered once the system has first found the required information elsewhere. In addition, the extraction of simple supporting passages is not the only valid use to which a background collection can be put; the use could be more intangible, such as extracting statistical data or causal rules.
Finally, in considering the ‘1’ runs it is important to remember that the best runs overall by c@1 on main questions were jucs1302enen (0.55) and kule1306enen (0.3), neither of which submitted ‘1’ runs. 7.6
The table in Appendix 3 summarises the set of techniques that participants have reported are being used in their systems. A more detailed explanation of each system is given by participants in the Working Notes.
Most of systems perform question analysis as it was shown in last editions. However, while in the last edition questions patterns were automatically obtained, this year's participants seem to prefer to create patterns manually.
The most common techniques applied for processing texts were, as usual, PoS tagging, NER and dependency analysis in a lower proportion. Nevertheless, participants did not report the application of deeper analysis techniques, except the vens system, which uses semantic role labelling for its logic representation. Therefore, it seems systems continue relying on lexical and simple syntactic analysis, which do not allow all the phenomena in language to be captured and limit the final performance of systems.
The task this year was significantly harder than in previous years, due to the introduction of NCA questions, and questions with modality and negation issues. While this year’s results show some improvement compared to first year, especially with respect to the respective baselines, the majority of systems are still far from being able to pass a Reading Comprehension test. Nevertheless, best systems are, in general, very close to achieving this goal.
The NoA option (i.e., system unable to determine with enough confidence an answer) shows an interesting trend. Comparing the overall NoA performance over the three years of QA4MRE. It is quite striking that the percentage of NoA answers returned by systems seems to halve every successive year: from around 40% in 2011 to under 20% to under 10% this year. This may suggest that systems are becoming more confident in their answering ability and hence more reluctant to use NoA answers unless they are sure these are appropriate. But the fact is that in most cases systems increase the proportion of wrong answer when they decide to give answer.
When we defined the task we kept in mind three main ideas: that we are developing a validation technology able to determine if a particular answer is correct or not; that knowledge is crucial for understanding; and that a large set of documents related to a topic could be an additional source of background knowledge. We discuss each in turn.
Regarding the second and third issues, results suggest that the use of external resources helps in general to improve results, although not so clearly in the case of Background Collections. Most participants do not seem to know how to gather usable background knowledge from these collections, while it seems that other external resources provide greater benefit. We need to decide whether to continue collection Background Collections, since the organization is spending a lot of resources doing so every year2. Somehow, we expected to gather some attention to Open Information Extraction and similar research fields aimed at acquiring knowledge from textual sources to enable textual inferences.
The first question is whether the technology developed so far is just ranking the options or is actually validating them. The difference is important: What happens if we don't provide the options? Most systems use a kind of similarity measure or they don’t use validation at all. Thus, more than validating the answers, systems are ranking them. This led us to introduce a change this year: an explicit assessment of the ability to reject candidate answers when they are incorrect, using the “None of the answers above are correct” option. Maybe due to the novelty or to a surprise effect, the fact is that systems performed consistently worse over these questions. Given the fact that 39% of questions were of this type, none of the systems except one, was able to achieve this baseline.
It is important to notice the difference between NCA questions and NoA responses. Systems should use NoA response when the risk of choosing a wrong answer is high. In order to choose NCA option as response, systems must be able to find evidences about the incorrectness of the candidate answer. This must lead research towards the development of the ability to reject answers more than the ability to accept them. This is in accordance to the main QA scenario were we expect some hypothesis over-generation that answer validation modules must manage. For this reason, we’ll work in future about how the evaluation methodology can reward systems with this desirable feature.
Anselmo Peñas and Alvaro Rodrigo work has been partially supported by the Spanish Government (MINECO) in the framework of CHIST-ERA program (READERS project), and the Regional Government of Madrid, through the project MA2VICMR (S2009/TIC1542). Pamela Forner work has been partially supported by the PROMISE Network of Excellence (258191). Special thanks are due to Giovanni Moretti (CELCT, Trento, Italy) for the technical support in the management of all data and evaluation scripts of the campaign.
2 Note Google’s API wasn’t available for research purposes. This significantly increased our collection work.
1 Anselmo Peñas, Pamela Forner, Richard Sutcliffe, Álvaro Rodrigo, Corina Forascu, Iñaki Alegria, Danilo Giampiccolo, Nicolas Moreau, Petya Osenova. Overview of ResPubliQA 2009: Question Answering Evaluation over European Legislation. In C. Peters, G. di Nunzio, M. Kurimo, Th. Mandl, D. Mostefa, A. Peñas, G. Roda (Eds.). Multilingual Information Access Evaluation Vol. I Text Retrieval Experiments. Workshop of the Cross-Language Evaluation Forum. CLEF 2009. Corfu. Greece. 30 September - 2 October. Revised Selected Papers. Lecture Notes in Computer Science 6241. Springer-Verlag. 2010.
APPENDIX 1: Overall results at TOPIC level: Median, Average, and Standard Deviation for all runs Run jucs1302enen kule1304enen kule1305enen kule1302enen buap1301enen kule1303enen kule1306enen nara1301enen buap1304enen buap1305enen kule1308enen buap1303enen buap1309enen kule1309enen buap1306eses csui1302enen csui1304enen csui1305enen csui1307enen kule1307enen nara1302enen csui1303enen csui1306enen csui1308enen csui1309enen buap1307eses buap1308eses csui1310enen lims1304enen lims1305enen onto1305bgbg AVERAGE MEDIAN onto1307bgbg onto1308bgbg evma1303enen lims1302enen lims1303enen onto1306bgbg uaic1303roro buap1302enen buap1310enen evma1301enen vens1301enen vens1302enen_ LATE_RUN uaic1302roro uaic1306roro onto1301bgbg onto1302bgbg onto1303bgbg onto1304bgbg btbn1301bgbg evma1302enen btbn1302bgbg uaic1304roro uaic1305roro STANDARD DEVIATION 0,26 0,25 0,25 0,25 0,24 0,24 0,24 0,24 0,24 0,23 0,23 0,22 0,22 0,22 0,22 0,19 0,19 0,18 0,17 0,17 Run jucs1302enen kule1304enen kule1305enen kule1302enen buap1301enen kule1303enen kule1306enen nara1301enen buap1304enen buap1305enen kule1308enen buap1303enen buap1309enen kule1309enen buap1306eses csui1302enen csui1304enen csui1305enen csui1307enen kule1307enen nara1302enen csui1303enen csui1306enen csui1308enen csui1309enen buap1307eses buap1308eses csui1310enen lims1304enen lims1305enen onto1305bgbg AVERAGE MEDIAN onto1307bgbg onto1308bgbg evma1303enen lims1302enen lims1303enen onto1306bgbg uaic1303roro buap1302enen buap1310enen evma1301enen vens1301enen vens1302enen _LATE_RUN uaic1302roro 0,59 0,35 0,35 0,34 0,33 0,33 0,33 0,33 0,32 0,32 0,32 0,31 0,31 0,31 0,30 0,30 0,30 0,30 0,30 0,30 0,30 0,29 0,29 0,29 0,29 0,28 0,28 0,28 0,28 0,28 0,28 0,28 0,28 0,27 0,27 0,26 0,26 0,25 0,25 0,25 0,24 0,24 0,24 0,24
C@ 1 r_1 0,80 0,45 0,40 0,47 0,23 0,48 0,47 0,57 0,53 0,30 0,42 0,47 0,27 0,40 0,33 0,51 0,51 0,51 0,45 0,37 0,36 0,45 0,48 0,48 0,48 0,33 0,33 0,48 0,33 0,33 0,28 0,38 0,38 0,33 0,33 0,48 0,33 0,33 0,27 0,47 0,20 0,13 0,34 0,33
C@ C@1 1 r_2 r_3 0,83 0,60 0,33 0,28 0,33 0,33 0,33 0,27 0,24 0,14 0,30 0,16 0,27 0,40 0,17 0,23 0,33 0,07 0,36 0,08 0,27 0,30 0,27 0,20 0,33 0,20 0,27 0,27 0,23 0,20 0,33 0,20 0,33 0,20 0,38 0,21 0,27 0,25 0,23 0,19 0,25 0,16 0,28 0,20 0,28 0,25 0,33 0,23 0,28 0,21 0,20 0,38 0,20 0,38 0,28 0,21 0,20 0,27 0,27 0,20 0,32 0,00 0,27 0,20 0,27 0,20 0,28 0,07 0,27 0,07 0,14 0,19 0,27 0,27 0,20 0,20 0,27 0,00 0,21 0,23 0,27 0,20 0,20 0,27 0,23 0,20 0,33 0,33 uaic1306roro onto1301bgbg onto1302bgbg onto1303bgbg onto1304bgbg btbn1301bgbg evma1302enen btbn1302bgbg uaic1304roro uaic1305roro STANDARD DEVIATION x s e s y l a n A n o i t s e u Q o N x x x s n r e t t a P e n o d y l l a u n a M x x r e h t
O Machine learning deep question linguistic processing x x x x x x x g n i g g a T h c e e p S f O tr a P x g n i k n u h C s m a r g n n o i t i n g o c e R y t i t n E d e m a N s n o i s s e r p x e l a r o p m e
T x x x x
x x x x x x x x x x x x s n o i s s e r p x e l a c ir e m u N x s n o i t a m r o f s n a r t c i t c a t n y S g n i s r a p c i t n a m e S g n i l e b a l e l o r c i t n a m e S n o i t a l e R f O s t e S d e n if e d e r P s e m a r F n o i t a t n e s e r p e r c i g o l r e v o r p m e r o e h
T s n o i t a m r o f s n a r t e s a r h P x s i s y l a n a y c n e d n e p e D ) c t e .j b o . b u s ( s n o i t c n u
F x x x
x x x x x e n o N r e h t
O Stemmer, lemmatisat ion Anaphora resolution Anaphora Resolution Stemming, stop-word filtering x x
x x x x x x x x x x x x x x x x
x x x x x x x x x x x x x x x x x x n o i t a d i l a v r e w s n a o N x x x n o i t p i r c s e D m e t eSm ehT ism syS
The system search for answers on the basis of dependency triples.
The system uses information retrieval techniques and a graph based representation to find similarity features vector between the answers and support text extracted from the documents.
The system performs two different approaches to determine the answer of a question. For factoid question, the system will use the list of named entities obtained from the passage that is relevant to the query.
This year we are testing a new system. The approach is similar to that used in the previous year, based on superficial analysis of the text, supplemented with POS and NER.
The system used textual entailment based answer validation technique. Per topic, one multi-document summary was generated form the background collection provided by the organiser. Then the summary is also used to select the correct answer.
Simple system employing set similarity metrics.