This paper describes the Question Answering for Machine Reading (QA4MRE) task at the 2012 Cross Language Evaluation Forum. In the main task, systems answered multiple-choice questions on documents concerned with four different topics. There were also two pilot tasks, Processing Modality and Negation for Machine Reading, and Machine Reading on Biomedical Texts about Alzheimer's disease. This paper describes the preparation of the data sets, the definition of the background collections, the metric used for the evaluation of the systems' submissions, and the results. Eleven groups participated in the task submitting a total of 43 runs in seven languages. Reading Comprehension tests are routinely used to assess the degree to which people comprehend what they read, so we work with the hypothesis that it is reasonable to use these tests to assess the degree to which a machine “comprehends” what it is reading.
When reading a text, a human performs two processes, namely: 4. Inferences that require composing several answers, in particular answering one part of the question using the background collection and then, with its answer, answering the other part of the initial question (e.g., “Who is the wife of the person who won the Nobel Peace Prize in 1992?”).
In 2012, we had a main task and two pilot exercises.
Main Task This remained the same for participants. Background collections, test documents and reading tests were available in Arabic, Bulgarian, English, German, Italian, Romanian, and Spanish. In addition to last year's topics (AIDS, Climate Change, Music and Society), we included a topic on Alzheimer's disease. This new topic is related to a new pilot on Biomedical texts. The difference is that the reference collection for the main task is built from general public sources and for the pilot the source is the PubMed repository.
Having these two parallel exercises about the same topic but in different domains opens the door to evaluate research approaching the challenges of domain and language adaptation, the use of knowledge in one domain captured in the other, the differences in the background knowledge acquired, the differences between questions and answers in each domain, etc.
Pilot on Processing Modality and Negation for Machine Reading. This exercise is aimed at evaluating whether systems are able to understand extra-propositional aspects of meaning like modality and negation. Modality is a grammatical category for expressing the attitude of the speaker towards his/her statements, such as expressions of certainty, factuality, and evidentiality. Negation is a grammatical category that allows changing the truth value of a proposition. In the pilot, participants received some texs where they have to decide whether some events are Asserted, Negated, or Speculated. Our plan is to integrate modality and negation in the main task next year.
Machine Reading on Biomedical Texts about Alzheimer's disease. This exercise is aimed at setting questions in the Biomedical domain with a special focus on one disease, namely Alzheimer's. This pilot task explored the ability of a system to answer questions using scientific language. Texts were taken from PubMed Central related to Alzheimer's and from 66,222 Medline abstracts. In order to keep the task reasonably simple for systems, participants were given the background collection already processed with Tok, Lem, POS, NER, and dependency parsing.
The two pilot tasks are described in detail in dedicated papers in these proceedings.
Tests were divided into: - 4 topics, namely “Aids”, “Climate change”, “Music and Society” and “Alzheimer”; - Each topic had four reading tests; - Each reading test consisted of one single document, with 10 questions and a set of five choices per question.
Overall, the following evaluation setting was proposed: - 16 test documents (4 documents for each of the four topics), - 160 questions (10 questions for each document) with , - 800 choices/options (5 for each question).
Test documents and questions were made available in English, German, Italian, Romanian, and Spanish and newly this year also in Arabic and Bulgarian. These materials were exactly the same in all languages, created using parallel translations.
This is a very important element of the evaluation setting. It connects the task also with the research in Information Retrieval. The goal of reference/background collections is to contextualize the reading of a single document related to the topic by collecting and fleshing out additional pertinent information. In the future this step may be done on the fly as a retrieval process once a single test text is provided. However, for now, we provide a carefully constructed background corpus for two main reasons: to allow more comparison among participant systems, and to focus on the Reading Comprehension problem. We believe it is important to develop a good methodology for building background collections for the evaluation task.
We define background knowledge in terms of the relation between the testing questions and answers, and the background collection. To determine the potential kinds of uses of the prior knowledge, we distinguish at least four main types of background knowledge (although in fact it’s a continuum): 1. Very specific facts related to the document under study. For example, the relevant relation between two concrete people involved in a specific event. 2. General facts not specific to any particular event. For example, geographical knowledge, main players in international affairs, movie stars, world wars. Also acronyms, transformations between quantities and measures, etc. 3. General abstractions that humans use to interpret language, to generate hypotheses or to fill missing or implicit information. For example, abstractions such as the result of observing the same event with different players (e.g. petroleum companies drill wells, quarterbacks throw passes, etc.) 4. Linguistic knowledge. For example, synonyms, hypernyms, transformations such as active/passive or nominalizations. Also transformations from words to numbers, meronymy, and metonymy. Obviously this is not an exhaustive list. For example, we do not include ontological relations that enable temporal and spatial reasoning, or reasoning on quantities, which are also all relevant.
Ideally, the background collection should cover completely the corresponding topic. This is feasible sometimes and unrealistic at others. For example, in the case of the pilot on Biomedical documents about Alzheimer's disease, a set of experts built a query (a set of conjunctions and disjunctions over 18 terms) that approximates very much the retrieval of all relevant documents (more than 66,000) without introducing much noise. However, this is not so easy in more open domains (e.g., Climate Change) or cases with non-specialized sources of information. In these cases, we crawl the web using, for each language and topic a list of keywords and a list of sources. Keywords are translated into English and then translated into the rest of the languages. Documents may be crawled from a variety of sources: newspapers, blogs, Wikipedia, journals, magazines, etc. The web sources are obviously language dependent, and each language also requires a list of possible web sites with documents related to the topic.
We realized in the past edition that, since we organizers knew the test set, we used that information to select the keywords, and ensure the coverage of the questions. The effect is not only that background collections don’t cover completely the topic, but also that the collections have some bias with respect to the real distribution of concepts. In this year's campaign, the assumption that the ideal background collection should include all relevant documents for the topic (and only them) is explicit, and we organizers bear it in mind. Thus, we face the same problem as traditional Information Retrieval: we want all relevant documents (and only them), and we use queries (keywords) to retrieve them The first strategy with the aim of ensuring the coverage of the topic as much as possible is to make the topic specific enough (e.g., AIDS medicaments rather than AIDS). The second strategy is to try to cover (at least partially) each of the possible “dimensions/aspects” of that topic. How? First, by detecting a good central overview text, such as a Wikipedia article that “defines” the topic, “suggests” its principal aspects, and provides links to additional good material. Then, organizers enumerate these dimensions and prepare a set of queries for each dimension. They document this process with three benefits: (i) to know what organizers and participants can expect or not from the collection; (ii) to give another dimension of re-usability; and (iii) to explore how Machine Reading will connect to Information Retrieval in the future.
TOPICS ALZHEIMER CLIMATE CHANGE MUSIC & SOCIETY 8,790 docs Next Table shows the keywords used for each topic. They are a sort of more concrete definition of each topic, giving an idea of the subtopics covered by the collection.
ALZHEIMER KEYWORDS CLIMATE CHANGE KEYWORDS (EXTENSION) Alzheimer's AND Alzheimer's disease Alzheimer's drugs Alzheimer's symptoms Alzheimer's treatment Alzheimer's causes senile dementia memory loss (memory testing OR neuropsychological tests) for Alzheimer brain disorder AND neurological disorder plaques and tangles Lewy bodies mental confusion AND Alzheimer wandering AND Alzheimer irritability AND Alzheimer sundowning depression AND Alzheimer (language problems OR aphasia) AND Alzheimer (perception problems OR agnosia) AND Alzheimer (disorder of motor planning OR apraxia) AND Alzheimer personality changes AND Alzheimer beta-amyloid (caregiving OR long-term care) AND Alzheimer nursing home AND Alzheimer (aging society OR geriatrics) AND Alzheimer healthcare costs AND Alzheimer cognitive reserve theory Auguste Deter Danae Chambers Alzheimer's Association Alzheimer diagnosis Alzheimers' associated disorders Alzheimers' clinical features Alzheimers' genetics Alzheimers' prevention Familial Alzheimer's Alzheimers' risk factors impact of Alzheimer's disease Neuropathology of Alzheimer's Disease solar radiation carbon capture fluorinated gases drought heat-trapping gases ground-Level ozone wind power biofuel gas emissions biomass AIDS KEYWORDS (EXTENSION) HIV/AIDS funding AIDS global crisis TRIPS Agreement AIDS pharmaceutical industry World Health Organization AIDS family planning AIDS pandemic AIDS life expectancy rate fighting AIDS AIDS virology MUSIC AND (EXTENSION) music criticism musicology history of violin technique music patronage rock and roll history of song electric musical instrument classical recording industry economics of classical music classical crossover music
SOCIETY
This year the datasets was created for the following seven languages: Arabic, Bulgarian, English, German, Italian, Romanian and Spanish. The dataset was created following the methodology developed last year consisting of the following steps: 1. Four English documents were selected for each of the four topics (Aids, Climate Change, Music and Society, Alzheimer's). These were selected from copyright-free sources (see Table 2) and these represented the test documents against which questions were asked. 2. In order to have a set of identical questions for the seven languages above, we needed to have the selected test documents translated. For this purpose, expert translators were recruited form the Translation for Progress1 platform for all languages. On the whole, 57 translators were contacted and asked to perform the translations in a couple of weeks’ time. Most of the translations were of a high quality and were delivered within the agreed timescale. 3. To ensure that translations were faithful to the original document in both meaning and style and of good quality, all the documents were manually checked and corrected when necessary. We wanted to avoid a situation where portions of the original English text were left out of the translation in a particular target language, or perhaps modified or interpreted in a particular manner which would have made the question impossible to answer in that language. 4. Ten multiple-choice questions were then devised for each test document. A question always had five candidate answers from which to choose, with one clearly correct answer and four clearly incorrect answers.
Once the questions had been composed in the language of the original author, each was then translated into English. The English versions of the questions and candidate answers were carefully checked by a referee to verify that they were clear, that the intended answer was clearly correct, that the intended answer was in the test document, and that the other candidate answers were clearly incorrect. Questions were modified accordingly. 6. The English versions were then used to translate each question into each of the seven languages of the task. The same process was used to translate each candidate answer (five per query) into the seven languages. 7. The result of this process was a set of 160 questions in seven languages, each with five multiple-choice answers, also in those seven languages. The final step was to check that the answer to each question was in fact present in the test document for all the languages of the task.
1 http://www.translationsforprogress.org/main.php A Translation Exchange site linking volunteer translators (e.g., linguistics students or professionals in foreign languages interested in building experience as translators can link up with low-budget organizations who are in need of translation work, but without the budget to pay for it. There are currently over 1450 registered volunteer translator members (for 13 language combinations) and over 160 organization members. Translation for Progress database is open for viewing for the general public, but if you wish to post your profile or contact a volunteer translator, a registration is required.
AIDS AIDS AIDS Climate Change Climate Change Climate Change Climate Change Music
& Society Music
& Society Music
& Society 2 3 4 5 6 7 8 9 10 11 http://archive.icommons.o rg/articles/pipelinepatents-compulsorylicensing-and-the-costsof-aids-treatment-in-brazil
Paula Martini http://www.fpif.org/report s/hivaids_in_africa_time_ to_stop_the_killing_fields
Chinua Akukwe and Melvin Foote, http://www.fpif.org/article s/african_women_confron t_bushs_aids_policy http://chevyvolt.cm.fmpu b.net/#http://boingboing.n et/2011/08/05/3-thingsyou-need-to-know-aboutbiofuels.html http://www.scidev.net/en/ policy-briefs/brazilclimate-change-a-countryprofile.html http://www.energybulletin .net/node/51370 http://www.scidev.net/en/ climate-change-andenergy/biofuels/opinions/r eality-check-for-miraclebiofuel-crop.html http://www.archive.org/str eam/encyclopaediabri04c hisrich/encyclopaediabri0 4chisrich_djvu.txt
Yifat Susskind Maggie KoerthBaker Emilio Lèbre La Rovere and André Santos Pereira Maude Barlow Miyuki Iiyama and James Onchieku Chisholm, Hugh (Ed.) http://www.bos.frb.org/ec onomic/nerr/rr2003/q2/re quiem.htm
Julie Lee http://en.wikipedia.org/wi ki/Pop_music
Unknown
Pop Music Pipeline patents, compulsory licensing and the costs of AIDS treatment
in Brazil "HIV/AIDS in Africa: Time to Stop the Killing Fields" (Washington, DC: Foreign Policy In Focus, October 6, 2005) "African Women Confront Bush’s AIDS Policy" (Washington, DC: Foreign Policy In Focus, December 2, 2005) 3 things you need to know about biofuels Brazil & climate change: a country profile To curb climate change, we need to protect water Reality check for 'miracle' biofuel crop
Requiem for Classical Music Charles Burney
Public Domain Alzheim
er Alzheim
er Alzheim
er Alzheim er 12 13 14 15 16 http://www.gutenberg.org /files/14884/14884h/14884-h.htm#page31 http://knol.google.com/k/l ara/alzheimer-sdisease/Ing3XNE/g1JpHQ# http://knol.google.com/k/ gloria-hschneider/creativityalzheimer-sdisease/1v6cy64kp9uk1/7 8# http://knol.google.com/k/ elder-care-elder-rageknow-the-warning-signsof-alzheimer-s http://knol.google.com/k/s tan-goldberg/it-s-onlyalzheimer-s-not-thebloody/32wlgicpxht73/5#
Henry C. Lahee Bruce Miller; Lara Heflin, Gloria Ha'o Schneider Jacqueline Marcell Stan Goldberg
Famous Violinists of ToDay and Yesterday Alzheimer's Disease Creativity & Alzheimer's Disease Caring for Aging Parents & Elder Rage: Know The Warning Signs of Alzheimer's! It's Only Alzheimer's, Not the Bloody Plague! Creative Commons Attribution Creative Commons Attribution
For each text in the test set 10 multiple choice questions were created. Each question had five answer options. The questions covered five different question types: purpose, method, causal, factoid, and which-is-true. Factoid questions were divided into the following sub-types: Location, Number, Person, List, Time and Unknown. Examples of the basic question types are given below. We took care to spread the question types evenly for a given test document, aiming for two questions per type. The exact breakdown of the number of questions per type in the test collection is provided in Table 3 below. Example questions: PURPOSE: What is the aim of Obama‘s cap-and-trade policy? METHOD: How could vast quantities of petrol be saved? CAUSAL: What is the reason for the high price of solar energy? FACTOID (time): When are bioethanol and biodiesel expected to become widely used? WHICH-IS-TRUE: Which of the following goals is Europe committed to? For all questions, the direct answer was contained in the test document; however answering the questions typically required some background knowledge and some form of inference. The required knowledge could be linguistic or could involve basic world knowledge. Linguistic knowledge concerns, for example, the ability to perform co-reference resolution or detect paraphrases on the lexical or syntactic level. World knowledge has to be inferred from the background collection. For instance, the text might mention Barack Obama while the question might refer to the first African American President. The fact that Barack Obama is the first African American President needs to be learnt from the background collection in order to be able to answer the question. Typical types of world knowledge involve, for instance, knowledge about the basic referents in a text, e.g., being aware that Yucca Mountain is in Nevada. Another type of world knowledge involves knowledge of “life scripts” such as “visiting a restaurant”. Finally, the inference required can also be complex, involving several steps. For example, answering a question might require combining knowledge from the background collection with knowledge from the test document itself. For instance, the question “Who is the wife of the person who won the Nobel Peace Prize in 1992?” contains two facts P and Q, where P=“wife of Y=?” and Q=“winner of Nobel Peace Prize in 1992=Y”. The latter information can be gleaned from the background collection whereas the former is contained within the test document itself.
For each test document, we aimed for a combination of simple, medium, and difficult questions. At most six questions per document did not require knowledge from the background collection. Two of these were simple questions, i.e., the answer and the fact questioned could be found in the same sentence in the test document. Four questions were of intermediate difficulty in that the answer and the fact questioned were not in the same sentence and could, in fact, be several sentences apart. Finally, the remaining four questions did require utilizing information from the background collection. While not all question types require inference based on the background collection, all of them required some form of textual and linguistic knowledge, such as the ability to detect paraphrases, as we made an effort to re-formulate questions in such a way that the answers could not be found by simple word overlap detection. For each question, we kept track of the inference required to answer it. This made it easier to ensure that that inference could in fact be drawn on the basis of the background collection, i.e., that the background collection did indeed contain the relevant fact. It also makes it possible to carry out further analyses regarding which questions or types of questions were difficult for the systems and why. When creating the questions, we took care not to introduce any artificial patterns that would help finding the correct answer. Thus we ensured that all answer choices for a question were approximately the same length and consistent with respect to formulation and content, that all of the wrong answers were plausible, and that the placement of the correct answers was random and balanced.
This year, CELCT developed a series of infrastructure components to help manage the QA4MRE exercise. Many processes and requirements were to be dealt with: o o
The need to develop a proper and coherent tool for the management of the data produced during the campaign, to store it and to make it re-usable, as well as to facilitate the analysis and comparison of results; The necessity of assisting the different organizing groups in the various tasks of the data set creation and to facilitate the process of collection and translation of questions; The possibility for participants to directly access the data, submit their own runs (this also implied some syntax checks of the format), and later, get the detailed viewing of the results and statistics. A series of automatic web interfaces were specifically designed for each of these purposes, with the aim of facilitating the data processing and, at the same time, showing the users only what they needed for the task they had to accomplish. The main characteristic of these interfaces is the flexibility of the system specifically centred on the user’s requirements.
While designing the interfaces for question collection and translation, one of the first issues to be dealt with was the fact of having many assessors, a big amount of data, and a long process. So tools must ensure an efficient and consistent management of the data, allowing:
Alteration of the data already entered at any time.
Revision of the data by the users themselves.
Consistency propagation ensuring that modifications automatically re-model the output in which they are involved.
Real time calculation of statistics and evaluation measures.
In particular, ensuring the consistency of data is a key feature in data management. For example, if a typo is corrected in the Translation Interface, the modification is automatically updated also in the Gold Standard files, in the Test Set files, etc.
Since one of the objectives of the task is to assess the ability of systems to understand texts through their answers to questions about those texts, the evaluation focuses on measuring this understanding by computing the correctness of the responses given to the multiple-choice tests. Furthermore, we follow the line introduced in ResPubliQA 2009 [1] of promoting the development of systems able to reason about the correctness of their responses with the aim of reducing the amount of incorrect answers given as output. Thus, this year’s evaluation remains quite similar to the one of the last edition.
Given a question with its corresponding candidate answers, a participant system can return two kinds of responses: o
Given these assessments, we decided to evaluate systems from two different perspectives:
A question-answering approach, as in the traditional evaluation performed in past campaigns, where we just evaluate the ability of systems answering a set of questions.
A reading-test evaluation, obtaining figures for each particular reading test and topics. This perspective permits us to evaluate whether a system was able to understand a document and to what degree. An answer selected from the set of candidate ones for that question, A NoA answer. This response is given when the system considers it is not able to find enough evidence about the correctness of candidate answers and it prefers not to answer the question instead of giving an incorrect answer. Thus, it gains some partial credit proportional to the performance shown with the answered questions. Moreover, the system can return as a hypothetical answer the candidate one that it would have been selected, which allows us to give some feedback about its validation performance. The assessments of system’s responses are given automatically by comparing them against the gold standard collection with human-made annotations. Therefore, no manual assessment was required, which reduces the effort of the evaluation once the collections have been created and facilitates the future development of systems. Each system’s response to a question receives one and only one of the following three possible assessments: Right if the system has selected the correct answer among the set of candidate ones of the given question; Wrong if the system has selected one of the wrong answers; NoA if the system has decided not to answer the question. Where the system returned a hypothetical answer, this answer was assessed as NoA_R in the case of it being correct or NoA_W if it was wrong.
We use c@1 as the main evaluation measure for this year's campaign. c@1 was first introduced in ResPubliQA
2009 [1] and is fully described in [2]. The formulation of c@1 is given in Formula (
nR: number of questions correctly answered.
nU: number of questions unanswered.
n: total number of questions
accuracy
n R nUR
n
(
n: total number of questions where where 5.2 Question Answering Perspective Evaluation In the Question Answering perspective we measure systems’ performance over a set of questions without considering the ability of a system to understand a certain document. This is an approach similar to the one applied in QA@CLEF campaigns before 2010.
The information considered for each system at this level is:
Total number of questions ANSWERED. This number is divided into: o total number of questions ANSWERED with a RIGHT answer, o total number of questions ANSWERED with a WRONG answer.
Total number of questions UNANSWERED (a NoA response was given). This number is divided into: o total number of questions UNANSWERED with a RIGHT candidate answer, o total number of questions UNANSWERED with a WRONG candidate answer, o total number of questions UNANSWERED with an EMPTY candidate answer.
This information is used for calculating the following scores:
An overall c@1 over the whole collection (a set of 160 questions),
A c@1 score for each topic (40 questions for each topic),
An overall accuracy score (over the 160 questions of the test collection, considering also the candidate
answers given to unanswered questions as it has been explained above),
The proportion of answers correctly discarded (see Formula (
correctly _ discarded nUW
nUE
nUR nUW
nUE
(
The objective of the reading perspective evaluation is to offer information about the performance of a system “understanding” the meaning of each single document. This understanding is evaluated by means of multiplechoice tests with ten questions per document.
This evaluation is performed taking as reference the c@1 scores achieved for each test (one document with its ten questions). Afterwards, these c@1 scores can be aggregated at topic and global levels in order to obtain the following values:
Median, average and standard deviation of c@1 scores at test level, grouped by topic,
Overall median, average and standard deviation of c@1 values at test level.
The median c@1 has been provided under the consideration that it can be more informative at reading level than average values. This is because median is less affected by outliers than average, and therefore, it offers more information about the ability of a system to understand a text.
This approach allows us to evaluate systems in a similar way to the manner new language learners are graded. Thus, we can consider that a system passes a test from this evaluation perspective if it achieves a score equal or higher than 0.5. In the case of obtaining an overall average c@1 higher than 0.5, we say that the system passes this evaluation perspective.
We propose here a simple baseline to which participants can be compared. Since participant systems can decide to answer or not to answer a given question, we must decide which behaviour must follow our baseline. For simplification purposes, the proposed baseline answers all the questions, randomly selecting each answer from the set of candidate ones.
This baseline has five possibilities when trying to answer a question: it can select the correct answer to the question, or it can select one of the four incorrect answers. Then, the overall result of this random baseline is 0.2 (both for accuracy and for c@1). Systems applying a certain kind of processing and reasoning should be able to outperform this baseline.
From an initial amount of 25 groups that registered to the main task and signed the license agreement to download the background collections, 11 of them finally submitted at least one run, resulting in 43 runs in 7 languages (Arabian, Bulgarian, German, English, Spanish, Italian and Romanian). Table 7 shows the number of runs per language.
There were only 3 cross-lingual runs and all from the same group. The language with the highest amount of runs was, as usual, English with 20 submissions, while Spanish and Italian received only one run per language. Thus, no comparison in these two languages can be performed. REGISTERED PARTICIPANTS 38
PARTICIPANTS DOWNLOADING THE TEST SETS
24 NUMBER of PARTICIPANTS MAIN BIOMEDICAL about ALZHEIMER MODALITY AND NEGATION
PARTICIPANTS SUBMITTING RUNS
TOTAL NUMBER OF RUNS AR 4 AR BG DE
EN ) s n t s e u q ( s g n la ES c IT e r o RO u S
Total 4 5 1 6 3 3 20 1 21
The mean values for all the tests where under 0.5, the value needed to pass the evaluation from the reading perspective. This result suggests that systems are still far away from obtaining satisfactory results according to this perspective. 2 It must be mentioned that there were 12 tests in QA4MRE 2011 We can see in Table 10 how results across tests in the same topic are more similar than in 2011, which suggests that this year’s collections are more homogenous. On the other hand, Table 10 shows the mean scores per topic. The scores across topics seem remarkably similar except for Topic 1 (AIDS), which seems to be a bit easier. uaic12062enen 0.29 0.28 0.28 0.26 0.25 0.65 0.40 0.38 0.35 0.34 0.33 0.31 0.31 The best results were obtained in English, where the highest score was obtained by jucs12013enen with 0.65. This value is 25 percentage points higher than the next system (vulc12014enen at 0.40). In fact, jucs12013enen was the only system able to pass the evaluation according to the reading perspective. This system obtained c@1 values over 0.60 for all the topics except for Topic 2 (Climate change). We can consider it a very good result if we compare that system with a person over such complex questions.
Regarding cross language runs, all of them were from the onto group over different target languages with Romanian as source, which does not allow to make any comparison. All these runs obtained the same result (0.29 of c@1).
This task tries also to promote the use and combination of external sources of knowledge in order to help answering questions as it has been said above. In order to study it, we asked participants to report the resources employed to assist in answering the questions and we summarise this information in Table 14. Only the test document and the associated background collection are used The test document and other resources are used, but not the background collection The test document together with both the background collection and other resources are used TOTAL of runs #of runs 53% of the submitted runs did not employ any kind of external resources, while 23% used only the background collection. The remainder of runs used additional resources, either with or without using the background collection. These observations suggest that the inclusion of such external sources and their exploitation is not yet widely adopted. Moreover, as shown in Table 14, more detailed information about the external sources used for each participant can be seen in Table 16 of Appendix 3.
A subsequent analysis of questions revels that questions requiring no extra knowledge were not much easier than the others. In fact, some of them seem to be considerably harder than some questions that require external resources. This observation suggests that in order to answer questions, the fact of having to compose two or more parts to form an answer is harder than just matching a single piece of text. However, whether the pieces of the answer are in the main text or in a background resource collection does not make much difference. It is more relevant for the performance how difficult the pieces are to match.
Regarding the linguistic processing, the most popular techniques were PoS tagging, the use of NER tools and dependency parsers, which were also some of the most applied techniques in previous editions. However, very few systems explore the use of deeper techniques relying on semantics, while only one relied on logic representation and a theorem prover.
Those two systems applying the most different techniques (jucs and idrq) were the ones that best performed in their languages (English and Arabic respectively). However, system vulc, which performs very well in English, reported only the use of phrase transformations. Therefore, it does not seem to be very clear which is the best combination of techniques in order to obtain a good performance. Evaluation frameworks such as the one presented in this paper must be used by researchers for exploring and answering such questions.
While this year’s results show some improvement compared to last year, specially respect to the respective baselines, the majority of systems are still far from being able to pass a Reading Comprehension test. Nevertheless, best systems are, in general, very close to achieve this goal.
When we defined the task we kept in mind three main ideas: that we are developing a validation technology able to determine if a particular answer is correct or not; that knowledge is crucial for understanding; and that a large set of documents related to a topic could be an additional source of background knowledge. We discuss each in turn: The first question is whether the technology developed so far is just ranking the options or it is validating them. The difference is important: What happens if we don't provide the options? Most systems use a kind of similarity measure or they don’t use validation at all. Thus, more than validating the answers, systems are ranking them. This leads to the need of a change for next campaign. Again, the option of gaining partial credit by leaving some questions unanswered and reduce the number of incorrect answers is not enough. We need to introduce an explicit assessment of the ability to reject candiate answers when they are incorrect. This could be done easily in our framework if we introduce a significant portion of questions where none of the options are correct and a last option in all questions “None of the answers above are correct”. If a significant portion of questions (up to 40%) have no correct answer among its options, this will give as a new baseline to beat: a dummy system that always chose there is no correct answer as default.
About the second and third issues, it seems that the use of external resources help to improve results, but this is not so clear in the case of background collections. Although we have refined the methodology to build the background collections this year, most participants don’t seem to know how to gather usable background knowledge from it. Moreover, it seems that the use of other external resources benefit more than the use of the background collections. We need to decide on this issue because the organization is spending a lot of resources in creating theses collections3.
This work has been partially supported by the Spanish Ministry of Science and Innovation, through the project Holopedia (TIN2010-21128-C02), and the Regional Government of Madrid, through the project MA2VICMR (S2009/TIC1542).
This work has been partially supported by the PROMISE Network of Excellence (258191). Special thanks are due Giovanni Moretti (CELCT, Trento, Italy) for the technical support in the management of all data and evaluation scripts of the campaign.
We would also like to acknowledge the volunteer translators that contributed to the creation of the dataset: Mercedes Marta Moreno; Juan Manuel Pérez Rojas; Adriana Pedemonte; Sophie; Ana Casillas Tomasin; Mayra Alvarez; MARÍA SOL ACCOSSATO; Natalia Steckel; María Constanza Galli; Fiorela; Hanna; Yessica V. Apolo Martínez; Fatima Alvarez; Alberto Mengibar Martin; Taras Giovanna; Pamela Aikpa Gnaba; Danielle; Marco Menegazzi; Alfredo Lo Bello; Chiara S.; Martina Scarano; Nunzio; Francesca Rubino; Katia G; Lucia Zirattu; Camilla Cosmelli; Sara Colombo; Chiara Gavasso; Katie M; Antti; Saskia Scharnowski; Jeffrey Bunce; Gabriele Mark; Melanie Liebchen; Helena Knaup; Judith Müller; Kathrin Meier; Anika Abel; Eva Wagle-Fopp; Franziska Bioh; Irina Rata; Nadia Bucurenci; Daniela Arsinel; Cristina Manoli; luminita Isaic; Nicoleta Mihaita; Mohamed ElGohary; Dina Awadallah; Nawel; Amal; Sarah; Shameem; Rabie Mustapha; Difaf Sharba; Amani; Khebouri Amina; Manel Rada. 7. Enhancing a Question Answering System with Textual Entailment for Machine Reading Evaluation. Adrian Iftene, Alexandru-Lucian Gînscă, Mihai Alex Moruz, Diana Trandabat, Maria Husarciuc, and Emanuela Boroș. In Proceedings of CLEF 2012 Evaluation Labs and Workshop - Working Notes Papers, 17-20 September, 2012, Rome, Italy
RUN_NAME btbn12011bgbg btbn12021bgbg btbn12031bgbg diue12012enen diue12024enen fdcs12011enen fdcs12021enen fdcs12032enen fdcs12042enen idrq12011arar idrq12021arar jucs12013enen l2fi12011enen l2fi12021enen l2fi12031enen l2fi12041enen loga12011dede loga12023dede mira12011arar mira12021arar onto12011bgbg onto12021enen onto12031itit onto12041robg onto12051roen onto12061roit onto12071bgbg onto12081roro onto12091dede onto12101eses uaic12014roro uaic12024roro uaic12034roro uaic12042roro uaic12052roro uaic12062enen uaic12072enen uaic12082enen uaic12092enen uaic12102enen vulc12014enen vulc12024enen vulc12034enen Average Median
level:
x x x x x analyzes the questions and possible answers r e h t
O weighted lexical matching, paraphrasing, sentential proximity x x x g n i g g a T h c e e p S f O t r a P x g n i k n u h C x n o i t i n g o s c m yeR )ER a r -n ittn (N g
E d e m a N x x s n o i s s e r p x e l a r o p m e T s n o i s s e r p x e l a c i r e m u N s n o i t a m r o f s n a r t e s a r h P x s i s y l a n a y c n e d n e p e D ) c t e . j b o . b u s ( s n o i t c n u F s n o i t a m r o f s n a r t c i t c a t n y S g n i s r a p c i t n a m e S g n i l e b a l e l o r c i t n a m e S x x x x x x x x x x x x x x x x x x
x x x x - - - - - - - - - - - - - - - - - Some basic rules for identificati on of abbreviatio x ns prior to sentence splitting.
Similarity metric: string overlap. x x x x
x x x x x e n o N r e h t
O Lemmatiza
tion Conceptu al Graphs; Similarit y scoring t s i L d r o W s r e e t t e z z a
G x x
B D n o i t a i r a V la i r o g e t a C m y ron ry c a A n -yonm iitcoD n y S x x x x y t i r a l i im ry S a
n ycn ito
c end iD e p e D y t i r a l i m i S y t i m i x o r
P x x - d n u o r g k c a b n
o eh itc t e rom llco f d e r u t p a c x x x x x x n o i t a d i l a v r e w s n a o N x x x x - g n i n r a e L e n i h c a M s e i c n a d n u d e r b e
W B D l a c i x e L s u r u a s e h T a i d e p o l c y c n
E x System analyzes the questions and the answers and it is looking for overlapping n-grams from the analyses of the question and the answers. The answer with greatest overlapping is selected.
The first run is based on surface text analysis, with no grammatical nor semantic processing.
The system uses redundancies in the collection The system "IDRAAQ" is developed for Arabic QA integrates three levels of processing: - Keyword-based level: uses a Arabic WordNet based Query Expansion module - Structure-based level: consists in measuring the Density Distance N-gram Model of candidate It uses Lexical, Syntactic, Semantic level.
The system uses lexical distances such as Word Proximity and similarity measures to select the candidate answer more related with the question. The system also uses Latent Semantic Analysis to extract latent topics from the test documents The system parses questions, answers and documents. Coreference resolution is applied to the document representation. The system constructs a hypothesis from question and answer parse. It then tries to prove the hypothesis from the logical document x x y g o l o t n
O m r e t (ty )gn ir i ila pp im lra ls ve a o c i x e L x x x y t i r a l i m i s c i t c a t n y S x x y t i r a l i m i s c i t a m e
S x x x x - x r a l i m i s r o g n i v o o r p m e r o e h T e l u R e c n R l a c i x e L freee saeB s n rodw iitsoo f p on rop itco ged lle le o w C o n K t n e m l i a t o i t c e l l o
C n o i t p i r c s e D m e t s y S x x onto uaic x x x