In this paper, we present a system for automatically answering opendomain, multiple choice reading comprehension questions about short English narrative texts. The system is based on state-of-the-art text similarity measures, textual entailment metrics and coreference resolution and does not make use of any additional domain specific background knowledge. Each answer option is scored with a combination of all evaluation metrics and ranked according to their overall score in order to determine the most likely correct answer. Our best configuration achieved the second highest score across all competing system in the entrance exam grading challenge with a c@1 score of 0.375.
Question Answering (QA) systems have always been a key focus of Information Retrieval and Natural Language Processing research. Such systems aim to automatically answer questions posed in natural language. The objective of the Entrance Exams task in the CLEF Question Answering Track is the creation of a system for determining the correct answers to a set of multiple choice reading comprehension questions about English narrative texts.
The task demands a deep understanding of a short passage of text and is closely
related to the QA4MRE1 main task [
Based on conclusions drawn from a state-of-the-art analysis, which is discussed in Section 3, our system uses three main components: coreference resolution, text similarity and textual entailment. Text-Hypothesis (T-H) pairs are generated for each entrance exam using combinations of sentences from the text, questions about the text and the corresponding answer options. These T-H pairs form the basic processing units for similarity and entailment analyses, which are employed to determine the correct answer to a question. We describe our approach and the architecture of our system in Section 4.
Finally, we present the results obtained with different system configuration in Section 5 and provide a detailed error analysis in Section 6. We close with a summarization of our findings and a discussion of future research directions in Section 7. 2
The objective of the task tackled in this paper is to identify the correct answer option for a multiple choice question in a reading comprehension test for a given English narrative text. Since the questions are not restricted to a particular topic, they require a wide range of inference. Additionally, no background information is provided and systems are supposed to determine answers based on common sense knowledge that high school students are supposed to possess.
The dataset provided in this task is composed of reading comprehension tests taken
from Japanese university entrance exams and is available in English, Russian, French,
Spanish and Italian. The labeled training data consists of 12 documents with 60
questions and an average of 5 questions for each document. The test data comprises 12
documents with a total of 56 questions. We consider a test item to be a combination of
the narrative text, a single question about this text and all answer options for this
question. An exam is then the set of all test items with the same text. Each exam is evaluated
with a score between 0 and 1 using the c@1 measure [
nc nc + nu n where, nc is the number of correctly answered questions, nu, the number of unanswered questions, and n is the total number of questions. The main idea of this metric is to encourage systems to reduce the number of incorrect answers while maintaining the number of correct ones by leaving some questions unanswered. 3
The question answering task on university entrance exams was first introduced in this form at the CLEF QA4MRE Lab in 2013. In this section, we review the previous work from the pilot task. Related work on enabling technologies such as textual entailment is introduced later in the paper.
The best performing system in the 2013 entrance exam task was presented by
Banerjee et al. [
The runner up system in the performance ranking was presented by Li et al. [
Finally, Arthur et al. [
As discussed by Pen˜as et al. [
We now present the architecture of our question answering system which is based on text similarity measures, textual entailment and coreference resolution. The general idea of the system is to retrieve sentences from the text that are relevant for a given question and to identify the correct answer option among the multiple choices based on textual entailment and similarity between each option and the retrieved sentences.
Our system builds upon the Darmstadt Knowledge Processing Software Repository
(DKPro) [
The system is divided into six main modules: corpus reader, preprocessing, sentence retrieval, answer similarity analysis, entailment analysis, and answer selection. Figure 1 gives a schematic overview of the system architecture. The individual components are described in the remainder of this section. 4.1
The corpus reader parses the provided XML document with the entrance exam data and initializes a separate UIMA Common Analysis Structure (CAS) for each test item. The text of each CAS consists of the narrative text appended with the question and the corresponding answer options. Both the question and the answer options are then marked-up
Corpus Reader
Preprocessing Sentence Retrieval
Answer Similarity
Textual Entailment
Answer Selection with UIMA annotations which identify the exact span of text and hold additional meta information contained in the XML source document. The final CAS is then passed on to the next module in the pipeline for further processing. 4.2
The preprocessing module consists of several NLP components that annotate the text
before the similarity and entailment analysis is carried out. All of these components
are based on the Stanford CoreNLP4 package and employed in DKPro Core wrappers.
We use the Stanford segmenter and lemmatizer for identifying tokens, sentences and
lemmas in the text. The Stanford parser [
Question: What did John want? Answer Option: He wanted a bike.
Resolved Answer: John wanted a bike.
To further improve the sentence retrieval, we also resolve all pronoun entities in the main text. This resolution facilitates the identification of relevant sentences in the text for each question and answer option, since it reduces the lexical variation introduced by referring expressions.
4.3
We found that the downstream textual entailment module is sensitive to irrelevant
information. That is, the longer the text from which a given hypothesis is to be inferred,
the more likely the system is to report a falsely positive inference. It is therefore
necessary to first identify sentences in the text that are most relevant to a particular question
before proceeding with further analyses of the answer options. We use three different
text similarity measures to identify sentences from the document that are most relevant
to a particular question:
1. Lexical similarity: The Word n-gram Jaccard measure from the DKPro Similarity
package implements a generalization of the Ferret measure [
We employ the Jaccard measure based on token unigrams.
2. ESA-based similarity: Text similarity based on explicit semantic analysis [
For all similarity measures, the input texts are stopword-filtered using a subset of the English stopword list from the DKPro WSD5 package. Each similarity measure provides a score between 0 and 1. Using a linear combination of these scores, we select the top k sentences for each question. In our experiments, we empirically found k = 5 to be an optimal value for our setup. We use only these sentences in the following modules to determine the correct answer among all possible options.
As a basic processing unit for the following analyses, we define Text-Hypothesis (T-H) pairs. We use each retrieved sentence from the narrative text as T and all answer options along with all possible combinations of the question with each answer option as H. Both for T and H, we use resolved and unresolved version of the text, depending on the system configuration.
We noticed that, for many questions, the correct answer lies in the sentence before or after the selected sentence. Hence, instead of restricting the set of T entities entirely to the selected sentences, we added an option to include the previous and next sentences as well. These are added as separate T entities to the set. This setting is referred to as S later on (see Table 1).
All downstream analyses are then performed on all possible T-H pairs. Our approach to T-H pair creation is prone to overgeneration, i.e. it will contain irrelevant pairs that can even degrade the system performance. We therefore implemented several selection techniques that only consider particular T-H pairs. An overview of the different configurations will be described in Section 5.
4.4
In order to compute textual entailment on the T-H pairs, we use the EXCITEMENT
Open Platform (EOP) [
We use the MaxEntClassificationEDA configuration provided in EOP, which
uses a maximum entropy model based on the OpenNLP MaxEnt [
While the idea of T-H pairs originally stems from the textual entailment theory, we generalize the idea to similarity analysis as well. As mentioned previously, each sentence (or resolved sentence) is labeled as text (T), while an answer option (or resolved answer option, or a question combined with an answer option) is labeled as the hypothesis. Consequently, similarity measures can be applied to these T-H pairs and corresponding scores can be used to determine the correct answer.
We explained earlier that we do not only include the answer options (and their resolved variants) in the set of hypotheses, but also a combination of the answer option with the question. The reason for this can be seen in the following example: Question: People are normally regarded as old when Answer Option: they are judged to be old by the society Combined: People are normally regarded as old when they are judged to be old by the society.
Relevant Sentence: But in general, people are old when society considers them to be old, that is, when they retire from work at around the age of sixty or sixty-five.
It is easy to see that the combination of question and answer makes the inference of the correct answer easier as compared to using the answer option alone.
We use the same similarity measures that we described earlier in Section 4.3 to measure the similarity between T and H in all T-H pairs. A linear combination of the scores generated by each measure is stored in the markup of each T-H pair and used together with the entailment score for the final answer selection. 4.6
The entailment confidence score and the answer similarity scores obtained for each T-H pair are finally used to select the correct answer option. We calculate a correctness score 1 Original Jaccard – – – S A, reA 2 Original Jaccard, ESA S A, reA Jaccard, ESA S a A, reA, Q+A 3 Original Jaccard, ESA – – – S A, reA, Q+A 4 Original PoS, ESA S A, reA ESA S A, reA, Q+A 5 Original Jaccard S A, reA Jaccard, ESA S A, reA, Q+A 6 Original ESA S A Jaccard, ESA S A, Q+A 7 Resolved Jaccard, ESA reS reA Jaccard, ESA reS reA, Q+A aSentences are selected based on the top 10 sentences according to their similarity to each answer option rather than the sentence selection procedure described in the text. We experimented with different linear combinations of the two scores and found that the best performance is achieved when the entailment score is given a higher weight compared to the similarity score. More specifically, we achieved good results with fx; yg = f2; 1g and fx; yg = f3; 2g. In cases where the entailment confidence exceeded a threshold of .90, we set the weights to fx; yg = f1; 0g thus eliminating the similarity score from the decision process. We finally select the answer option corresponding to the T-H pair with the highest correctness score as the correct answer for a given question. 5
For the Entrance Exams 2014 task, we submitted a total 7 configurations. In this section, we discuss the individual configurations, the scores achieved with each setup and finally provide a detailed error analysis.
Each configuration consists of three main components, the sentence selection strategy, the answer similarity analysis and the entailment analysis. The exact setup for each configuration is provided in Table 1. Overall, configuration 7 achieved the best results with a c@1 score of 0:375. Moreover, configurations 3; 5; and 6 performed above or equal to the random baseline of 0:25, while 1; 2; and 4 performed poorly with c@1 scores below the baseline. The reason configuration 1 failed was mainly due to the fact that answer selection was merely performed on the sentences pertaining to the questions, without considering those surrounding the selected sentences. Hence the idea of using the sentence before and after a selected sentence, was not implemented for this configuration.
Configuration 2, on the other hand, used the previous and next sentences, but failed to clear the baseline. One of the main reasons for this was the fact that sentences chosen as H for the entailment analysis were chosen based on high answer similarity scores rather than the sentence selection procedure based on the question. This resulted in the selection of certain sentences that were related to incorrect answer options, and not directly related to the question. For configurations 4, the linear combination of answer similarity and entailment scores used to determine the final answer was ineffective, and led to incorrect answers. Configurations 3 and 5 did well because the weight assigned to the entailment score was increased, and a lower weight, (0 in case of configuration 3,) was assigned to the answer similarity score. Nonetheless, answer similarity helped determine a few answers where entailment analysis alone had failed, for instance, in configurations 5 and 7.
Additionally, in all but the last run, we used only the original sentences as T. However, for 7, we introduced the idea of using resolved sentences as T. This improved our scores dramatically, giving a c@1 score of 0.375. As for configuration 6, without the use of resolved answers, it failed to perform as well as 3; 5 or 7.
This leads us to conclude that coreference resolution in the sentences and answer options is highly beneficial in determining the correct answer. Additionally, using the previous as well as the next sentence in addition to the selected sentence as T is better than using only the selected sentence. The results confirm that many answers can be identified from a sentence close to the selected one using the question text. Finally, when using a linear combination of the answer similarity and entailment scores, a higher weight should be assigned to the entailment score. 6
In order to identify systematic errors in the decision making process of our system, we conducted an error analysis on the output of each configuration described in the previous section.
The implementation of coreference resolution replaces pronouns and phrases describing a person entity with the proper noun. This generates some issues when the phrase describing a person is vital to answer determination. The coreference resolution in the preprocessing module can further be improved by replacing only certain terms, mainly pronouns, in sentences with the corresponding named entities.
While the sentence extraction works well in most cases, there are two problems that can be encountered. The first problem is that the correct sentence pertaining to a question is not the one with the highest score. Consequently, we select k (=5) sentences rather than just one. Nonetheless, an improvement in identifying the most relevant sentence, and reducing k to 1 or 2 sentences will help to narrow down possible mistakes with answer determination. The second problem is when the question is not sufficient to identify the correct sentence in the text. For example, a question like the following requires sentence selection based on the answer options instead of the question text:
Question: The main point the author wishes to make is that While the answer similarity module was able to assist in determining certain answers, it was generally outperformed by the other measures employed in our system, in particular by the entailment module. It particularly underperformed in case of similarly phrased answer options for a given question, which caused false predictions. Consequently, a lower weight was assigned to the similarity score. With an improved linear combination of the various similarity measures integrated in this module, the performance can further be improved.
While the entailment module based on EOP provides good results, especially in the case of resolved sentences and answer options, it does not perform optimally in certain scenarios. For instance, the module does not perform well when more than one answer options are very similar. This is illustrated by the following example:
Excerpt: “I’m from Georgia,” he said in a loud voice,“and proud of it.” “Sorry I made the mistake,” I told him, though I just couldn’t see what difference it made whether he came from Georgia or Alabama.
Question: What mistake did the writer make about the man with glasses? Incorrect Answer: He thought the man came from Georgia
Correct Sentence: He thought the man’s home state was Alabama Moreover, certain questions require advanced inference based on world knowledge rather than relying on the document surface text alone. In these cases, the correct answers are difficult to determine with the entailment module. For example: Excerpt: He was holding onto the arms of his seat so tightly that the blood had left his fingers. He was what is known as a “white-knuckle flier.” He would not look out the window, and he was sweating so much that the stewardesses had to keep bringing him towels.
Question: What is meant by a “white-knuckle flier?” Answer: It is a person who is extremely nervous on an airplane.
We also encountered errors when a question and answer pair spanned multiple sentences. Since our system considers at most 3 sentences at a time, i.e. the selected, previous and next sentence, it sometimes fails to obtain the desired inference for the correct answer option. With a more generalized retrieval module that supports larger text windows, we could be able to circumvent this problem but have to take into account the added noise that is introduced by larger spans of text.
Thus, to summarize, a refined approach to coreference resolution along with an improved answer similarity module will help to improve scores to a great extent. In addition, the entailment module can be trained with different datasets to improve the classifier, while a tighter sentence selection module will help to narrow down possible sentences relevant to a particular question. 7
Automatically answering multiple choice reading comprehension questions is a challenging task. In this paper, we presented a system designed to solve such questions without any domain-specific background knowledge. Our strategy for determining answers to reading comprehension questions is based on three main components: text similarity, textual entailment and coreference resolution.
As already proposed in related work, textual entailment is the key technology for tackling this task. However, it is sensitive to irrelevant information both in the text and the hypothesis, so a strong filter or retrieval component based on text similarity measures drastically helps to improve the quality of the entailment results. Due to the nature of this task, which focused on reading comprehension questions about narrative texts, coreference resolution furthermore helped to improve the discriminative power of the entailment and similarity scores for the final answer selection.
With our best configuration, we achieved a c@1 score of 0:375, which puts our system on the second place in the overall performance ranking. Moreover, 3 other configurations performed better than or equal to the random baseline of c@1 = 0:25. In our error analysis, we finally identified the most prominent error types our system encountered.
As part of future work, our priorities are to refine the coreference resolution module, and to work on improving the performance of the answer similarity module. In addition, the entailment module can be trained on datasets more suited to the task, in order to improve the classifier, while a tighter sentence selection module will help narrow down possible sentences relevant to a question.
This work has been supported by the Volkswagen Foundation as part of the LichtenbergProfessorship Program under grant No. I/82806.