This paper describes our system submission to the CLEF Question Answering Track 2014 Entrance Exam shared task competition, where the task is to correctly answer multiple choice reading comprehension exercises. Our system is a straightforward adaptation of a model originally designed for scoring short answers given by language learners to reading comprehension questions. Our model implements a two step procedure, where both steps use the same set of metrics for evaluating similarity between pairs of input sentences/questions. In the rst step, we automatically select the sentence of the reading text that best matches the question. In the second step, the selected sentence is compared to each of the four answers, and the answer with the highest similarity score is chosen as the correct answer. Although the model has not been tuned to this speci c task, we obtain scores that are competitive with other top-performing systems in the challenge. Additionally, we make no use of the training material but rather treat the task as one of general determination of semantic similarity between text sentences and provided answers.
Reading comprehension exercises are a widely-used and important means of assessing students' ability to understand the material they read. Questions in reading comprehension exercises generally span a range of di culty levels, from simple extraction of facts contained in reading texts to more sophisticated inference, requiring both information in the text and general (or subject-speci c) background knowledge. As such, they provide a challenging context for automated analysis.
The CLEF Question Answering Track 2014 Entrance Exam shared task asks systems to read a given document and answer a set of multiple-choice questions based on the reading text. We approach this task by adopting a model originally designed for a di erent reading comprehension context: scoring short answers to reading comprehension questions given by language learners. The tasks have in common that they require assessing the suitability of answers to questions based on reading texts, but they di er in their inputs and expected outputs.
For short answer scoring, the system is provided with a reading text, a set of questions, a target answer for each question, and a set of learner answers to be scored as correct or incorrect. Most short answer scoring systems work by comparing learner answers to a sample solution (aka target answer), but language learners tend to replicate chunks of the reading text in their answers. Thus, it is often straightforward to match a sentence of the text to an answer written by a learner. In previous work we used this tendency to develop a scoring model that incorporated features based on the relationship between reading text sentences and learner answers [3].
In the Entrance Exam challenge, the system is again provided with a reading text and a set of questions based on the text. Instead of a target answer, though, there is a set of four answers: one best answer and three distractor answers. Often there is high similarity between the best answer and one or more other answers. To accomplish this task, we again use a model that evaluates similarity between text sentences and both the question and the set of answers. The basic idea of the model (which is described in more detail in Section 3) is to use a common set of similarity metrics (following [4]) in a two-step procedure. First, we automatically identify the sentence of the reading text that best matches the question, on the assumption that this sentence has a reasonable likelihood of containing the question's answer. We will see (in Section 5) that this assumption does not always hold. Second, we choose as the best answer the one our system evaluates as most similar to the selected sentence from the text.
It should be noted that this approach requires no training material, as it simply relies on evaluating similarity between either the question and a sentence from the reading text or a reading text sentence and each answer from the set of four in the multiple-choice setting. Although our model has not been tuned to the speci c task, it still performs at a level that is comparable to other topperforming systems in the challenge. This suggests that the general approach captures key aspects of semantic similarity. 2
The aim of the given task is to provide a computational solution towards automatic question answering. The data consist of reading comprehension exercises which were part of Japanese university entrance exams. Those exams are meant to be used for checking new student's capabilities of various skills by testing them with the help of reading exercises and are collected from the Japanese Center Tests of 2013 and 2014.1
The task now is to detect the right answer for a given question and reading text. Contrary to earlier shared tasks from this scenario, questions are rather unconstrained here and range from \simple" comprehension questions, that require the student or computer just to nd a paraphrase, over sentence completion 1 http://nlp.uned.es/entrance-exams/ questions up to complex questions that demand a deeper knowledge of text coherence.
The data consists of 12 test documents with 56 questions and 4 answers for each question. The texts vary in terms of length, complexity and content. Task organizers provided two data sets: one for training and development, with the correct answer for every question indicated, and a second for testing and ranking systems participating in the challenge. Though we used training material to test the general feasibility of using our pre-existing model for this new task, we do not use correct-answer annotations for any actual model training or even parameter setting. 3
As described above, our system was originally developed for the related task of short answer scoring. In short answer scoring, a number of di erent criteria, ranging from token overlap to various syntactic and semantic features, are used to determine whether a given student answer is correct or incorrect. A crucial di erence between short answer scoring and answering multiple-choice questions is the absence in the latter of an \ideal" target answer. When we use this model for short answer scoring, we compare each student answer to the target answer, and then do supervised classi cation to learn how this comparison looks for correct vs. incorrect answers. In this section we describe how we adapt the model for scoring answers when we don't have a target answer to compare to.
Figure 1 schematically shows the work ow of the system. In general the system consists of two components. In the rst step, answers, texts and questions are preprocessed and annotated with linguistic information. The output of this preprocessing afterwards serves as the input for the alignment module. Our adapted alignment model for evaluating answers as such consists of two sub-modules again, the sentence selection module and the answer selection module. Both modules rely on alignment between sentences. For sentence selection, we nd the best reading text sentence for a given question via alignment ; for answer selection we align each answer with this best sentence. We rst describe the alignment model and then the general work ow of our two-step answer evaluation model. 3.1
In our alignment model we follow the methodology that has been proposed by [4] for grading short answer questions. In such a task, the content of a learner answer is aligned to that of a target answer, and features measuring the overlap between target and learner answer are extracted in order to approximate the determination of semantic equivalence between target answer and learner answer. During alignment, the model identi es similarities between a learner answer and its corresponding target answer on a number of pairs on a number of linguistic levels: tokens, chunks, and dependency triples. For the current task, aligning answers and questions to text sentence, we mainly consider alignments between tokens.
We preprocess all material (texts, questions and answers) using standard NLP tools: for sentence splitting (OpenNLP) and tokenization (Stanford CoreNLP),23 POS tagging and stemming (both Treetagger [6]),4 NP chunking (Treetagger) and synonym extraction (WordNet [1]).5 For synonyms we use not only words that occur in the same synset but also words that are in a hypernym relation and have maximally one node in between them.
On the token level, we use several di erent metrics for identity between tokens, with each metric associated with a certain alignment weight. We use the following types of identity (id), weighted in descending order: token id > lemma id > synonym id. After weights have been determined for all possible token pairs, 2 http://opennlp.apache.org/index.html 3 http://nlp.stanford.edu/software/corenlp.shtml 4 http://www.cis.uni-muenchen.de/ schmid/tools/TreeTagger/ 5 http://wordnet.princeton.edu/ the best applicable weight is used as input for a traditional marriage alignment algorithm [2].
The token alignment is afterwards re ned by chunk-alignment: For chunk alignment, we use the percentage of aligned tokens between chunks for two sentences as input for the alignment process. Two chunks can only be aligned if at least one of the tokens has been aligned. If an aligned token pair from the previous token alignment step ends up in two not-aligned chunks, the token alignment is split up again.
In short-answer scoring, the resulting alignment is used to extract a variety of overlap features, like e.g. how many tokens of the learner answer can be found in the target answer. In the task at hand, where we have to identify the one sentence out of a set of sentences that ts some other sentence best, we instead use the alignment directly to compute one overall alignment weight by summing up the weights of all token-level links of the nal alignment between two sentences. 3.2
Sentence Selection. In this rst step of our evaluation, we aim at nding the text sentence which best matches the lexical material in the question. To do so, we align each sentence in the reading text with the question and compute the overall alignment weight for each pairing. The text sentence with the highest weight is then assumed to be the sentence that carries the most crucial information for answering the question.
Answer Selection. After the best-matching sentence has been identi ed, we align this sentence with each of the four potential answers. Again the answer with the highest alignment weight is proposed as the correct answer.
Figure 2 schematically illustrates the process of selecting the correct answer. The black arrow indicates the selection of the best sentence by aligning it with the question. The answer that aligns best with this sentence is taken to be correct. Other answers that were not selected might potentially link to other regions of the text (as indicated with red arrows).
One technical problem that may occur is that two answers to the same question can end up with the same alignment weight. In the test data, this happens for 11 out of 56 questions. We investigate two di erent ways of handling this outcome, submitting them as two runs of our model. In the rst run (csgs-1 in Table 1), we simply choose the rst (in linear sequence) of the equally-weighted answers. In the second run (csgs-2), we mark such questions as unanswered.
Example (
Question: At the beginning of the story, what did the boy think George did for a living? Text sentence: \Are you a carpenter, sir" the boy asked, looking up at the old man's face.
{He thought he made and repaired wooden objects. {He thought he painted pictures or houses. {He thought he xed musical instruments. {He thought he grew plants and owers. 4
In this section we provide error analysis to show and discuss the strengths and weaknesses of our model in the context of this task. In order to analyze the
System Run 0.446 EE2014{SynapseDeveloppement{1{Output english.xml 0.375 EE2014{DIPF{7{dipf1407enen.xml 0.375 EE2014{cicnlp{8{cicnlp-8.xml 0.362 EE2014{csgs{2{02 1.xml 0.357 EE2014{csgs{1{01 1.xml 0.357 EE2014{cicnlp{7{cicnlp-7.xml 0.339 EE2014{cicnlp{2{cicnlp-2.xml 0.304 EE2014{cicnlp{4{cicnlp-4.xml 0.304 EE2014{cicnlp{3{cicnlp-3.xml 0.286 EE2014{DIPF{5{dipf1405enen.xml 0.286 EE2014{DIPF{3{dipf1403enen.xml 0.286 EE2014{cicnlp{6{cicnlp-6.xml 0.286 EE2014{cicnlp{1{cicnlp-1.xml 0.25 EE2014{LIMSI-CNRS{4{dude4.xml 0.25 EE2014{DIPF{6{dipf1406enen.xml source of the errors, we annotate the English data set with the best sentence that matches the meaning of the correct answer for each question. In this manner, we establish a gold standard (GS) for the rst step of our model, which is the sentence selection described in Section 3.2. We also discuss the performance of the two modules of the system for more detailed evaluation. 5.1
Our answer selection mechanism comprises two steps: the selection of a relevant text sentence that potentially contains the correct answer and the alignment of the proposed answer alternatives against this passage.
For a better understanding of these two components, we assess their contributions separately: We rst assess how often our sentence selection module nds the correct sentence in the text. In a second step we examined how good the alignment of answers to the text would be given that we had oracle information about this best sentence. In order to be able to conduct these experiments we need some additional gold standard information we marked the text sentence that we thought contained the relevant material necessary to answer the question, i.e. the gold standard for the best text sentence.
These annotations have been done by two annotators each, in case of a disagreement they have been adjudicated by a third annotator. The two annotators agreed in 61% of the cases. This shows that the task for selecting the best sentence is not trivial even for humans. As a matter of fact usually one needs more than one sentence to answer a question. Especially for a general question like \What is this story about?" it is necessary to infer the main point of the story, which would require a deeper semantic analysis that would need to include processing on the discourse level.
Having the GS for the best matching sentence we re-ran the classi cation of the answers and obtained an accuracy of 50%, which is an improvement of 16% compared to the performance with the automatically selected sentences. Even using GS sentences from the text we are able to detect the correct answer for only half of the questions. 5.2
In this section we show examples for which our model worked and also didn't work and discuss the reasons for that.
Example (
What was the problem the author had with his house?
My son and I were trying to sell the house we had restored but in the barn attached to it there were bats and they wouldn't leave. Answers ordered by alignment weight, best tting rst: {Bats were living in the barn and wouldn't go away. {The author and his son might not be able to stay for the season. {The author and his son couldn't sleep well because of the muttering sounds.
{The house was still badly in need of repair.
Another case that is well handled by our model is not a direct question but
rather a completion of a sentence like in example (
Rats that live with their brothers and sisters during their early days
It has been found that while baby rats kept with their brothers and sisters engage in a lot of rough play, those raised alone with their mothers play just a little.
{spend a lot of the time playing roughly with them. {hurt each other a lot through their rough play. {quickly learn to be independent of their mothers.
{still want to play with their mothers.
In case our system fails to detect the best sentence, the consequence is that
the selection of the correct answer also fails. The reason for that is because we
compare the answers to a di erent piece of the given text. The incorrect answers
in the multiple choice are also related to the text but not to the relevant part for
the question under consideration. This idea is illustrated with example (
How did the author obtain Margaret's address? b. System Best Sentence: \I don't know how my address got into a magazine in Japan, because I have never asked for a pen pal, but it's so nice hearing from someone in such a fascinating country, and I look forward to corresponding with you."
I was reading a popular youth magazine when I noticed a list of addresses of young people from all over the world who were seeking pen pals in Japan.
Answers: {He wrote to a popular magazine for her address.
(best for system sentence) {He found it in a popular magazine.
(best for gold standard sentence) {He received it from one of his classmates.
{He selected it from a list given by his teacher. 5.3
We compared how many of the sentences in the GS are also selected by the
sentence detection component of our system. It turned out that only 11 of 56
sentences matched the GS sentences. In other words the accuracy on the sentence
selection task is 24%. It is particularly challenging for an automatic method
to distinguish in case there is a high overlap with the lexical material in the
question, whether the sentence contains the information relevant for answering
the question or not. In example (
Why was the writer in Arizona during World War II?
I'd been sent to a special camp in Arizona for Japanese-Americans during World War II, before I joined the army.
I'd been sent to a special camp in Arizona for Japanese-Americans during World War II, before I joined the army.
In contrast, in example (
Why did the author ask Margaret for her picture?
Margaret had asked her friend to send it only in the case of her death.
I knew it would be impolite to ask a girl her age, but thought it would be all right to ask her to send a picture.
In this paper we have described the adaption of a model originally developed for short answer scoring to the task of answering multiple choice questions in an entrance exam scenario. The model has been shown to be suitable for both tasks, since both require evaluating a set of responses according to how well they answer reading comprehension questions. With no task-speci c tuning, and without using the training material provided, the system achieves performance comparable to the best-performing runs submitted to the CLEF Question Answering Track 2014 Entrance Exam shared task.
That said, there is ample room for improving the system in order to better handle the task at hand. In our two-step approach, aspects of both modules could be improved. Error analysis shows that the rst step of the procedure, sentence selection, performs quite poorly compared to gold standard annotations. High overlap with the question material is simply not enough to choose the sentence from the text that contains the highest proportion of the answer material. For this task we need deeper semantic analysis. In particular, a rst step would be to incorporate a co-reference system; this would be bene cial for the many sentences in which pronouns occur instead of full noun phrases. We could further improve this module by taking into account the type of the question and the corresponding expected answer type.
One particular weakness of our approach is that the system, by maximizing percentage overlap, tends to prefer shorter sentences to longer ones. One potential way to address this problem would be to use some metric for identifying the most important words in the reading text and then give these terms more weight when determining the overall alignment score. We would also like to expand our approach to lexical similarity to better identify words taht are not covered by Wordnet synset relations.