This paper describes a lexical system developed for the main task of Question Answering for Machine Reading Evaluation 2013 (QA4MRE). The presented system executes the preprocessing of test documents, and generates hypotheses consisting of the question text combined with text from possible answers for the question. The hypotheses are compared to sentences from the text by the means of a set similarity measure. The k best similarity scores obtained by each hypothesis are averaged as ranking score for the hypothesis. Two variations of the developed system were utilized, one of them employing coreference detection and resolution techniques in order to take advantage of the discourse structure on the question answering process. The results generated by the systems in QA4MRE 2013 edition are presented and analyzed. The presented system should serve as a solid base for the development of a semantic approach on the task.
The QA4MRE competition [
The main task of the competition is currently composed by four topics: \AIDS", \Climate Change", \Music and Society" and \Alzheimer's". A background collection of texts is provided for all topics. This collection attempts to encompass all speci c domain knowledge of the topic.
The 2013 edition consists of 16 reading tests, 4 on each topic. Each reading test presents a text document followed by 15 to 20 questions about it. Those are multiple choice questions with 5 alternatives, the last one being \None of the above".
Questions are distributed over di erent degrees of complexity as to the knowledge and inference required to devise the correct answer. The simplest ones have both the question fact and the answer appearing directly in the same sentence of the text. Others have the question fact and the answer appearing in distinct sentences. Some questions require background knowledge or inference, and some may require the use of both.
Question: What caused an improvement in sound quality in 1950? Alternatives: 1. the introduction of soundtrack recording on 35 mm magnetic tape 2. the use of an optical soundtrack 3. the adoption of a quadraphonic sountrack 4. the specialisation in silent lms 5. none of the above
An example question is presented in table 1. For this question, alternative 1 should be identi ed as correct.
The system described in this work is based on a text mining baseline
system [
Two di erent systems were employed for the competition: a main system was elaborated, and employed both as standalone and as a base for a variation including coreference resolution. The structure of both systems is presented in gure 1.
Section 2.1 presents the text preprocessing employed on the test documents. Section 2.2 explains the procedure responsible for ranking the alternatives, while section 2.3 discusses the di erent techniques employed in cases where a tie occurs in the ranking. While all the previously mentioned characteristics are common to the two systems, section 2.4 explains the additions only present in the coreference variation. 2.1
Preprocessing of the test documents proceeds according to the following sequence: 1. Unicode decoding: treats special unicode characters, generating ASCIIsafe strings. 2. Text xing: corrects some of the formatting issues present in the 2012 test set, through the use of regular expressions. Although similar issues were not detected with 2013's test set, the procedure was maintained for safety and compatibility reasons. 3. Sentence tokenization: splits sentences from text into separate strings. 4. Word tokenzation: segments individual words and punctuation signs from text strings.
(a) Base System
5. Bag of words model: sentences are represented as sets of word strings.
6. Stopword and punctuation removal: punctuation signs and words
classi ed as stopwords according to the English stopword corpus from NLTK [
primitive form of discourse analysis, a procedure where the sets of words from adjacent sentences are joined prior to ranking was tested. Results generated by this strategy on QA4MRE'12 test set presented an overall improvement in accuracy, so this procedure was maintained for the base system.
With respect to item 7, the choice between stemming and lemmatization took into consideration the accuracy observed on 2012 test set. For the base system, lemmatization yielded marginally better results, while the coreference system presented improved results when stemming was employed. 2.2
Ranking is performed by computing the similarity between sentences and hypotheses, as presented in Algorithm 1. The hypotheses are generated by joining question text and the text from alternatives:
Hi = Q [ Ai Algorithm 1 Ranking Procedure ranks list() for all hypothesis hyp in hypotheses list do similarities list() for all sentence sent in document do sim similarity(sent; hyp) similarities:append(sim) end for average 0 for all top k values sim in similarities do
average average + sim end for average average=k ranks:append(average) end for selected indexOf (maximum(ranks))
The similarity metric employed during the ranking procedure is the MASI
similarity [
jset1 \ set2j max(jset1j; jset2j)
According to the similarity scores calculated between hypotheses and text sentences, each hypothesis receives an overall ranking score computed as the average of the k best sentence similarities. The candidate answer chosen by the system corresponds to the hypothesis which presents the highest ranking score. The value of k is a parameter provided to the system, and was set to k = 2, considering the test results obtained with the QA4MRE'12 test set. 2.3
In some cases, the ranking procedure results in a tie between two or more hypotheses. To handle those cases, four di erent strategies were envised: 1. All questions where no candidate answer was found (there was a ranking tie) were answered as \None of the above". 2. All questions where no candidate answer was found were left unanswered. 3. If the maximum ranking score between the alternatives is inferior to a certain threshold, the question is answered as \None of the above". Otherwise, it is left unanswered. 4. If the maximum ranking score between the alternatives is inferior to a certain threshold, the question is answered as \None of the above". Otherwise, one of the tied alternatives is selected at random.
The reasoning behind the threshold value utilized in 3 and 4 is that in questions where the ranking values were lower, there would be a higher chance that none of the alternatives was correct and the question had no answer. In contrast, in questions where the ranking values were higher, it would be more likely that there was a correct answer, but the system was unable to nd it. This threshold was empirically set to 0.1. 2.4
The system illustrated in Figure 1b includes a coreference detection phase, as well as a coreference resolver. This addition intends to take advantage of discourse analysis, allowing for the resolution af anaphoric pronouns, as well as other types of coreferences.
Coreference detection is performed by Stanford's NLP suite [
This simple strategy presents good results with regards to the resolution of referential pronouns, in the context of a hypothesis ranking computed through set similarity: since the words from the representative reference are included in the word set of referencing sentences, this has a positive impact on the similarity between those sentences and hypotheses that mention the same entity. 3
Eight distinct runs were submitted to the competition, where the two presented
systems were paired with each one of the four tie strategies described in section
2.3. General results from each run are presented in table 2. In this table,
performance is measured according to two metrics: accuracy and the c@1 measure
[
nnR ) n where nR = number of correctly answered questions nU = number of unanswered questions n = total number of questions
The competition consisted of a total of 240 main questions, of which 44 required inference in the answering process. Those inference-demanding questions had simpler duplicates where the question was phrased in a way the inference was no longer required. The \c@1 main" accuracy only takes the 240 main questions into consideration, and \c@1 all " is calculated over all 284 questions.
Run number 6 presented the best general results, with a c@1 measure of 0.30 on the main questions. The considerable increase in the c@1 metric between the main set of questions and the complete set reinforces the weakness of the employed systems with inference demanding questions. The presented methodology is entirely based on lexical similarity. Possible directions for improvement are the inclusion of techniques for proper handling of questions involving negation (added to 2013 main task, but not present in 2012). The system could also bene t from a weighted similarity measure that would prioritize words according to importance.
Although there is still room for improvement while maintaining the lexical character of the system, we believe that the ideal focus of future work would be on establishing a system able to deal with semantic relations through the development of strategies aiming at textual and logic inference. We also consider of crucial importance the development of techniques for knowledge base construction, which can perform the extraction of domain-speci c knowledge from the background collection. 5
Guilherme de Oliveira da Costa Marques is funded by Brazil's National Counsel of Technological and Scienti c Development (Conselho Nacional de Desenvolvimento Cient co e Tecnologico - CNPq). Mathias Verbeke is funded by the Research Foundation Flanders (FWO-project G.0478.10 - Statistical Relational Learning of Natural Language).