=Paper=
{{Paper
|id=Vol-1175/CLEF2009wn-QACLEF-BharadwajEt2009
|storemode=property
|title=A Naïve Approach for Monolingual Question Answering
|pdfUrl=https://ceur-ws.org/Vol-1175/CLEF2009wn-QACLEF-BharadwajEt2009.pdf
|volume=Vol-1175
|dblpUrl=https://dblp.org/rec/conf/clef/BharadwajV09
}}
==A Naïve Approach for Monolingual Question Answering==
https://ceur-ws.org/Vol-1175/CLEF2009wn-QACLEF-BharadwajEt2009.pdf
1
A Naïve Approach for Monolingual Question
Answering
Rohit Bharadwaj, Surya Ganesh, Vasudeva Varma,
{bharadwaj, suryag}@research.iiit.ac.in, vv@iiit.ac.in
LTRC, IIIT-Hyderabad
track with both source and destination languages as English.
We indexed the data using Lucene, an open source search
library. Lucene implements okapi BM25 retrieval model [6].
Abstract— This paper talks about the system which we have Using this search library we have built passage index, that is,
submitted for the ResPubliQA task. We participated in building each passage in a document is considered as a retrieval unit.
the QA system for en-en part. We followed a different method
for each question type. In this paper we outline the methods
which we adapted and the results which we obtained. 3 OUR APPROACH
Our QA system incorporates pipeline architecture as shown in
figure 1. It consists of three core components: 1) Question
1 INTRODUCTION analysis, 2) Passage retrieval and 3) Passage selection. The
implementation details of all the three components are
T HE main focus of QA is to gain the knowledge of the
user’s question and retrieve the sentences that are
described below.
close to the answer. The ResPubliQA task expects the system
to understand the question and retrieve the corresponding Question
passage in the text which contains the answer. The
architecture of our QA system is 1) Question Analysis 2)
Passage retrieval and 3) Passage selection. Question analysis Question Analysis
involves the classification of the question into pre-defined
question types, extraction of query words and determining the
answer type. Passage retrieval searches for passages in the Passage Retrieval
document collection which are likely to contain the answer.
Passage selection ranks the list of candidate answers to
determine the final answer. First, we introduce the task, and
then we describe the pre-processing we carried over the data Passage Selection
in section 2. In section 3 we describe the approach which we
have followed for answering the questions. Section 4
describes the results we obtained while section 5 presents the Answer
analysis and conclusion.
Figure 1: Pipeline Architecture of our QA system
2 PREPROCESSING THE DATA Question Analysis
In question analysis, we classified the given 95 questions into
In ResPubliQA task, we are provided with the data that is
one of the pre-defined classes. The pre-defined question types
delimited into passages and we are expected to return the
or classes are Factoid, Definitive, Reason, Procedure,
passage that contains the answer. We are provided with both
purpose. The classification is semi-automatic. As the question
the question language and the target language in which the
classes are fixed, by observation we identified patterns for
answers are to be present. The task is mainly directed towards
each of the classes. The patterns for Factoid and Definitive
cross language question answering. We participated in the
are inter-related and hence we classified the questions under
these categories into a single class FactDef which was later
2
sub-divided into factoid and definitive classes. The observed an accuracy of 78.2% under fine grained
patterns are shown in Table 1. classification. As the classification accuracy is
higher for coarse grained classification and also
because of the limitations of many NER systems to
Question class Words that Words that Number of recognize fine grained named entity types in
(Answer type) must be should not be questions passages, only coarse answer type is used to identify
present in the present in the passages with answer candidates.
question (case question (case 2. Density: Tellex et al. [5] showed that density based
folded) folded) measures work well for passage retrieval in QA. So,
FactDef what, how, aim, goal, 46 the passages resulting from the above step are then
defin, who, objective, re-ranked based on the density of the question
where, name reason, keywords in them. Density is defined as the average
procedure, distance between the answer and question keywords
purpose in a passage. There are several ways to compute
Reason why, reason - 33 density. We adopt a simple formula as described in
Procedure Procedure - 10 [4] to compute density of query terms in a passage.
Purpose aim, objective, - 6
goal, purpose Finally, among the re-ranked passages, the top ranked
passage is produced as the answer given a question.
Table 1: Generic patterns in questions from different classes
Definitive
Out of the 46 questions in FactDef class, 27 questions are
from factoid class and remaining 19 are from definitive class. We used answer patterns for definitive questions and used
them for passage selection. The question focus or Qword are
The methods followed for each of the question class which extracted by removing the stop words (a pre-compiled list)
includes both passage retrieval and passage selection from the question.
methodologies are described below.
The main answer patterns for definitive questions as given in
Factoid [1] are (where A is the Qword and X is the expected answer)
1)
To answer a factoid question, first, we retrieve a set of
relevant passages. So, a keyword query constructed by 2)
stripping of all the stop words and interrogative words (when,
where, which etc.) in the question. This keyword query is
given to Lucene to retrieve a ranked set of relevant passages.
From this set, one of the passages is given as the answer to a 3)
question. Our approach for selecting an answer containing
4)
passage is a two step process as described below.
< X; [comma]; [also] called; A [comma]>
1. Answer type: Using the answer type of a question,
we identify a set of passages which contain answer
candidates. To obtain the answer type of a question,
we have implemented a question classifier using 5)
support vector machines (SVM) [3]. The classifier
6)
was trained on UIUC [2] dataset which consists of
5,500 questions for training and 500 questions for As our system does not need to extract the answer but to
testing. Every question in the dataset was labeled retrieve the passage, we modified the patterns and extended
into a coarse grained and a fine grained category them by adding few more patterns like “Qword + means/
from a total number of 6 and 50 categories mean/ has/”. So effectively the queries used to search the
respectively. We have used the bag-of-words feature index are the modified queries which are formed by adding
to predict the category, that is, the answer type of a the answer patterns. We also queried the index by adding
question. The classifier showed an accuracy of various versions of the modified query like "Qword means",
86.8%, when tested on 500 questions from the UIUC Qword + means, “Qword, called” etc. This resulted in various
dataset under the coarse grained classification. And, results for each modified query. For identifying the correct
3
answer, we performed various experiments like giving boost Total questions 95
to results of a particular query, giving weight to each query
and calculating the final weight of each result, performing Question answered correctly 54
various intersection and union operations for finding the final Incorrectly answered 37
result on the development dataset. From these experiments, questions
the one that gave most correct results was “prioritizing the
Questions unanswered 4
search patterns and preserving that order while searching”.
The optimal priority order of search patterns is shown below. Table 2: Results for monolingual English ResPubliQA task
1. "Qword means"
2. “Qword mean"
5 CONCLUSION
3. “Qword is/are" We described our participation in ResPubliQA task. We have
developed a monolingual English QA system. Our system
4. “Qword, called” does not rely on any external knowledge resources or any
complex information retrieval, information extraction and
5. “called Qword”
natural language processing techniques. Instead, it uses an
6. “Qword was” effective combination of naive techniques from the above
areas to achieve a decent performance. Our system analyses
7. Qword passages from passage retrieval output to identify the correct
answer in the case of factoid and definition questions,
whereas, the top ranked passage was produced as an answer
So if we achieve result for the first query, then it is given as for the remaining questions in the test set. The analysis
the final answer. If there are no results then we proceed to the method used for selecting the answer differs for factoid and
next query. Finally if we have no results for any of the definition questions.
patterns, then the system doesn't answer the question.
All the remaining questions, that is, questions from Reason, 6 References
Procedure and Purpose types are answered naively by giving a
top ranked passage, with a minimum length of N words, from [1] Soubbotin Soubbotin Insightsoft-M. Patterns of
the passage retrieval component as the answer. In our Potential Answer Expressions as Clues to the Right
experiments on the development data, we have observed Answers. In Proceedings of the Tenth Text REtrieval
better results for N=25. So, we used the same value for Conference (TREC), 2001.
finding the answer to the questions in test data.
[2] Xin Li Dan, Xin Li, and Dan Roth. 2002. Learning
question classifiers. pages 556–562.
4 RESULTS
[3] Zhang, Dell and Lee, Wee Sun. Question classification
using support vector machines. In proceedings of the 26th
The test dataset for ResPubliQA task consists of a subset of annual international ACM SIGIR conference on Research
the JRC-ACQUIS Multilingual Parallel Corpus4, and 500 and development in information retrieval, 2003.
questions distributed over factoid, definitive, procedure,
reason and purpose classes. JRC-ACQUIS is a freely available [4] G. G. Lee, J. Seo, S. Lee, H. Jung, B.-H. Cho, C. Lee, B.-K.
parallel corpus of European Union legal documents. It Kwak, J. Cha, D. Kim, J. An, H. Kim, and K. Kim. SiteQ:
Engineering high performance QA system using lexico-
comprises selected texts written between 1950 and 2006 with
semantic pattern matching and shallow NLP. In
parallel translation in 22 European languages. We used Proceedings of the Tenth Text REtrieval Conference
English documents in the corpus. Out of the 500 questions in (TREC 2001), 2001.
all languages, we have answered 95 questions which are
categorized under monolingual English QA task. The results [5] Stefanie Tellex and Boris Katz and Jimmy Lin and Aaron
of our system as provided by the CLEF are shown in table 2. Fernandes and Gregory Marton. Quantitative Evaluation of
Passage Retrieval Algorithms for Question Answering. In
Proceedings of the 26th Annual International ACM SIGIR
Conference on Research and Development in Information
Retrieval (SIGIR), 2003.
[6] S. E. Robertson, S. Walker, M. Hancock-Beaulieu, M.
Gatford, and A. Payne. Okapi at TREC-4. In Proceedings
of the 4th Text REtrieval Conference (TREC-4), 1995.