1 A Naïve Approach for Monolingual Question Answering Rohit Bharadwaj, Surya Ganesh, Vasudeva Varma, {bharadwaj, suryag}@research.iiit.ac.in, vv@iiit.ac.in LTRC, IIIT-Hyderabad  track with both source and destination languages as English. We indexed the data using Lucene, an open source search library. Lucene implements okapi BM25 retrieval model [6]. Abstract— This paper talks about the system which we have Using this search library we have built passage index, that is, submitted for the ResPubliQA task. We participated in building each passage in a document is considered as a retrieval unit. the QA system for en-en part. We followed a different method for each question type. In this paper we outline the methods which we adapted and the results which we obtained. 3 OUR APPROACH Our QA system incorporates pipeline architecture as shown in figure 1. It consists of three core components: 1) Question 1 INTRODUCTION analysis, 2) Passage retrieval and 3) Passage selection. The implementation details of all the three components are T HE main focus of QA is to gain the knowledge of the user’s question and retrieve the sentences that are described below. close to the answer. The ResPubliQA task expects the system to understand the question and retrieve the corresponding Question passage in the text which contains the answer. The architecture of our QA system is 1) Question Analysis 2) Passage retrieval and 3) Passage selection. Question analysis Question Analysis involves the classification of the question into pre-defined question types, extraction of query words and determining the answer type. Passage retrieval searches for passages in the Passage Retrieval document collection which are likely to contain the answer. Passage selection ranks the list of candidate answers to determine the final answer. First, we introduce the task, and then we describe the pre-processing we carried over the data Passage Selection in section 2. In section 3 we describe the approach which we have followed for answering the questions. Section 4 describes the results we obtained while section 5 presents the Answer analysis and conclusion. Figure 1: Pipeline Architecture of our QA system 2 PREPROCESSING THE DATA Question Analysis In question analysis, we classified the given 95 questions into In ResPubliQA task, we are provided with the data that is one of the pre-defined classes. The pre-defined question types delimited into passages and we are expected to return the or classes are Factoid, Definitive, Reason, Procedure, passage that contains the answer. We are provided with both purpose. The classification is semi-automatic. As the question the question language and the target language in which the classes are fixed, by observation we identified patterns for answers are to be present. The task is mainly directed towards each of the classes. The patterns for Factoid and Definitive cross language question answering. We participated in the are inter-related and hence we classified the questions under  these categories into a single class FactDef which was later 2 sub-divided into factoid and definitive classes. The observed an accuracy of 78.2% under fine grained patterns are shown in Table 1. classification. As the classification accuracy is higher for coarse grained classification and also because of the limitations of many NER systems to Question class Words that Words that Number of recognize fine grained named entity types in (Answer type) must be should not be questions passages, only coarse answer type is used to identify present in the present in the passages with answer candidates. question (case question (case 2. Density: Tellex et al. [5] showed that density based folded) folded) measures work well for passage retrieval in QA. So, FactDef what, how, aim, goal, 46 the passages resulting from the above step are then defin, who, objective, re-ranked based on the density of the question where, name reason, keywords in them. Density is defined as the average procedure, distance between the answer and question keywords purpose in a passage. There are several ways to compute Reason why, reason - 33 density. We adopt a simple formula as described in Procedure Procedure - 10 [4] to compute density of query terms in a passage. Purpose aim, objective, - 6 goal, purpose Finally, among the re-ranked passages, the top ranked passage is produced as the answer given a question. Table 1: Generic patterns in questions from different classes Definitive Out of the 46 questions in FactDef class, 27 questions are from factoid class and remaining 19 are from definitive class. We used answer patterns for definitive questions and used them for passage selection. The question focus or Qword are The methods followed for each of the question class which extracted by removing the stop words (a pre-compiled list) includes both passage retrieval and passage selection from the question. methodologies are described below. The main answer patterns for definitive questions as given in Factoid [1] are (where A is the Qword and X is the expected answer) 1) To answer a factoid question, first, we retrieve a set of relevant passages. So, a keyword query constructed by 2) stripping of all the stop words and interrogative words (when, where, which etc.) in the question. This keyword query is given to Lucene to retrieve a ranked set of relevant passages. From this set, one of the passages is given as the answer to a 3) question. Our approach for selecting an answer containing 4) passage is a two step process as described below. < X; [comma]; [also] called; A [comma]> 1. Answer type: Using the answer type of a question, we identify a set of passages which contain answer candidates. To obtain the answer type of a question, we have implemented a question classifier using 5) support vector machines (SVM) [3]. The classifier 6) was trained on UIUC [2] dataset which consists of 5,500 questions for training and 500 questions for As our system does not need to extract the answer but to testing. Every question in the dataset was labeled retrieve the passage, we modified the patterns and extended into a coarse grained and a fine grained category them by adding few more patterns like “Qword + means/ from a total number of 6 and 50 categories mean/ has/”. So effectively the queries used to search the respectively. We have used the bag-of-words feature index are the modified queries which are formed by adding to predict the category, that is, the answer type of a the answer patterns. We also queried the index by adding question. The classifier showed an accuracy of various versions of the modified query like "Qword means", 86.8%, when tested on 500 questions from the UIUC Qword + means, “Qword, called” etc. This resulted in various dataset under the coarse grained classification. And, results for each modified query. For identifying the correct 3 answer, we performed various experiments like giving boost Total questions 95 to results of a particular query, giving weight to each query and calculating the final weight of each result, performing Question answered correctly 54 various intersection and union operations for finding the final Incorrectly answered 37 result on the development dataset. From these experiments, questions the one that gave most correct results was “prioritizing the Questions unanswered 4 search patterns and preserving that order while searching”. The optimal priority order of search patterns is shown below. Table 2: Results for monolingual English ResPubliQA task 1. "Qword means" 2. “Qword mean" 5 CONCLUSION 3. “Qword is/are" We described our participation in ResPubliQA task. We have developed a monolingual English QA system. Our system 4. “Qword, called” does not rely on any external knowledge resources or any complex information retrieval, information extraction and 5. “called Qword” natural language processing techniques. Instead, it uses an 6. “Qword was” effective combination of naive techniques from the above areas to achieve a decent performance. Our system analyses 7. Qword passages from passage retrieval output to identify the correct answer in the case of factoid and definition questions, whereas, the top ranked passage was produced as an answer So if we achieve result for the first query, then it is given as for the remaining questions in the test set. The analysis the final answer. If there are no results then we proceed to the method used for selecting the answer differs for factoid and next query. Finally if we have no results for any of the definition questions. patterns, then the system doesn't answer the question. All the remaining questions, that is, questions from Reason, 6 References Procedure and Purpose types are answered naively by giving a top ranked passage, with a minimum length of N words, from [1] Soubbotin Soubbotin Insightsoft-M. Patterns of the passage retrieval component as the answer. In our Potential Answer Expressions as Clues to the Right experiments on the development data, we have observed Answers. In Proceedings of the Tenth Text REtrieval better results for N=25. So, we used the same value for Conference (TREC), 2001. finding the answer to the questions in test data. [2] Xin Li Dan, Xin Li, and Dan Roth. 2002. Learning question classifiers. pages 556–562. 4 RESULTS [3] Zhang, Dell and Lee, Wee Sun. Question classification using support vector machines. In proceedings of the 26th The test dataset for ResPubliQA task consists of a subset of annual international ACM SIGIR conference on Research the JRC-ACQUIS Multilingual Parallel Corpus4, and 500 and development in information retrieval, 2003. questions distributed over factoid, definitive, procedure, reason and purpose classes. JRC-ACQUIS is a freely available [4] G. G. Lee, J. Seo, S. Lee, H. Jung, B.-H. Cho, C. Lee, B.-K. parallel corpus of European Union legal documents. It Kwak, J. Cha, D. Kim, J. An, H. Kim, and K. Kim. SiteQ: Engineering high performance QA system using lexico- comprises selected texts written between 1950 and 2006 with semantic pattern matching and shallow NLP. In parallel translation in 22 European languages. We used Proceedings of the Tenth Text REtrieval Conference English documents in the corpus. Out of the 500 questions in (TREC 2001), 2001. all languages, we have answered 95 questions which are categorized under monolingual English QA task. The results [5] Stefanie Tellex and Boris Katz and Jimmy Lin and Aaron of our system as provided by the CLEF are shown in table 2. Fernandes and Gregory Marton. Quantitative Evaluation of Passage Retrieval Algorithms for Question Answering. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2003. [6] S. E. Robertson, S. Walker, M. Hancock-Beaulieu, M. Gatford, and A. Payne. Okapi at TREC-4. In Proceedings of the 4th Text REtrieval Conference (TREC-4), 1995.