This paper summarizes the results of our rst participation at ResPubliQA. Lacking a true question answering system, we relied on our traditional search engine to nd the answers. We submitted two runs: a pure IR baseline and one where de nition question identi cation and corpus-speci c techniques were employed to improve IR performance. Both runs performed well, and our second run obtained better results than the baseline. However, the drawbacks of relying solely on an IR phase are clearly seen in our runs.
The architecture and functions of our system can be seen on Figure 1. The system
is built around the Hungarian Academy of Sciences search engine [
As already mentioned, the system is not a complete question answering
system, as it lacks any form of document analysis, document validation or answer
selection. The answer to a question is simply the rst document returned by the
IR module. The role of the question processing module is to formulate queries
that maximize IR performance. These components are described in more detail
in the next sections.
Our indexer module generates a simple unigram index from the documents in
the collection. We employ the Porter stemmer [
The selection of indexing units needed some e ort. In the JRC-Acquis corpus, a le contains one document, which is divided into numbered paragraphs. In the EuroParl collection, an XML le consists of several chapters, which represent what can be safely considered a "document". These chapters are, in turn, segmented into numbered paragraphs. We considered two options. Document: the whole le (Acquis) or chapter (EuroParl) is indexed, including the title, the contents of all paragraphs and certain metadata, such as the executor id in EuroParl Texts Adopted documents.
Paragraph: Only the paragraph contents are indexed. This is also the obvious choice for the Paragraph Selection task.
We tested both options with the questions in the 2009 training data. Our results are summarized in Table 1.
Even if the numbers are not entirely accurate (the 2009 Gold Standard lists only one valid paragraph for a question, and it disregards duplicates), it can be seen that higher MAP can be achieved for document retrieval. However, due to the lack of an answer selection component, we had to resort to paragraphs as the indexing unit. To compensate for this loss of accuracy, the title of a document or chapter was added to the text of all its paragraphs, albeit with lower weight. 2.2
Instead of generating one query for every question, we decided to create a list of query variations. The rst query formulates the strictest constraints of the documents it accepts, and it is followed by increasingly more permissive variants. The default query consists of all non-stopword terms of the question in an OR relationship. The queries are sent to the IR module in this order. Processing of the list stops if at least one paragraph is found for a particular query.
The idea behind using a list of queries is that we assume that for some question types, the passages we are looking for may exhibit certain peculiarities, which might be exploited with an adequately phrased query. We considered the following features: "Codes": words that contain both letters and digits (legal entries, etc.) We considered this feature important, so such words resulted in two query variants: one, where the word is in an AND relationship with the rest of the words, and the default one.
Quotations: in the JRC-Acquis corpus, quotation marks (") are denoted by the %quot% string. This enabled us to include these in the search with regular IR techniques. Therefore, two queries were created for questions that contained quoted word(s): one with the "AND quot" string appended to the query, and one without. 2.3
Our system does not yet include a full- edged question classi cation module. However, we assumed that in legal texts, such as the JRC-Acquis and the EuroParl corpora, great care is taken to de ne terms and expressions clearly and unambiguously. If this was the case, recognition of de nition questions would bene t the precision of our system substantially. Therefore, an experimental component that identi es de nition questions was implemented. Pattern creation We classi ed the questions in the 2009 training data manually, and collected a set of patterns that occurred in de nition questions. Typical examples include "What is a XY?" and "What is meant by XY?". These informal patterns were then formalized as regular expressions, and matched against the questions in this year's evaluation set. Regular expressions allowed us to decide if the question was a de nition question and retrieve the terms or expressions to be de ned at the same time.
Out of the about 24 de nition questions we found manually (some of these choices may be debatable), our method successfully identi ed 18, or 75%. Exact matching With the expressions to be de ned extracted, we needed a way to use them to enhance IR precision. We use question 66: "What is sports footwear? " as an example to present our method.
Our rst assumption was that the passage where the expression (e.g. "sports footwear ") is de ned may either include it in the same form as in the question, or at least contain all the terms that make up the expression ("sport " and "footwear "). If both of these assumptions are false, we need to fall back to the default OR query. Therefore, three query variations are added to the list for every de nition question: { the whole expression in quotation marks, which denotes exact matching; { the terms connected by an AND relation; { the terms connected by an OR relation (the default).
Based on the formality of the language used, we also assumed that the words "mean", "de nition" and their synonyms may actually occur in the paragraph where the term is de ned. Hence, for every query in the list, we created an expanded variation that included these words in and AND relation with the rest of the query, and added these new, stricter queries to the list as well.
The listing below shows the query list created from our example question. <query id="66"> <subquery index="1">"sports footwear" AND (mean OR define OR ...) </subquery> <subquery index="2">"sports footwear"</subquery> <subquery index="3">sport AND footwear AND (mean OR define OR ...) </subquery> <subquery index="4">sport AND footwear</subquery> <subquery index="5">sport OR footwear</subquery> </query>
In our system, the IR phase is responsible for selecting the paragraph that answers the question. We used our own search engine, which ranked the passages with Okapi-BM25 ranking function. Our implementation takes document structure into account and it is possible to assign di erent weights to the various elds (e.g. title, body, meta-data).
BM25 is a TF-IDF-like ranking, with two additional features. Firstly, it is possible to adjust the e ect of term frequency on the score via the k1 parameter. Secondly, a normalization based on document length is applied to the nal score. The degree of normalization can be adjusted by the b parameter.
The following settings were used for passage retrieval: { The b parameter was set to 0. While document length is usually an important feature in ad-hoc retrieval, our experiences with the 2009 training set indicate no correlation between paragraph length and relevancy. Therefore, we decided against length normalization. { The weight of the title eld is set to 0.5, as it does not belong to the paragraph itself.
As mentioned earlier, the queries in the query list associated with a question are processed until a paragraph is retrieved. If more than one is found, as is usually the case, the one with the highest BM25 score is returned as the answer to the question. Our system did not tag any questions as NOA.
It is worth mentioning that out of the 23 questions that had a query list of at least two elements (in line with section 2.2 and 2.3), there was only one for which the rst query did not return any paragraphs. On the one hand, this means that our assumptions about the corpus were right. On the other, since it would be di cult to add further, meaningful constraints to the queries, it also shows that perhaps not much improvement potential is left in this method. 3
We submitted two runs for the English monolingual task.
Run 1 was our baseline run. Only the paragraph texts were indexed, and default Okapi parameters (k1 = 2, b = 0.75) were used. However, quotation and code detection { and the resulting query variants { were included in this run as well, mostly by mistake.
Run 2 featured all additional techniques described in the earlier chapters, namely: { the paragraphs were augmented with the document titles, { Okapi parameter b was set to 0, { query expansion was performed and exact matches were preferred for de nition questions. 1 We believe that some of the questions were judged incorrectly. Hence, while the table shows the o cial results, the numbers may not be 100% accurate.
The higher rate of correct answers for de nition questions validates our assumptions about the nature of the corpus and proves that exact matching and using meta-information, such as including synonyms of "mean" and "de ne" in the query can indeed increase precision. However, this nave implementation may also mislead the retrieval engine by giving too much importance to terms not actually part of the question, as shown by the following example. This paragraph was returned because the term "port facility" occurred in the document title. Q. What is a port facility? A. 2.4 Terms not otherwise de ned in this part shall have the same meaning as the meaning attributed to them in chapters I and XI-2.
This problem underlines the importance of paragraph validation and analysis, which could have ltered such a passage from the result list. Analysis of the returned paragraphs may also increase precision. Table 3 lists the recall at 1, 5 and 10 passages our system achieved on the 2009 English training data. (Gold standard for the 2010 evaluation data is not available yet.) An answer validator component that examines the top ten candidates may, in the optimal case, improve precision by as much as 77%. In this paper, we have presented our QA system and our results for the English monolingual task. Our system did not use any post-processing of answer candidates and relied completely on the IR module. Our e orts were mainly focused on improving IR performance by tuning retrieval settings, preferring exact matching for de nition questions and including collection-speci c elements in the queries.
The results have proven the validity of our approach. We achieved an almost 50% increase in accuracy for de nition questions, compared to our baseline run. Tuning of retrieval settings, however, has not yielded signi cant improvement.
Our results have also shown the fragility of depending on the IR phase alone for question answering. In the future, we intend to extend our system to a complete QA system by implementing question classi cation for all question types, paragraph validation and answer extraction. Several options to improve the IR phase, such as using thesauri for query expansion and ne tuning retrieval parameters are also yet to be explored.