This paper describes the development of a question answering system for monolingual and cross-lingual tasks for the languages English and German. We developed the question answering system from a document and retrieval focused perspective. The system consists of question and answering taxonomies, named entity recognition, term expansion modules, a multi-lingual search engine based on Lucene and a passage extraction and ranking component. The overall architecture and heuristics applied during development are described. We discuss the results at CLEF 2005 and show potential future work.
The query processing includes the assignment of a question and expected answer type, named entity recognition, translation and stopword removal.
A question taxonomy based on the questions of previous QA tracks
An evaluation based on the CLEF QA topics form the years 2003 and 2004 showed that overall, for 73% of the questions, the answer category was assigned correctly. For further 14%, the categorization was partly correct and for another14% of the questions, a wrong category was assigned. The taxonomy was most reliable for the question types WHEN, WITH WHAT and FOR WHAT. Questions starting with WHAT were categorized worst. An evaluation revealed a recognition rate of 60% for correct recognition and 42% for correct categorization into the following four classes: Person (PER), Organization (ORG), Place (LOC) und Miscellaneous (MISC). Named entity recognition was applied to the queries and to the document corpus. 2.3
The key component for cross-lingual QA is a translation utility. As underlying systems, we used Babelfish, FreeTranslation and Linguatec4. To avoid a large influence of wrongly translated named entities, we replaced all named entities found in the query except for the category MISC with a dummy which was not translated by the translation tools. In addition, the named entities were sent to the translation tool without context subsequently. All translated sentences and terms were collected and only stopwords were removed. 2.4
For retrieving German answers, the translated keywords were expanded using GermaNet5. However, to avoid the addition of too many senses, the expansion was only carried out, when GermaNet included only one meaning of the word under question. For English, the synonym function of WordNet6 was used to expand all translated terms. The effect of term expansion has not been evaluated for our system yet. 3
For stemming, indexing and retrieval we employed Lucene7 as it has been used in
3 http://www.ldc.upenn.edu/
4 http://babelfish.altavista.com/, http://www.freetranslation.com/, http://www.linguatec.net/online/ptwebtext/
5 http://www.sfs.nphil.uni-tuebingen.de/lsd/
6 http://wordnet.princeton.edu/doc
7 http://lucene.apache.org/
These passages were again indexed as documents by Lucene and ranked according to a scoring algorithm which
rewards the frequency of occurrence of keywords in the passage
A few heuristics were implemented to improve performance. We focused on named entities especially. · If named entity is the expected answer type and there are documents in the answer set which contain named entities of the appropriate type, then only these documents are forwarded to the passage extraction. · If named entity is the expected answer type the most frequent named entities of the expected type within all passages are determined and the first passages containing these named entities are returned. · If no answer with named entities is found, then the first 90 characters of the most highly ranked passage are returned. · Trivial answers are not returned. Answers are considered trivial if they contain only one word, if they consist in the name of a known news agency of if the answer string is a subset of the question string. · When the expected answer type is named entity, then all named entities in the first 20 passages are extracted and the most frequent named entity is returned.
The confidence weight returned by the system is the retrieval status value returned by Lucene for the returned passage. NIL is returned when no document is found by Lucene and in this case, a confidence value of 1.0 is assigned. 4
The quality of the results was only satisfying for definition questions. For this first participation and considering the focus on named entities, this seems acceptable. The results are shown in table 1. The weak performance is probably due to several reasons. The time and effort dedicated to evaluation was mainly aimed at system stability and the integration of all tools. Parameter tuning based on previous CLEF experiments were not carried out so far. In addition, this year CLEF required a very short answer. Our system returns passages of at least the length 200 and no further processing is done to extract a short answer. This was probably an advantage for our system for definition questions, where the performance was good. 5
The system for QA can be improved by further integrating the question analysis and the search process. So far, the knowledge gained from the question in not fully exploited. Furthermore, the system needs to be evaluated more thoroughly.
We would like to acknowledge the work of several students from the University of Hildesheim who implemented the components of the QA system as part of their course work8.
We also want to thank Maarten de Rijke for his comments on an earlier version of our QA approach.