This article presents the participation of University of Wolverhampton in the Romanian to English Question Answering task at CLEF-2008. This year we employed a modular framework which allows di®erent modules to be easily plugged in and customised. The main components of our system deal with the three standard stages used in question answering: question processing, paragraph retrieval and answer extraction, and the system's cross-linguality is ensured by a term translator. The question processor analyses Romanian questions and produces a detailed representation of each question including the terms it contains. English translations are then generated for all question terms by exploiting information included in the Romanian and English WordNets, as well as aligned Wikipedia pages. They form the query that Lucene uses to extract English paragraphs which constitute the input for an answer extractor largely based on the one distributed with the OpenEphyra framework. The results indicate a small improvement in comparison with last year's performance.
Our system adheres to the established pipeline architecture for QA that consists of a question
processor, a passage retrieval module and an answer extractor [
At this ¯rst stage of the system, each Romanian question is analysed with the aim of identifying relevant information necessary in the subsequent stages of the system's answering process. The question processing module produces a custom representation of the question that includes: the question focus, the expected answer type (EAT), the question type, the relevant terms identi¯ed in the question and the question topic. The way each of these is determined is explained below.
The question processor employed this year is an improved version of the one embedded in last year's system. As before, this module involves several preprocessing stages such as part-of-speech, noun and verb phrase identi¯cation, numerical and temporal expression annotation, all improved to address problems identi¯ed during last year's participation in QA@CLEF. This year's system uses a revamped phrase and named entity recogniser based on dictionaries and entity mappings extracted from the aligned Romanian and English Wikipedias as described in Section 3.
The linguistic information obtained during preprocessing is used to identify the question focus, which is important in the search for an answer, as it normally reveals what the question is asking for or what the question is about.
The semantic type of the expected answer is then determined by matching question focus and the ¯rst question verb phrase against syntactic constraints and semantic hierarchies extracted from WordNet (e.g. the noun arhitectro/architecten, as hyponym of the synset person, individual, someone, somebody, mortal, soul in WordNet, is automatically mapped to the EAT category person). As we did last year, prede¯ned mappings between categories of expected answers and WordNet synset hierarchies are employed. A more detailed EAT categorisation is covered by this year's question processing module and includes the following classes: person, location, organization, name, type, nationality, language, occupation, definition, numeric (with the sub-classes quantity, measure, economic, percentage) and temporal (with the sub-classes century, year, month, week, date, time, duration). The number of categories employed by our system was increased after noticing that a number of questions from last year's test set could not be answered due to the fact that we had too general categories.
As in previous editions, this year's questions are concerned with facts or events (factoid
questions), de¯nitions of people, organisations or things (definition questions), or lists of people,
objects or dates (list questions). These question types are recognised using the same approach
as last year [
The question processor also produces a list of keywords in decreasing order of their relevance. The list contains noun and verb phrases, named entities, temporal and numeric expressions that appear in the question to be answered, and is used as the input of the term translation module. The topic corresponding to a cluster of questions, whenever it can be identi¯ed following the procedure described below, is also added to the keyword list of each question in the cluster.
The fact that questions are grouped in clusters related to the same topic is exploited by the question processor to improve the results of the system. This topic is usually present in the ¯rst question or represents the answer to the ¯rst question. Due to the fact that the current architecture does not allow us to feed answers back into the system, we consider as topic the ¯rst named entity of type person that appears in the ¯rst question of each cluster. If no such entity can be found, the ¯rst entity of the question is selected as the topic. If the question contains no named entities, the topic is considered to be the question focus. This approach was designed after empirical analysis of last year's questions. The topic is used to boost the retrieval scores of the Wikipedia articles describing it. 3
The cross-linguality of the system is ensured by a term translation module that takes as input the keywords identi¯ed during question processing and generates a ranked list of translation equivalents. Firstly, a list of translation equivalents mined from the mappings between the Romanian and English Wikipedias is used in order to obtain high quality translations of entities. This method provides a high precision, but low recall term translation.
The Inter-Lingual Index (ILI) between the Romanian [
For example, given the Romanian term plan general, the translation equivalents generated using the Romanian and English Wordnets are the following: general plan, general plane, general mind, general idea, general program, general sheet, general design, general cadre, general canvas, general programme, general inclined plane, general architectural plan
The ranking method then identi¯es the words that most frequently appear together in Wikipedia and eliminates those with infrequent use, and the resulted translations are: general design, general plan, general program, general idea, general programme 4
The purpose of this module is to provide the answer extractor with sentences from the document
collection that are relevant to the question. This is achieved by using the identi¯ed topic and the
translated terms. A two stage retrieval approach is employed: ¯rst, the most relevant documents
are selected, and then all their sentences are extracted and re-ranked according to their relevance
to the query. In order to do this the corpus was preprocessed and indexed using the Lucene
retrieval engine [
As last year, the answers had to be extracted from a heterogeneous collection consisting of two news corpora: Los Angeles Times from 1994 and Glasgow Herald from 1995, as well as English Wikipedia pages from November 2006. Wikipedia can be processed either as a static HTML dump, or in its native wikisource format. The former has a lot of content that is not part of the article itself and could have a negative impact on the answer extraction process, whereas the later is easier to process as it contains less information irrelevant to our purposes. After analysing the advantages and disadvantages of each format, we decided to convert the wikisource dump to plain text, preserving the information included in infoboxes, lists and tables, together with their markers. The inter-language links between the Romanian and English Wikipedias were also preserved in order to create the Romanian to English bilingual entity dictionary that was employed by the term translation module (see Section 3).
The text of the articles was indexed using a standard approach. No stopword ¯ltering was
employed, since stopwords are important for question answering. The index was enriched with a
stemmed version of the text obtained using Porter Stemmer [
The ¯rst retrieval stage uses the translated query to extract documents which might contain the answer to a question. As we previously mentioned, our system relies heavily on information extracted from Wikipedia. To this end, we ¯rst try to identify documents related to the topic of the question by retrieving documents which contain the question topic in their title. In cases where the topic is not reliably identi¯ed or no documents that contain the topic in their title can be found, documents which contain the query terms are retrieved.
In order to create complex queries that would rank higher the documents describing the target entities, we use a combination of phrase, fuzzy and proximity queries, as well as term boosting techniques o®ered by Lucene. A maximum of 5 documents from the query result set are considered. 4.3
The second stage of paragraph retrieval tries to identify sentences which are relevant to the
question. In order to do this, the documents retrieved at the previous stage are segmented into
sentences using LingPipe [
The answer extractor employed by our system relies heavily on the answer extraction modules provided by OpenEphyra. These modules implement several answer extraction strategies depending on the type of question, source of answer, and above all, expected answer type. In order to employ these components, our EAT hierarchy had to be mapped to the one used by OpenEphyra. In this way, most of the answers retrieved by our system are named entities of the type given by the expected answer type. This approach yielded good results for the categories identi¯ed by usual named entity recognisers (i.e. location, organization, person, number, date), but proved unsuitable for questions requiring generic answers such as Numiti doua instrumente la care canta Emerson./Name two instruments played by Emerson. Due to time restrictions, no attempt was made to change the ranking algorithm implemented by OpenEphyra. For this reason, in a large number of cases, the correct answer was among the ¯rst three answers retrieved by the system, but not the top one.
De¯nition questions are answered using a di®erent answer extraction module from the one provided by OpenEphyra. Our module relies on a cascade of high precision ¯lters designed after analysis of the data sources and experience gained in previous CLEF participations.
Wikipedia is a great source for de¯nitions. For this reason, whenever we try to ¯nd a de¯nition for a given term, Wikipedia is the ¯rst place to look for one. We start by locating Wikipedia pages which contain the term to be de¯ned in their title. In some cases this approach fails because the title of the page has a di®erent surface form even though it refers to the same concept (e.g. the page about CORGI has the title Council for Registered Gas Installers ). In these cases, we check whether the term to be de¯ned is used as the name of a ¯le (i.e. considering the previous example we check whether there is a Wikipedia ¯le called CORGI ). Once candidate pages are located, a set of patterns is applied in order to extract the de¯nition of the term. The patterns are designed in such a way that they can cater for situations where alternative forms are used to refer to the same concept (e.g. Steve Redgrave is referred to in an Wikipedia article as Sir Stephen Geo®rey Redgrave). The de¯nition of a term is considered to be the whole sentence to which a pattern can be applied.
If no de¯nition can be located in a Wikipedia page using the method described above, we then search the whole collection for sentences that contain not only the term to be de¯ned, but also other terms that might appear in a definition question (sometimes the question provides disambiguation clues for the term to be de¯ned, e.g. Ce este "bungo" in japoneza? /What is "bungo" in Japanese? ). A di®erent set of manually created patterns is applied to these sentences in order to extract the de¯nition of the term. In contrast to the de¯nitions extracted by the previous method, at this stage we extract only the noun phrase which is considered to de¯ne a term (e.g. Richard D. Farman is de¯ned as CEO of Southern California Gas Co.). 6
The evaluation results reveal an improvement in comparison with last year's accuracy. They are presented in Table 1.
Question Type De¯nition Factoid Lists Overall
As expected, the best results are obtained for de¯nition questions which can easily be answered thanks to the structure of Wikipedia pages. The main source of errors in the case of de¯nition questions is the incorrect translation of the term to be de¯ned and errors introduced by preprocessing tools. For example, one de¯nition was marked as inexact due to the fact that the NP extractor wrongly identi¯ed a bigger chunk of text which included the NP that constituted the answer. Translation errors also contributed to wrong answers or no answers being extracted for factoid questions. However, the main source of errors was the fact that no mapping between the generic expected answer type and the named entity classes covered by OpenEphyra was used. As a result, all the questions expecting generic answers were answered with NIL.
A large number of questions with numeric EAT were wrongly answered due to errors in the named entity recogniser employed by OpenEphyra. Numeric parts of date expressions were wrongly labelled as numbers and returned as answers (e.g. the number 12,1853 is wrongly extracted as answer from the date March 12, 1853 ). In many cases the correct answer was extracted as well, but received a lower con¯dence.
Currently our system does not have a way to deal with list questions and for this reason it returned NIL for all these questions.
The modular structure of our system also enables us to assess the performance of each individual component. The question analysis module identi¯es the question type with an accuracy of 98% and the expected answer type with 94%.
Empirical observation of the translation output indicates that there are still issues to be addressed in the future such as the ranking of the translation equivalents and translation of named entities especially when the questions contain words from several languages (e.g. La ce data a scris Mathieu Or¯la al sau "Tr¶aite des poisons"? /When did Mathieu Or¯la write his "Tra¶ite des poisons"? ). In this article we presented our participation in the Romanian to English Question Answering task at CLEF-2008. We employed a modular system consisting of the three standard stages of a QA system: question processing, paragraph retrieval and answer extraction. The question processor analyses Romanian questions and produces a detailed representation of each question including the terms it contains. These terms are then translated to English using several techniques based on the Romanian and English WordNets and aligned Wikipedia pages. Lucene is used to extract English paragraphs which constitute the input for the answer extractor employed. Due to time restrictions, some of the modules are in early stages of development or have been adapted from other projects. For example, the answer extractor is largely based on the one distributed with the OpenEphyra framework.
The existing components of OpenEphyra were designed to process English questions and retrieve answers from English document collections. For the future, we plan to use existing translation services such as Google Translate to obtain full translations of the Romanian questions as an additional source of information for our system. These translations can also be used by a monolingual English QA system such as OpenEphyra. Comparison between the two approaches can give us further insights into the best approach for cross-lingual question answering, and how and whether they can be combined. 8
This work has been supported by the EU funded project QALL-ME (FP6 IST-033860).