We describe our system for the monolingual Dutch and multilingual English to Dutch QA tasks. First, we present a brief overview of our QA-system, which makes heavy use of syntactic information. Next, we describe the modules that were developed especially for CLEF 2007, i.e. preprocessing of Wikipedia, inclusion of query expansion in IR, anaphora resolution in follow-up questions, and a question classification module for the multilingual task. We achieved 25.5% accuracy for the Dutch monolingual task, and 13.5% accuracy for the multilingual task.
The Question Answering task for CLEF 2007 contained two innovations. First, the document collection was extended with Wikipedia, the online encyclopedia that is available for many different languages. As described in section 3, we preprocessed the XML source files for this document collection so that we could index it adequately for the purposes of Information Retrieval. In addition, we extracted all relevant plain text, and parsed it.
Second, the test questions were grouped in topics. Within a topic, questions might refer to or presuppose information from previous questions or answers to these questions. We developed a simple anaphora resolution system (described in section 5) that detects anaphoric elements in a question, and tries to find a suitable antecedent in the first question of a topic, or in the answer to that question.
∗This research was carried out as part of the research program for Interactive Multimedia Information Extraction, imix, financed by nwo, the Dutch Organisation for Scientific Research.
In addition to these innovations, we also improved the Information Retrieval component of our QA system. In section 4, we show that query expansion based on (automatically acquired) synonym-lists and blind relevance feedback improves the mean reciprocal rank of the IR module.
In section 6 we describe a question classification module for the multilingual QA system, which uses both the question class assigned to the English source question and the class assigned to the automatically translated Dutch target question. This leads to a modest improvement.
The results of our system are discussed in section 7, and some suggestions for future work are given in section 8. 2
Joost
The architecture of our system is depicted in figure 1. Apart from the standard components question analysis, passage retrieval, answer extraction and answer ranking, the system also contains a component called Qatar, which collects all answers to questions of a specific type (i.e. birthdates) off-line. Answers to questions for which a Qatar-table exists are found by means of table look-up.
Alpino Passage Retrieval Answer Extraction
Question Question
Analysis Question Type
and
Keywords Potential Answers Answer Ranking Answer
Off-line raenlastwioenr extraction
The first processing stage is question analysis. The input to the question analysis component
is a natural language question in Dutch, which is parsed by Alpino. The goal of question analysis
is to determine the question type and to identify keywords in the question. Depending on the
question type the next stage is either passage retrieval or table look-up (using Qatar). If the
question type matches one of the table categories, it will be answered by Qatar. Qatar consists
of a number of manually written syntactic patterns for extraction of interesting relations (i.e.
creator-object tuples such as Heinrich Mann - Der Untertan). Recall is improved by using a set
of equivalence rules for syntactic dependency patterns
For all questions that cannot be answered by Qatar, we follow the other path through the QAsystem to the passage retrieval component. Instead of retrieving full documents, the IR module retrieves passages (see section 3). The 40 most relevant passages retrieved by IR are passed on to the answer extraction module. Here, we use patterns similar to those used by Qatar, but now slightly more general, to find the actual answer strings. Per sentence, at most one potential answer string is selected.
The final step is answer selection. Given a list of potential answers from Qatar or the IRbased QA module, the most promising answer is selected. Answers are ranked using various features, such as syntactic overlap between question and answer sentence, word overlap, proper name overlap, the reliability of the pattern used to extract the answer, and the frequency of the answer. The answer ranked first is returned to the user. 3
New in this year’s CLEF QA tracks was the inclusion of Wikipedia in the corpus. The Wikipedia corpus is different from the newspaper texts that were used so far in a number of ways. First of all, whereas the newspaper collection is relatively redundant (there are two newspapers covering the same period in the Dutch collection, and news stories tend to contain a fair amount of repetition), this is far less the case for the encyclopedia, which contains many facts that are mentioned only in one article. Thus, we expect redundancy-based techniques (typically using patterns that are noisy but provide high recall in combination with frequency-based ranking of results) to less effective for Wikipedia. Second, Wikipedia consists of structured web-documents, containing many lists, tables, and cross-references. In the newspaper collection, only article titles, and paragraphs are provided. By mining the structure of Wikipedia documents, it is possible to extract a large number of facts that cannot be found using syntactic patterns. Due to time constraints, we applied only the syntactic patterns that were developed for the newspaper collection.
An XML-version of the Dutch Wikipedia was provided by the University of Amsterdam.1 For IR and parsing, we were interested in obtaining just the text in each of the articles. We developed a series of stylesheets which removes material that was irrelevant for our task (i.e. navigation and pictures), and which returns the remaining content as highly simplified XML, containing only information that is required to identify the segmentation of the text into titles, sections, and lists. The segmentation is used in the IR index. From the simplified XML, plain text can be extracted easily. The result is tokenized and split into 4.7 million sentences. The sentences were parsed with the Alpino-parser.
The Qatar relation extraction module searches the corpus exhaustively for interesting facts, and stores these facts in a database. For Wikipedia, we used the patterns as they were developed for the newspaper corpus, with only minor modifications. In particular, we did not try to extract facts from lists, or using the XML structure.
Our IR system retrieves passages rather than complete articles. For previous CLEF tasks, we used the existing paragraph markup in the newspaper data to split documents into passages to be retrieved. For Wikipedia, similar markup exists but often refers to very small units, in many cases only single sentences. The Dutch Wikipedia corpus contains about 4.7 million sentences split into 1http://ilps.science.uva.nl/WikiXML/ about 2 million units. Single sentence units usually correspond to headers and subsection headers which often contain important keywords that match well with given queries. Unfortunately, including these as separate passages results in a strong preference for these units when retrieving passages. To avoid this we implemented a simple solution that merges consecutive units until they are bigger than a pre-defined size of 200 characters.
Despite its simplicity this approach works sufficiently well and made it possible to easily integrate the new Wikipedia corpus into our IR index. The same approach has also been applied to the newspaper corpus in order to create and index with a uniform segmentation. 4
In CLEF 2007 we submitted two runs of our system, applying two different settings of the information retrieval (IR) component which is used to retrieve relevant passages for a given question. The main difference between these two settings is the inclusion of query expansion techniques in one of them.
Common to both settings is the approach, previously described in
The second run includes various forms of query expansion. The main purpose of expanding
the query is to increase recall of the passage retrieval component in order to minimize the risk of
missing relevant information. We experimented with two general techniques for query expansion:
global methods using fixed lists and local techniques using blind relevance feedback. For the
latter we applied an implementation of the Rocchio algorithm for Lucene, LucQE
Furthermore, we used global expansion techniques using several lists of expansion terms.
Firstly, we used redirects from the Dutch Wikipedia. Redirects link search terms similar to
existing Wikipedia lemmas to corresponding articles. Redirects mainly cover spelling variations
but also include various synonyms. Secondly, we used synonyms of nouns, verbs, and adjectives
automatically extracted from word-aligned parallel corpora
A new feature in the 2007 QA task are follow-up questions. Questions are grouped in topics, consisting of a number of questions. Answering non-initial questions may require information from previous questions or answers to previous questions. The TREC QA task has included follow-up questions for a number of years. As no development data was available for the CLEF task, we used English examples from previous TREC QA tasks for inspiration.2 Note however that in TREC descriptive topics are explicitly provided, whereas in CLEF only an numeric topic id is given.
The most important aspect of follow-up questions is anaphora resolution, i.e. the process of detecting anaphoric phrases that depend on a previous antecedent expression for their interpretation, and assigning a correct antecedent to them.
A noun phrase was considered to be anaphoric if it was a personal (1-b) or impersonal (2-b) pronoun, a possessive pronoun (1-c), a deictic pronoun (3-b), an NP introduced by a deictic determiner (4-b), or an NP introduced by a definite determiner and not containing any modifiers (5-b). (1) (2) (3) (4) (5)
Which title was introduced by him? Who were his parents?
When was it founded?
When was the flag of the country designed?
Antecedents were restricted to named entities from the first question/answer pair of a topic. The
answer was chosen as antecedent if the initial question was one of a limited number of question
types which ask for a named entity (i.e. what is the capital of, who wrote/founded/.. , who
is the chair/president/.. of ). In other cases, the first named entity from the question was
chosen. We adopted this naive approach mostly because we lacked data to test and evaluate more
sophisticated approaches. Note also that quite a few TREC systems limit anaphora resolution to
resolving anaphoric expressions to the topic of the question (see
Our anaphora resolution system operates on the result of the syntactic dependency parse of the sentence. If anaphora resolution applies, and an antecedent is found, the set of dependency relations for the question is extended with dependency relations for the antecedent. That is, given an Anaphor resolved to Antecedent, for each dependency relation hHead, Rel, Anaphori in the question, we add a dependency relation hHead, Rel, Antecedenti. Note that, as the IR system described in the previous section constructs queries on the basis of the dependency parse of the question, this ensures that the Antecedent is also included in the IR query.
According to our inspection of the best monolingual run, there were 56 questions which required anaphora resolution. For 29 questions (52%), a correct antecedent for an anaphoric expression was found. In 15 cases (27%), a wrong antecedent was given. An important source of errors were cases where the answer to the initial question was correctly chosen as antecedent, but the answer 2i.e. trec.nist.gov/data/qa/2006 qadata/QA2006 testset.xml
IT
FQs was wrong. Incorrect antecedents also occurred when the intended antecedent was not (analysed by the parser as) a named entity. In cases such as (3-b) above, the antecedent is a common noun (ecological footprint). In cases such as ((4-b)) the antecedent (Russian-Turkish War) is analysed by the parser as an NP headed by a noun (war), and thus not recognized as a named entity. There were only a few cases where the antecedent was not in the first question/answer pair of a topic but in a later question.
12 cases (21%) were missed altogether by the anaphora module. These are due to the fact that no attempt was made to treat temporal anaphora such as toen, destijds (during that moment/period), and daarvoor (before this date), to treat locative uses of er (there), and a number of highly implicit anaphoric relations (i.e. given a question about the theme park de Efteling, the question which attraction opened in 1993? should be interpreted as asking about an attraction in de Efteling). A few antecedents were missed because we resolved at most one anaphoric element per question, whereas some questions actually contain two anaphoric elements (i.e. Who was he in the eighties version of the cartoon?).
Finally, there were 4 cases where anaphora resolution was triggered by an element that was not anaphoric (false alarms). These were all caused by the fact that relative pronouns were misclassified as deictic pronouns by the resolution component.
An interesting feature of CLEF is the fact that similar tasks are being executed for different languages. Follow-up questions were included in all tasks, but table 1 shows that the number of such questions varies considerably per target language. This suggests that the number of anaphoric expressions is also likely to vary considerably between tasks. 6
Our system for multilingual QA performs English to Dutch QA, i.e. questions are in English,
and answers are to be found in the Dutch document collection. English questions are translated
to Dutch using Babelfish/Systran. As explained in
This year, we improved the system by using newer (and much expanded) versions of Wikipedia, inclusion of redirect pages, and the online geographical database geonames3 for translation of geographical locations. Inspection of the translation results suggests that the coverage of these resources is quite good, although some problems remain. The use of redirects, for instance, causes Google to be mapped to the less frequent term Google Inc. Also, abbreviations tend to be replaced
377 200 200
MRR by their expanded meanings. Although both IR and the linguistic QA modules recognize many abbreviations and expanded terms as synonyms, this may still cause problems. An obvious case is a question asking for the meaning of an abbreviation. Another problem is the fact that many common words, which are not concepts occur as lemmas in Wikipedia. If no corresponding Dutch page exists, this causes some terms to show up untranslated in the Dutch question (i.e. for the adjectives French and Eastern, an English Wikipedia page exists, but no Dutch counterpart).
A second important aspect of QA-systems is question classification. As many automatically
generated translations are grammatically poor, parsing may lead to unexpected results, and, as
a consequence, question classification is often incorrect or impossible. To remedy this problem,
we also included a question classifier for English, which we ran on the English source questions.
We manually constructed a mapping from the question types used for English to the question
types used in Joost. We expected that such a mapping might give more accurate results than
classification of the automatically translated questions. Both the (mapped) English question type
and the Joost type assigned to the translated are used to find an answer to the question. Note
that question classification of the source language question is used in many MLQA systems (see
There are various question classifiers for English which use the question classes of
Joost uses over 40 question types, some of which correspond quite well to those of Li and Roth. Mismatches are problematic especially in those cases where Joost expects a more fine-grained class than the class produced by Li and Roth. For instance, Li and Roth classify what is the capital of Togo as loc:city whereas Joost has the class capital. Furthermore, the question classes assigned by Joost are not just labels, but typically consist of a label combined with one or more phrases from the question that are crucial for answering the question. I.e. the question what does NASA stand for? is assigned the type abbr:exp by Quest, whereas it is assigned the label abbreviation(NASA) by Joost. The mapping therefore tries to fill in missing arguments (usually names) on the basis of the syntactic parse of the Dutch translated question.
In many cases, the question class assigned by Joost is more helpful than the class assigned after mapping the English question class. An important exception, however, are questions that were assigned no class by Joost itself (usually, because the translated question contained syntactic errors that made the parser fail). In those cases, using a mapped question class is preferable over using no class at all. We tested our approach on data from previous years. The effect of including question classification based on the original question turned out to be small, however, as can be seen in figure 2. For the 2006 dataset, the effect was even negative.
4Available until recently at sds.colorado.edu/QUEST Dutch-mono Dutch-mono + QE En-Du
En-Du + QE
Factoids List Definition Temp. Restricted NIL The results from the CLEF evaluation are given in table 3. Table 4 gives results per question type for the best Dutch monolingual run. For 20 questions no answer was given (i.e. nil was returned by the system as answer).5 There are two main reasons for this: mistakes in anaphora resolution, which made it impossible to find documents or answers matching the question and lack of coverage of the question analysis component. Although there were 28 definition questions, only 18 were classified as such by Joost. List questions were an important source of errors.
The impact of adding Wikipedia to the document collection was significant. Although the text version of the Dutch Wikipedia is smaller than the newspaper text collection (approximately 50M and 80M words respectively), 150 of the 180 questions (i.e. over 80%) that received an answer were answered using Wikipedia.
Definition questions are answered using a relation-table that was created of-line. In addition to these, 24 questions were assigned a question type for which a relation-table existed. This number is lower than for previous CLEF tasks.
The system normally checks that answers suggested by the system do not occur in the question. It turned out that, in the context of follow-up questions, this filter needs to take into account anaphora resolution as well. That is, if a question contained an anaphor that was resolved to antecedent A, in some cases the system would still suggest A as an answer to the question, and such answers were not filtered (as A did not occur in the question string). 8
The inclusion of Wikipedia in the CLEF QA-task has made the task more realistic and attractive. We believe that performance on this task can be improved by taking the structure of Wikipedia more seriously, and by developing methods for relation and answer extraction that combine NLP with XML-based extraction.
Follow-up questions required the incorporation of a anaphora resolution component for questions. The current version of this module performs reasonably well, but its coverage should be extended (to cover locative anaphors and multiple anaphors). The proper treatment of lexical 5At the moment of writing, it is not clear to us whether there actually were questions for which nil was the correct answer. knowledge within the system remains an issue that requires more attention. The performance of the IR-module was improved using automatically acquired synonyms, but this knowledge has not been integrated yet in the relation and answer extraction modules.