The German question answering (QA) system IRSAW (formerly: InSicht) participated in QA@CLEF for the fifth time. IRSAW was introduced in 2007, by integrating the deep answer producer InSicht, several shallow answer producers, and a logical validator. InSicht realizes a deep QA approach: it transforms documents to semantic representations using a parser, draws inferences on semantic representations with rules, and matches semantic representations derived from questions and documents. InSicht was improved for QA@CLEF 2008 mainly in the following areas. The coreference resolver was trained on question series instead of newspaper texts in order to be better applicable for follow-up questions in question series. Questions are decomposed by several methods on the level of semantic representations. On the shallow processing side, the number of answer producers was increased from 2 to 4, by adding FACT and SHASE. The answer validator introduced in the previous year was replaced with the faster RAVE validator designed for logic-based answer validation under time constraints. Using RAVE for merging the results of the answer producers, monolingual German runs and bilingual runs with source language English and Spanish were produced by applying a machine translation web service. An error analysis showed the main problems for the precision-oriented deep answer producer InSicht and the potential offered by the recall-oriented shallow answer producers.
The German question answering (QA) system IRSAW (Intelligent Information Retrieval on the Basis of a Semantically Annotated Web) employs deep and shallow methods. The deep answer producer is InSicht, which transforms documents to semantic representations using a syntactico-semantic parser, draws inferences on semantic representations with rules, matches semantic representations derived from questions and documents, and generates natural language answers from the semantic representations of relevant documents. Specialized modules refine the semantic representations in several directions: resolving coreferences in documents (and questions) and resolving temporal deixis in documents. To provide a robust strategy for difficult text passages or passages mixing text and other elements, four shallow answer producers are employed. The resulting five streams of answer candidates, which are produced in parallel, are logically validated and merged by RAVE. Based on the results of validation, RAVE scores the answer candidates and selects the final results. 2 The deep answer producer InSicht was changed in three main aspects that are described in the following subsections. 2.1
In contrast to last year, we retrained the coreference resolver CORUDIS
Question decomposition was systematically added to InSicht for QA@CLEF 2008. A decomposition method tries to simplify answering complex questions by first asking a subquestion whose answer is used to form a revised question which is often more easy to answer than the original question.1 For example, question decomposition for Welches Metall wird zur Goldwa¨sche benutzt?/‘Which metal is used for washing gold?’ (qa08 192) leads to the subquestion Nenne Metalle/‘Name metals’ with answers like Eisen/‘iron’ and Quecksilber/‘quicksilver’ and the revised question Wird Quecksilber zur Goldwa¨sche benutzt?/‘Is quicksilver used for washing gold?’ Note that answers to original questions found by question decomposition often require support for the answered subquestions and the revised question, i.e. the answer to the original question is supported by sentences from different documents.
To evaluate question decomposition after QA@CLEF 2008, we annotated all German QA@CLEF
questions since 2003 with decomposition classes (see
Adding features to the deep producer InSicht yields better results, but often with a longer runtime. Therefore, several performance improvements were tried. As query expansion by logical rules (applied in backward chaining) expands the search space dramatically, the search space should be reduced by some efficient heuristics that do not eliminate good answers. To this end, statistics on successful rule applications (i.e. combinations of logical rules that led to at least one correct answer) were collected from the test collections of QA@CLEF from 2003 to 2007 and some separate question collections. When restricting query expansion to successful rule combinations, results for the QA@CLEF 2008 questions stayed stable while runtime decreased by 56%. This simple technique turned out to be very effective. 3
In addition to the deep producer, IRSAW now employs four shallow producers of answer candidates:
QAP
SHASE uses the semantic network representation of both question and document sentences to
produce answer candidates. The core node representing an answer node is identified in the question semantic
network (i.e. the question focus node determined by the syntactico-semantic parser). To find answer
candidates, the semantic relations for the core node, its semantic sort, and its semantic entity are calculated;
see
The answer candidates in the InSicht stream and the shallow QA streams are validated and merged by RAVE (Real-time Answer Validation Engine), a logic-based answer validator designed for real-time QA. It is crucial for the efficiency of RAVE that no answer must ever be parsed at query time – computing a deep linguistic analysis for hundreds of extracted answer candidates during validation is not realistic in the real-time QA setting. This problem is solved by using logic only for validating support passages, i.e. for 2The correct answer could also be found directly without decomposition.
3The relation type name2date of death is viewed as the first component of the triple. Other common date formats are explicitly generated and indexed as well because no normalization takes place at this level, yet. deciding if the given passage contains a correct answer at all. Establishing the relationship between the considered answer candidate and the information obtained from the question-passage proof is not part of the logic-based processing. Therefore a logical analysis of the answer candidate is no longer needed for validation. Moreover, one question-passage proof is sufficient per passage. The deep features determined by proving the question from the support passage can be re-used if several answer candidates were extracted from the same passage. The parses of the sentences needed for the question-passage proofs are computed before indexing and fetched along with retrieving the support passages.
Local validation scores are determined by an ML method applied to shallow and (if available) also
logic-based features. Separate models were trained for each producer in order to tailor the validation
criterion to the characteristics of each answer stream. Notice that both a combined model (using the full set
of deep and shallow features) and a shallow-only model was generated for each stream. Training data was
obtained from a run of the system on the QA@CLEF 2007 test set for German. A total of 21,447 answer
candidates extracted from 27,919 retrieved passages were annotated as the basis for machine learning.
Preparatory experiments based on cross-validation on the training set have shown that bagging of decision
trees with reweighting of training examples is particularly suited for the task. The local ML-based scores,
which estimate the probability that an answer is correct judging from a specific supporting snippet, are
then aggregated in order to determine the total evidence for each answer candidate. RAVE uses a novel
aggregation model for that purpose, which aims at robustness against duplicated information; see
RAVE uses an anytime validation technique based on incremental processing of the available answer streams. In order to implement incremental validation, RAVE maintains a priority queue for retrieved passages and a second priority queue for pending answer candidates. The processing loop of RAVE works as follows: If more than t = 100 ms have elapsed since the last attempt of reading new passages or new answer candidates, then RAVE reads all passages and answer candidates that have arrived in the meantime. By applying a model learned from passage annotations to the shallow passage features, each passage is assigned a quality score which estimates the probability that the passage contains a correct answer, and the shallow features are cached. Passages with a deep parse are added to the priority queue for later logic-based processing.
The newly arriving answer candidates are first checked for violation of sanity checks. If an answer passes the test, a score based on shallow features is computed. If the supporting passage from which the answer candidate was extracted has a deep parse, then the candidate is added to the priority queue of answer candidates awaiting deep validation. If the supporting passage has no deep parse, however, then the score based on shallow features is directly aggregated, provided that it exceeds a given quality threshold. Having integrated all newly arriving data, the system fetches the answer candidate with the best shallow score from the queue of answer candidates. If no logic-based features are already cached for the corresponding support passage yet, the passage is subjected to logical processing by trying to prove the question literals from the logical representation of the passage and storing the resulting logic-based features. Then a ‘deep’ answer score based on the combination of deep and shallow features is computed and the result is aggregated if it exceeds the quality threshold. In the event that the priority queue of answer candidates is empty, the available time is utilized by pre-computing logic-based features for the best item in the priority queue of support passages. This will speed up the later evaluation of answer candidates extracted from this passage since the logic-based features can then be fetched from the cache.
This incremental reading/processing loop is repeated until all streams are exhausted or until the specified time limit is exceeded. The system then iterates over all remaining items in the priority queue of answer candidates. If logic-based features are already cached for such a remaining answer candidate, the candidate is assigned a deep score; otherwise only shallow features are used. In any case, the validation 4The version of RAVE used for the submitted runs was still lacking the test for compatibility of measurement units of question and answer and the test for fulfillment of temporal restrictions described in the reference. Two additional features were incorporated for the QA@CLEF runs that were not available in the Answer Validation Exercise, viz irScore (retrieval score for the supporting passage determined by the IRSAW retrieval system) and producerScore (a quality score assigned by each answer producer when generating an answer candidate). score is aggregated provided that it exceeds the quality threshold. Finally, RAVE determines the three distinct answers with the highest aggregated score. These answers are returned together with the best supporting snippet found for each answer. If there is no aggregated evidence for any answer at all, then a NIL answer with zero confidence is generated. Answer candidates generated by InSicht are always directly aggregated – these answers result from a precision-oriented QA technique and do not require logical validation. Since InSicht works independently of the passage retrieval step, RAVE lacks the morpho-lexical information needed for computing shallow passage features and assigning a validation score in the usual way. Two methods for assigning validation scores to the InSicht results were tried. The first was training an ML-based classifier using a special set of features for InSicht, which consists only of the producerScore assigned by InSicht itself and an occurrences count for the number of alternative justifications that InSicht has found for the answer. The second method was directly using the self-assessment of InSicht, i.e. the producerScore, as the local validation score for answer candidates contributed by InSicht. 5
All runs with prefix fuha081 were generated using the ML-based validation scores for InSicht, whereas
the runs with prefix fuha082 used the self-assessment of InSicht. For bilingual QA experiments, the Promt
Online Translator5 was employed to translate the questions from English or Spanish to German. From
experience in previous CLEF campaigns, it was expected that this web service would return translations
containing fewer errors than other comparable web services for machine translation, which becomes
important when deep NLP is applied, i.e. when the translated questions are parsed. However, we found that, in
this year, Promt offers a new machine translation service (in beta status) and experiments using translations
from other web services had a higher performance, see Leveling and
We submitted two runs for the German monolingual task in QA@CLEF 2008 and four bilingual runs with English and Spanish as source language and German as target language (see Table 1). The syntacticosemantic parser employed in InSicht was used to provide an approximate complexity measure for the German questions by counting the semantic relations in parse results (after coreference resolution). This showed a decrease compared to previous years: 9.05 relations per question on average (2007: 11.41; 2006: 11.34; 2005: 11.33; 2004: 9.84). q.error q.parse error q.no parse q.chunk parse q.incorrect coreference q.incorrect parse q.ungrammatical d.error d.parse error d.no parse d.chunk parse d.incorrect parse d.ungrammatical q-d.error q-d.failed generation q-d.matching error q-d.missing cotext q-d.missing inferences error on question side question parse is not complete and correct parse fails only chunk parse result a coreference is resolved incorrectly parser generates full parse result, but it contains errors question is ungrammatical error on document side document sentence parse is not complete and correct parse fails only chunk parse result parser generates full parse result, but it contains errors document sentence is ungrammatical error in connecting question and document no answer string can be generated for a found answer match between semantic networks is incorrect answer is spread across several sentences inferential knowledge is missing
In the bilingual experiments with English and Spanish about 60% and 40%, respectively, of the performance (measured in right answers) for monolingual German were achieved. Results may have been better with another machine translation service for QA@CLEF 2008.
The evaluation of dialog treatment for the 2008 questions showed that the coreference resolver performed correctly, with one exception: The anaphors in the four questions that referred to the answer of the preceding question were incorrectly resolved because this case was not allowed in the trained coreference model (see Sect. 2.1).
Table 2 contains an error analysis for the deep answer producer InSicht with a predefined
classification. The classes are problem classes that lead to not finding the correct answer; the same classes
were used for our first participation, QA@CLEF 2004
The performance of the shallow QA subsystem has also been assessed. For the 200 questions, a total number of 36,757 distinct supporting passages was retrieved (183.8 per question). 1,264 of these passages contain a correct answer, i.e. the precision of passage retrieval is 3.44%. For 165 of the questions, there is at least one passage that contains an answer to the question. Since these passages form the basis for answer extraction by the shallow producers MIRA, QAP, FACT and SHASE, this means that for perfect answer extraction and validation, it would theoretically be possible for the shallow subsystem to answer 165 non-NIL questions correctly (or 175 questions including the NIL case). More details on the number of available correct passages for each question are shown in Figure 1.
The actual extraction performance achieved by the answer producers of the shallow subsystem of IRSAW has also been investigated, see Table 3. The following labels are used in the table: #candidates (average number of extracted answer candidates per question), #answers (average number of right answers per question), precision (correctness rate of answer extraction, i.e. #answers=#candidates), pass-rate (fraction of the 1,264 correct passages from which a correct answer is extracted), pass-prec (precision of answer s itno 20 s e u #q 15
20 30 40 #retrieved passages with correct answer 50 60 extraction for correct passages), #answered (number of questions for which at least one right answer is extracted), and answer-rate (answered questions divided by total number of questions with a correct supporting passage, i.e. #answered=165 in this case). As witnessed by the answer-rate of 0.8 for all shallow producers in combination, the answer candidates extracted by the shallow producers cover most of the correct answers contained in the retrieved passages. However, the precision of answer extraction is very low (only 3% when considering all producers), and even for those passages that contain an answer, the precision of extraction (i.e., pass-prec), is only 29%. While the strong recall-orientedness of the shallow subsystem provides a good basis for answer selection, the very low precision also means that the burden of spotting the correct answers is shifted to the validation component.
Assuming perfect validation, it would be possible to answer 132 non-NIL questions correctly based on
the results of the shallow subsystem (or 142 if one includes the NIL questions). There are even more correct
answers if InSicht is also taken into account. However, subsequent processing by RAVE only resulted in 46
correct answers in the best submitted run (fuha082dede). This clearly demonstrates that improvements of
the validator are necessary. While some minor bugs of the validator have already been fixed (see description
of recent changes in
Another problem is the incomplete implementation of the answer-type test which checks the compatibility of the expected answer type of the question and the found answer type. This test is limited
QAP SHASE FACT all
10 15 processing time (s) 20 25 to a few special answer types at the moment, and it is even more restricted for support passages with a failed parse.
The third problem is concerned with the training set of RAVE. Due to unstable operation of IRSAW when the training set was generated, the annotations cover only 151 questions of the QA@CLEF07 test set and less than 30 definition questions. In order to provide a suitable basis for machine learning, a considerable increase in the number of annotated questions is necessary. Moreover the training data for SHASE is not representative of the current version of the producer since SHASE was apparently broken when generating the training set and hardly produced any correct answers. Therefore the machine learning approach did not result in a useful model for SHASE.
Despite this need for additional validation features and a larger, up-to-date training set, the observed processing times for RAVE confirm that the validator is suitable for application in real-time QA. The processing times for a complete logical validation, i.e. without using a time limit, are shown in Figure 2. The average time needed for answer validation and selection is 1.48 seconds per question.6 This process involved an average of 79 question-passage proofs per question, which cover all parseable snippets retrieved by IRSAW. Notice that the complete logical validation takes less than 0.95 seconds for half of the questions, and less than 2.45 seconds for 90% of the questions. By specifying a time limit, these processing times can be constrained even further. 7
The QA system IRSAW was successfully improved in several ways for QA@CLEF 2008. Coreference resolution for questions was strengthened by generating suitable training data. Question decomposition in the deep answer producer InSicht opens interesting ways to a fusion of information from different documents or corpora. With increasing system complexity, runtime performance becomes critical, but specialized optimization techniques allow to provide useful answers in near real-time. The latter aspect is still open for improvements in the future, especially with the advent of computers with more and more CPU cores. Adding two more shallow answer producers turned out beneficial for robustness, although the integration in the validator must be improved further. The first prototype of the RAVE answer validator already demonstrates that logic-based processing and real-time answer validation can be reconciled, but additional validation features and an improved training set must be provided in the next development phase.
6Processing times were measured by running RAVE in a single thread on an Athlon64X2 4800+ CPU with 2.4 GHz clock rate.