The LogAnswer project investigates the potential of deep linguistic processing and logical reasoning for question answering. The paragraph selection task of ResPubliQA 2010 o ered the opportunity to validate improvements of the LogAnswer QA system that re ect our experience from ResPubliQA 2009. Another objective was to demonstrate the bene t of QA technologies over a pure IR approach. Two runs were produced for ResPubliQA 2010: The rst run corresponds to LogAnswer with standard con guration. The accuracy of 0.52 and c@1 score of 0.59 witness that LogAnswer has matured (in 2009, accuracy was 0.40 and c@1 was 0.44). In the second run, a special index that only indexes terms from the de niendum of de nitions was used for answering de nition questions. The resulting accuracy was 0.54 with c@1 score 0.62. For de nition questions, accuracy increased by 21%. The deep linguistic analysis of LogAnswer and its validation techniques made a substantial di erence compared to a pure IR approach. Using the retrieval stage of LogAnswer as the IR baseline, we found a 27% gain in accuracy and 37% gain in c@1 due to the powerful validation techniques of LogAnswer.
{ LogAnswer showed a very low accuracy for DEFINITION questions (16.8%).
One reason for the low accuracy for de nition questions was the restrictive way in which queries to the passage retrieval system were constructed; another problem was that the domain-speci c way in which de nitions are expressed in regulations was not recognized by LogAnswer. In ResPubliQA 2010, we have addressed these problems by improving the use of the retrieval system for de nition questions and by building a dedicated de nition index that also covers de nitions in the form typically found in regulations. { LogAnswer also showed a low accuracy for PROCEDURE questions (29.1%).
We now tackle the di culty to recognize sentences that describe a procedure by additional procedure triggers whose presence in the text marks a sentence as expressing a procedure. Moreover the question classi cation for PROCEDURE questions was incomplete (only 20.3% of these questions were recognized as such), so recognition rules had to be added to close this gap. { LogAnswer depends on the parsing quality of the linguistic analysis stage since only sentences with a full parse allow a logic-based validation. The document collections of ResPubliQA with their administrative language are very di cult to parse, though { in the last year, the parser used by LogAnswer only managed to nd a full parse for 26.2% of all sentences. Therefore one of our goals for ResPubliQA 2010 was to improve the parsing rate for JRC Acquis, and to ensure acceptable results for the new Europarl collection. { In order to achieve a general performance improvement, we decided to try a more thorough use of compound decomposition, knowing that compound nouns abound in administrative texts written in German.
Apart from evaluating the current state of the LogAnswer prototype, our main objective was that of demonstrating a clear advantage of using QA technologies over a pure IR approach. We also wanted to show that LogAnswer can cope with the novel challenges of ResPubliQA 2010, i.e. with the Europarl corpus and with OPINION questions.
In the paper, we rst explain how the LogAnswer QA system works and how we prepared it for ResPubliQA 2010. We then present the results of LogAnswer on the ResPubliQA question set for German. The discussion of these results and some additional experiments will highlight the strengths and some remaining weak points of the system. We conclude with a summary of the progress made. 2
LogAnswer rests on a deep linguistic analysis of the document collections by
the WOCADI parser [
SUB c17
OBJ SUBS herr.1.1 c14
SUB
c10 CIRC
E P TM
A G T
ATTR
MCONT c23 SB US c11 was.1.1
SUB
VAL
nachname.1.1
barroso.0
unterzeichnung.1.1
past.0
sagen.1.1
lexical overlap). For sentences with a full parse, a relaxation proof of the
question from sentence and background knowledge provides additional logic-based
features [
The questions are rst subjected to a linguistic analysis, using the WOCADI
parser for German [
Was hat Herr Barroso bei der Unterzeichnung des Nabucco-Abkommens gesagt? (What did Mr Barroso say at the time of signing the Nabucco agreement?) The relational structure4 of the MultiNet representation generated for the example question is shown in Fig. 1. The indexed symbols like sagen.1.1 are word sense identi ers. The MultiNet representation provides the basis for question classi cation and for the subsequent generation of a logical query.
Note that a complete MultiNet representation is only available if WOCADI nds a full question parse. This requirement was not problematic since WOCADI achieved a full parse rate of 97.5% on the ResPubliQA 2010 question set.
Apart from generating a semantic representation, the parser also provides the results of its morphological and lexical analysis stage. Note that the inclusion 4 The labeling of nodes with additional `layer attributes' is not shown for simplicity. of name lexica allows the parser to tag named entities in the text by types such as last-name (the family name of person), etc. WOCADI further provides information on the decomposition of compound words that occur in a sentence. In our running example, it nds out that nabucco-abkommen.1.1 is a regular compound built from abkommen.1.1 (agreement) and nabucco (an unknown word that could represent a name nabucco.0, or a regular concept nabucco.1.1).
This kind of morpho-lexical and named-entity information is also available for sentences with a partial or failed parse and can thus be used for implementing fallback methods that replace a logical validation in the case of a parsing failure. 2.2
A rule-based question classi cation is used to determine the expected answer type of the question and to identify the descriptive core of the question. The expected answer types known to LogAnswer are a re nement of the question categories of ResPubliQA. Apart from OPINION, PROCEDURE, PURPOSE, REASON, and DEFINITION questions, the system distinguishes several types of factoid questions such as city-name, mountain-name, rst-name, last-name, island-name, etc. These types correspond to the supported named entity types that WOCADI recognizes in the text.
For ResPubliQA 2010, the existing rule base for question classi cation was extended. In particular, the coverage of rules for PROCEDURE and PURPOSE questions was improved. In order to recognize compound triggers like Arbeitsverfahren (working procedure) or Hauptaufgabe (main task), LogAnswer now treats all nominal compounds that modify a known trigger word as additional trigger words for the corresponding question type. Finally, 25 rules for recognizing OPINION questions were added. The resulting rule base now comprises 240 classi cation rules, compared to 165 rules used for ResPubliQA 2009. 2.3
Experiments with a paragraph-level and document-level segmentation of the
texts have shown no bene t over sentence segmentation in the ResPubliQA 2009
task [
Prior to indexing, all documents in the considered JRC Acquis and Europarl
collections5 must be analyzed by WOCADI. In order to achieve an acceptable
parsing rate, some automatic regularizations of the documents were performed
(such as removal of paragraph numbers at the beginning of sentences). The
preprocessing also included the application of an n-gram recognizer for complex
references to (sections of) regulations, such as \(EWG) Nr. 1408/71 [
The pre-analysed sentences are stored in a Lucene-based retrieval system.6 Note that instead of word forms or stems, the system indexes all possible word senses for each sentence. Moreover nominalization relationships are utilized for enriching the index. A system of 49,000 synonym classes involving 112,000 lexemes is used for normalizing synonyms, i.e. a canonical representative is chosen for all terms in a given synonym class. A special treatment of compounds was added so that all parts of the compound are indexed in addition to the compound itself. Moreover, occurrences of expected answer types (like PERSON, DATE) in each sentence are indexed. The recognition of these answer types rests on the named entity information provided by WOCADI. The presence of certain word senses and regular expressions de ned on the morpho-lexical analysis of WOCADI can also trigger the recognition of these answer types. Currently there are 897 such triggers (including 704 newly added triggers for OPINION questions, and some additional triggers for the PROCEDURE type).
For de nition questions, a special de nition index was generated. In this index, only the de niendum of a de nition recognized in a sentence is used for indexing. For example, consider this de nition:
Hopfenpulver: Das durch Mahlen des Hopfens gewonnene Erzeugnis, das alle naturlichen Bestandteile des Hopfens enthalt. (Hop powder: the product obtained by milling the hops, containing all the natural elements thereof) Here, only the word sense hopfenpulver.1.1 of the de ned term Hopfenpulver (Hop powder) is added to the de nition index. This ensures a high retrieval precision for de nition questions. The recognition of de nitions in the texts was adjusted such as to cover the typical forms of de nitions in administrative texts.
For retrieving a set of candidate sentences, the JRC Acquis index and the Europarl index are searched in parallel (using Lucene's MultiSearcher), and the 200 best sentences for the given query are fetched.
The retrieval query is constructed from disambiguated word senses in the question analysis if a full parse exists. Otherwise a frequency criterion is used
to select a unique word sense from the set of alternatives for each word in the question. Synonyms are again normalized by choosing a canonical representative. In the example, the retrieval query becomes: herr.1.1 barroso.0 bei.1.1 unterzeichnung.1.1 nabucco-abkommen.1.1 nabucco.0 abkommen.1.1 sagen.1.1 The expansion of the compound Nabucco-Abkommen is intended to improve recall. The retrieval query will be extended by an additional term that expresses the expected answer type, in this case atype:OPINION. Note that the atype:x term is always an optional part of the retrieval query (it can be dropped at the expense of the retrieval score). This is di erent from the approach in ResPubliQA 2009 where for de nition questions, atype:DEFINITION was treated as a required subexpression so that sentences not recognized as containing a de nition were completely dropped. 2.4
As the basis for selecting the best retrieved sentence, several features that describe the quality of the question/snippet match are computed. We start by describing some shallow features that can be computed for arbitrary sentences regardless of the success of parsing. Apart from the obvious expected answer type check, another important method is a lexical overlap test. To this end, LogAnswer determines the list of all (synonym normalized) word senses for each word in the question (except stopwords). Each list of alternative word senses is treated as a disjunction one of whose elements must nd a match in the sentence to be validated. In our runnung example, we get: (herr.1.1) (barroso.0) (signieren.1.1 unterzeichnung.1.1) (nabucco-abkommen.1.1 nabucco.0 nabucco.1.1) (abkommen.1.1 nabucco-abkommen.1.1) (sagen.1.1 besagen.1.1 sagen.2.1), where the canonical synonym signieren.1.1 replaces the original unterzeichnen.1.1.
A recent change to LogAnswer is the treatment of nominal compounds: Each compound is split in two conjuncts so that a full match is possible if the text either contains the compound directly (Nabucco-Abkommen), or alternatively, if it contains the components of the compound (i.e. both Nabucco and Abkommen).
In the example, this candidate sentence is found that answers the question:
Am 13. Juli bei der Unterzeichnung des Nabucco-Abkommens in Ankara
sagte Herr Barroso, die Gas-Pipelines seien aus Stahl. (On 13 July in
Ankara, at the time of signing the Nabucco agreement, Mr Barroso said
that the gas pipelines were made from steel.)
LogAnswer then extracts the following (synonym normalized) word senses and
numerals from the morpho-lexical analysis of this sentence:
abkommen.1.1, ankara.0, barroso.0, familiename.1.1, gas.1.1, gaspipeline.1.1,
herr.1.1, monat.1.1, nabucco-abkommen.1.1, name.1.1, past.0, pipeline.1.1, present.0,
sagen.1.1, sein.3.8, stadt.1.1, stahl.1.1, tag.1.1, unterzeichnung.1.1, 7, 13.
Here, every list of alternative word senses from the question representation nds a
match in the shallow sentence representation. Thus, there are 0 cases of matching
failure (100% matching rate). For details on the shallow features, see e.g. [
For sentences with a full parse, a relaxation proof of the (logical
representation of the) question from the logical representation of the sentence and the
available background knowledge is also tried, resulting in additional logic-based
features. The prover works on synonym-normalized representations. The
background knowledge comprises more than 10,500 facts (e.g. describing
nominalizations), and 114 rules for basic inferences, see e.g. [
Recalling our running example, the following logical query is constructed, based on the question parse and the result of question classi cation: attr(X1; X2); sub(X1; herr.1.1); val(X2; barroso.0); sub(X2; familiename.1.1); obj(X3; X4); subs(X3; unterzeichnung.1.1); sub(X4; nabucco-abkommen.1.1); agt(X5; X1); circ(X5; X3); subs(X5; sagen.1.1); mcont(X5; F ) All variables are assumed to be existentially quanti ed. Comma means conjunction. The variable F is the question focus (it expresses the queried information).
The logical representation of the correct answer sentence shown above is: val(c10; c7); sub(c10; monat.1.1); obj(c14; c17); subs(c14; unterzeichnung.1.1); loc(c17; c262); sub(c17; nabucco-abkommen.1.1); origm(c180; c257); pred(c180; gaspipeline.1.1); arg1(c182; c180); arg2(c182; c257); temp(c182; present.0); subs(c182; sein.3.8); assoc(c22; c14); mcont(c22; c182); agt(c22; c31); temp(c22; c8); temp(c22; past.0); subs(c22; sagen.1.1); attr(c24; c25); sub(c24; stadt.1.1); val(c25; ankara.0); sub(c25; name.1.1); sub(c257; stahl.1.1); in(c262; c24); attr(c31; c32); sub(c31; herr.1.1); val(c32; barroso.0); sub(c32; familienname.1.1); attr(c8; c10); attr(c8; c9); val(c9; c6); sub(c9; tag.1.1); assoc(gaspipeline.1.1; gas.1.1); sub(gaspipeline.1.1; pipeline.1.1); sub(nabucco-abkommen.1.1; abkommen.1.1) Due to a parsing error, the logical query shown above cannot be proved from the representation of the sentence { the parser has used an unspeci c assoc relation instead of the circ relation in the query. Thus, the critical literal will be skipped from the query, resulting in a proof of the remaining query fragment.
Among the logic-based features that summarize the relaxation proof, there is
one feature reporting that a single literal had to be skipped, and another feature
reporting that 10=11 91% of the query literals have been proved. For details
on these logic-based features and the use of relaxation in LogAnswer, see [
The selection of the best answer candidate is based on the shallow features
obtained by matching the question and the sentence terms, and (for sentences with
a full parse) also on features obtained from a relaxation proof of the question
from the candidate sentence. Rank-optimizing decision trees [
The c@1 evaluation metric of ResPubliQA rewards QA systems that validate their answers and prefer not answering over wrong answers. Thus results with a low validation score should be dropped since their probability of being correct is so low that showing these results would reduce the c@1 score of LogAnswer. The threshold = 0:09 for accepting the best answer, or generating a `NO ANSWER' response if the validation score falls below the threshold, was chosen such as to optimize the c@1 score of LogAnswer on the ResPubliQA 2009 test set.
Finally, if the best sentence is not rejected, then it is expanded to the corresponding full paragraph, as required by the ResPubliQA PS task. 2.7
E-KRHyper [
While we developed this prover for embedding in knowledge representation
applications, it can operate as a stand-alone theorem prover which accepts input
problems in the syntax used by the TPTP logic problem library [
E-KRHyper has several features which support its role as a reasoning server within an application like LogAnswer. Logic extensions like equational reasoning,
negation as failure, arithmetic evaluation and list processing enable the prover to handle aspects of knowledge representation systems which cannot be expressed within the limits of rst-order logic. Most ATP systems are designed to work on a single problem only. They terminate once they have found a result, and they must be started anew for each problem. This mode of operation would be impractical for a reasoning server within LogAnswer, as it would entail reloading the extensive logical knowledge base for each query, every time rebuilding the indexing structures which the prover requires for fast clause access. Instead EKRHyper can remain in operation inde nitely, loading the knowledge base only once during the initialization of LogAnswer. Any additional clauses required by the queries can be loaded and retracted during the operation.
For query relaxation E-KRHyper supplies LogAnswer with information about partially successful proof attempts. LogAnswer selects the most promising way to relax the query, and then the prover continues with the shortened query, keeping any previous derivation results to avoid repeating inferences.
When E-KRHyper nds a proof, it extracts the answer substitution from the proof and transfers it to the main LogAnswer system for further processing. 3
Results on the ResPubliQA 2010 Test Set for German Two runs were produced for ResPubliQA 2010: The rst run, loga101PSdede, represents the LogAnswer system in its standard con guration without the experimental de nition index. In loga102PSdede, by contrast, the de nition index was activated. While the system was con gured to retrieve 200 candidate sentences for each question, the actual number of available sentences was smaller in some cases. In the loga101PSdede run, a total of 39,200 candidate sentences was retrieved (196 per question). About 37.0% of the retrieved candidate sentences had a full parse, thus allowing logical validation. The remaining candidate sentences with a chunk parse (41.9%) or failed parse (21.1%) were only subjected to a shallow validation (numbers for the loga102PSdede run are similar). The results obtained for the two submitted runs are shown in Table 2. The use of the de nition index in loga102PSdede was a clear improvement.
Table 3 shows the results of corresponding shallow-only runs (generated with the prover switched o ), and the results of two IR baseline runs in which the top-ranked sentence of the retrieval stage was directly used for choosing the corresponding answer paragraph. Note that a di erent `no answer' threshold of = 0:76 was used for the IR baseline runs. In this case the treshold was used for cutting o results with a poor Lucene retrieval score. The value of = 0:76 was again chosen such as to optimize the corresponding c@1 score on the ResPubliQA 2009 questions. The table clearly shows that the use of logical validation techniques provided no extra bene t in the ResPubliQA task { the results of the standard system (with logical validation) and of the con guration that only uses shallow features for validation was about the same. Comparing the detailed results of the runs of the full system (loga101PSdede) and the shallow only conguration (SH-101), we found that the validation models resulted in a di erent choice of shown answer only for 15% of the questions. For loga102PSdede vs. SH-102, the chosen answer paragraphs were di erent for 14% of the questions. Given the high accuracy achieved by the shallow validation technique, logical validation was obviously not called for by the ResPubliQA task. An interesting nding was that (with one exception), all questions for which deep validation outperformed the shallow-only technique were DEFINITION questions.
On the other hand, the validation techniques of LogAnswer achieved a strong bene t compared to using the retrieval score only. Comparing loga101PSdede and the IR-101 run, for example, accuracy increases by 27% and the c@1 score increases by 37% due the use of validation instead of the plain retrieval result.
A breakdown of results by question category is shown in Table 4. LogAnswer
was best for REASON/PURPOSE and FACTOID questions. DEFINITION and
OTHER questions performed worst. OPINION questions also proved di cult.
The use of a special de nition index in the second LogAnswer run increased the
accuracy for de nition questions from 34% to 41% compared to the rst run with
a single index for all questions. This result is encouraging, though more work has
to be done here. Compared to the results of LogAnswer in ResPubliQA 2009 [
Results on the success rate of question classi cation are shown in Table 5. Despite the high overall recognition rate (81.5%), the novel class of OPINION questions was obviously not yet well covered by the classi cation rules. Moreover, some more ways of expressing PROCEDURE questions should be supported.
The threshold = 0:09 on validation quality that serves for cutting of poor answers was chosen such as to maximize the c@1 score of LogAnswer on last year's ResPubliQA question set. In retrospect, the optimal threshold for loga101PSdede would have been = 0:11, resulting in a c@1 score of 0.60 (overall accuracy 0.52). For loga102PSdede, the optimal threshold would have been = 0:13, yielding a c@1 score of 0.63 (overall accuracy 0.54). The closeness of these optimal values to the results obtained using = 0:09 demonstrates that the method for determining the NOA threshold was e ective.
We have determined the reason of failure for a sample of questions with
wrong answers in the LogAnswer runs. This analysis has shown that the Lucene
scoring metric in the retrieval stage of LogAnswer has a strong bias to short
sentences. A short sentence that contains only one term from the IR query is
often prefered to a longer sentence that contains all query terms. We thus agree
with Perez at al [
In ResPubliQA 2010, LogAnswer scored much better than in the last year. The improved accuracy for all question types shows that the general improvements of LogAnswer and the thorough utilization of compound decompositions were e ective. Speci cally, PROCEDURE questions are no longer problematic for LogAnswer. Compared to a system con guration with a single index, the accuracy for de nition questions was increased by 21% by adding a specialized de nition index. The large di erence between the c@1 and accuracy scores of LogAnswer indicates a good validation performance.
Due to the low parsing rate for JRC Acquis and Europarl, we were unable to demonstrate a bene t of logical processing on the quality of results. However, as witnessed by the results of the shallow-only matching technique, the ResPubliQA task did not seem to call for sophisticated logic-based validation either.
The deep linguistic analysis of LogAnswer and its validation techniques made a substantial di erence compared to a pure IR approach. Using the retrieval stage of LogAnswer as the IR baseline, we found a 27% gain in accuracy and 37% gain in c@1 due to the powerful validation techniques of LogAnswer.