The University of E´vora participation in QA@CLEF2008 was focused on the Portuguese monolingual task and was based on the updated Senso Question Answering System. This system uses a local knowledge base, providing semantic information for text search terms expansion. The solver module uses two components to collect plausible answers: the logic and the ad-hoc solvers. The logic solver starts by producing a FirstOrder Logic expression representing the question and a logic facts list representing the texts information and then it looks for answers within the facts list that unify and validate the question logic form. The ad-hoc solver is designed for cases where the answer can be directly detected in the text. Then all the results are merged for answer list validation, to filter and adjust answers weight. The submitted run had only single answers (the system best answer). The overall accuracy was 46.5% and the overall Confidence Weighted Score was 0.23979. This paper has an overview of the system and its approach to QA@CLEF.
The system mantains the usage of a common sense knowledge base to assist on sentence analysis
and text retrieval. The participation on QA@CLEF-2007 [
The next section explains the system architecture. The followed methodology is described with examples in section 3. The evaluation of the obtained results is presented in section 4. Finally, some conclusions and future work are pointed out in section 5. 2
The main components are the same that were present in 2007. The updates occured only inside some modules. Figure 1 illustrates the system major modules: Libs, Query, Solver, Local KB and Web Interface.
The Query Module.performs the question analysis and selects a set of relevant documents for each question, as explained in section 3. The query group identifier determines if a query will be associated with the first from that group, to work with the same topic.
The Libs Module manages the text collections: news documents from Pu´blico and Folha de Sa˜o Paulo from years 1994 and 1995, plus the Wikipedia documents. These collections are seen as libraries that contain the information needed for question answering. All texts are indexed with Lucene3, a full-featured text search engine, with a Senso personalized stemmer for Portuguese.
The Local KB module has a starting knowledge base, containing common sense facts about places, entities and events. The information is available to a logic tool with some inference capabilities, that helps the automatic capture of sentence meaning.
Most of the changes happened in the Solver Module. It performs a search for plausible answers using two parallel approaches: • logic solver : a logic-programming based tool that looks for answers to a question, being aware of the semantic expressed in Local KB • ad-hoc solver : case-based answer detection for questions where the answer can be directly detected in the text
The results are merged, forming a global and weight sorted list of candidate answers. This list
is then processed by the new component: the answer validator. Some answer values are refused
if they are not in accordance with the question type. Besides filtering, each answer weight may
suffer an adjustment to a more reliable value. The Web redundancy can be exploited as a method
for answer validation in QA [
The Web Interface layer allows an easier usage of the system. It is possible to consult text documents or check each intermediate step in the question analysis process and answer search. Next section explains the QA process. 3
The Senso approach for QA was firstly inspired on the authors previous work [
The XML collection files were processed and split into single texts, along with their metadata. The Libs Module keeps all these individual documents, being aware of their collection temporal context. Because we needed to perform some text search operations, the collections were indexed.
Each text was processed with Palavras[
Each question is also processed with the syntactical parser Palavras and a semantic analyzer
able to obtain a partial semantic representation. The technique used for this process is based on
Discourse Representation Structures (DRS) [
Let us consider the following QA@CLEF question:
Que instrumento tocava Ringo Starr ? (What instrument did Ringo Starr play ?)
The morpho-syntactical representation for the question, in figure 2, shows the parser tags identifying the subject, the predicate and the interrogative form que (What ). Figure 3 has the DRS for the same question, with the semantic representation used by the system for later logic inference process. The system will search for something that is an instrument. At this point, QUE:fcl =SUBJ:np ==>N:pron-det(’que’ <interr> M/F S) ==H:n(’instrumento’ <tool> <tool-mus> M S) =P:v-fin(’tocar’ IMPF 1/3S IND) =ACC:prop(’Ringo_Starr’ <hum> M S) =
Que instrumento tocava Ringo_Starr ? a preliminary document retrieval task is done with Lucene, in order to elect a set of potentially relevant documents for each question. If no candidate documents are found the system cannot find an answer and the result is NIL.
The Query Module creates the Lucene search query. This is done with the question text terms and, for some, with their related terms as expressed in Local KB. So, when we are looking for instrument it is important to get documents that include synonyms or specialization terms (as 4Tool developed by Eckhard Bick. VISL Project: http://visl.hum.sdu.dk/visl query(clef08qa0045, [ q(_220, ’que’ , [’M/F’, ’S’, ’que’ ],
[ modif(nd,’instrumento’, [’M’,’S’,’instrumento’] ) ] ), name(_221, ’Ringo_Starr’ , [’M’, ’S’, ’Ringo_Starr’ ],
[ ] ), ’tocar’(_220,_221,
[ modif(verb,’tocar’, [’IMPF’,’1/3S’,’IND’] ) ] ), [ ] ], [ ref(_220), ref(_221) ] ). piano, drums and other), because those documents might be relevant as a possible answer source. When the question belongs to a cluster and it is not the first from that group, the query is fed with more terms in order to include the implicit topic. The system goes back to that cluster’s first question and gets their search terms and answer into the Lucene query. 3.3
The Solver Module is responsible for finding a list of answers to a question, appreciate them and elect one as the best system answer. The list of candidate documents is processed in parallel by two different tools: the logic solver and the ad-hoc solver.
The semantic analyzer used before for the query will now create a DRS list for the selected texts. This list is a question dedicated Knowledge Base: the facts list. The logic solver is a logicprogramming based module that performs a pragmatic interpretation of the query DRS over the full system knowledge base (the Local KB and the facts list). It tries to find the best explanations for the question logic form to be true. The inference process is done with the Prolog resolution algorithm.
The ad-hoc solver is a case-based answer selector for questions where the answer can be directly
detected in the text. It uses a text pattern and structure based approach, explored before in [
A concrete example for this situation is the question:
O que ´e uma c´ıtara ? (What is a c´ıtara ?)
A ’c´ıtara’ ´e um instrumento musical de v´arias cordas presas sobre um arco de madeira, com ou sem caixa de ressonˆancia, que se tocavam com ambas as m˜aos.
This direct match gave one candidate answer, defining a c´ıtara as a musical instrument having strings attached to a wood arc.
Each candidate answer has a weight and a snippet: sentence or expression justifying the answer and its document identifier. The results are merged, forming a global and weight sorted list that is processed by the answer validator. This new component does an appreciation of each answer value. Some might be refused if they are not in accordance with the question type.
Based on the principle that the Web redundancy can be exploited as a method for answer
validation in question answering [
If there is no candidate answer the system returns NIL as result. When the system finds more than one answer for a question, then the QA@CLEF answer is the one with the highest weight. 4
As before, the University of E´vora’s group QA@CLEF participation was focused on the monolingual Portuguese task. The set of 200 questions was processed by the Senso QA system and one run output was sent for evaluation. The system answers were classified as Right for 93 questions, which corresponds to an overall accuracy score of 46.50% (4.5% more than obtained last year).
The system returned only 21 NIL answers, significantly less when comparing to 2007. Despite this appears to be a better value, the accuracy for the NIL question type went down from 10.81% to 9.52%. In the Factoids category the system had an accuracy of 40.74%, quite similar to last year. The best relative accuracy result was again on the Definition question type with 85.71%, which represents an improvement since 2007.
Table 1 shows the system detailed results, specifying the accuracy values per question type.
Right
Wrong
Unsupported
Inexact
Accuracy Question Type
Nil Temporally Restricted
Definition
Lists
Factoids All Questions
#
The overall Confidence Weighted Score over all assessed questions is 47.959/200 = 0.23979. The overall accuracy is slightly better than the obtained last year. The next section has some conclusions about this participation. 5
The University of E´vora’s participation on QA@CLEF 2008 was based on their Senso Question Answering System. Being the second time this QA system is used, the results are in line with the expected: close to or a little better than last year.
The document retrieval process was updated. One of the changes was in the Portuguese stemmer used in the Lucene indexing and search operations. This and the text search query generation process led to the identification of more candidate documents, decreasing the number of NIL answers.
The logic solver has a few problems with DRS generation, specially when analyzing morphosyntactical representation of non-trivial sentences. Other problems happen in the pragmatic interpretation of the DRS. Most of the answers found by the system are from the ad-hoc solver.
The answer validation process gave a relevant contribution to accuracy. The redundancy of Web information allows the usage of a search result as a clue for the validity of an answer.
The ad-hoc solver is a rule based answer generator. Some of those rules need an adjustment. Before applying our system to other source languages, there is some work to do in order to make the system components independent from the language. The rules in ad-hoc solver are language dependent. One possibility for a future participation is the submission of multiple answers per question. That can be accomplished with this system by selecting the N most weighted answers.