The University of E´ vora team in QA@CLEF-2007 tested their Senso system in the Portuguese monolingual task. The system uses an ontology semantic information for text search terms expansion and for verification of concept equivalency or IsA/specialization relations. The full text collection is indexed and for each question it's done a search, for retrieval of possible relevant documents that may have one answer. The solver module engine starts by producing a First-Order Logic expression representing the question and a logic facts list representing the texts information. There is a logic-programming based module that looks for answers within the facts list that unify and validate the question logic form. For cases where the answer can be directly detected in the text there is an ad-hoc module. Then the logic and ad-hoc found results are merged and the solution with highest weight is selected. We sent one run result for evaluation. The overall accuracy was 42% and the Confidence Weighted Score was 0.19524. This paper has a description of our system.
Experimentation answer in the texts. Besides the usual newspapers collections from Pu´blico and Folha de S˜ao Paulo, the system had to consider also the Portuguese articles from Wikipedia. Other relevant innovation was the existence of clusters or groups of questions about the same implicit topic, with possible anaphoric references between one question and the others. In such case, the system identifies the topic either in the first question or in the first answer, as referred in the guidelines.
This QA system (Senso) is based on the authors previous work [
Senso Question Answer System has five major modules: Libs, Query, Solver, Ontology and Web Interface. Figure 1 represents the way they are connected.
The Libs Module contains collections of text documents. These collections are seen as libraries that contain information needed for question answering. It had the five collections: Pu´blico and Folha de S˜ao Paulo from years 1994 and 1995, plus the Wikipedia documents. All the questions are firstly analyzed by the Query Module. The query group identifier determines if a query will be associated with the first from that group. This module will also select a set of relevant documents for each question, as explained later.
When we have an isolated sentence it’s usually difficult to automatically capture its meaning. The Senso Ontology module has a starting knowledge base with semantic information that helps to perform the sentence analysis and the subsequent inference processes. This information is structured by an OWL2 Ontology including concepts, relations and properties. The OWL language has the intended semantic features and it is suitable for web publications, allowing us to share parts of our knowledge base in a direct and appropriate manner.
Besides concept “IsA” relations, the ontology includes some simple facts about everyday life that might be very useful for text analysis. Our current ontology contains about 3500 concepts and has several relations connecting them: isA, usedFor, locatedAt, capableOf and madeOf. These concepts and relations represent a small common sense knowledge base about places, entities and events. Some of the top-level concepts are shown in figure 2.
The Solver Module performs a search for plausible answers in the identified relevant documents, being aware of the semantic expressed in the ontology. It has a logic-programming based tool and an ad-hoc answer selector.
2OWL[
Next section explains the methodology used to find the answers.
This section explains our approach to CLEF-2007 Question Answer track in detail. 3.1
The starting point is the information source: the document collections. This year we had 210734 newspaper texts and 336622 Wikipedia texts for the Portuguese target language. The XML collection files were processed and split in single texts, along with important metadata. The Libs Module keeps all these individual documents, being aware of their temporal context, which is obtained from the collection.
Because we needed to perform some text search operations, the collections were indexed at this point with Lucene3, a full-featured text search engine library. Lucene scoring uses a combination of the Vector Space Model and the Boolean model to determine how relevant a given document is to a query.
Each text was then processed with Palavras[
Each question is processed with the syntactical parser Palavras[
Let us consider the following definition question, in this year’s edition:
Quem ´e Boaventura Kloppenburg ? 3Apache Lucene is an open source project. http://lucene.apache.org/ 4Tool developed by Eckhard Bick. VISL Project: http://visl.hum.sdu.dk/visl be from zero to several hundreds. This avoids the computational complexity of dealing with more than a half million texts. In the case where no candidate documents are found the system cannot find an answer and the result is NIL.
The Query Module produces the Lucene search query. This is done with the question text terms and, for some, their related terms. So, if a question has something like “Which bird...” the text search query will include synonyms of bird and specialization terms given by the Senso ontology, such as eagle. This semantic operation in the query allows the retrieval of a text that may not have the word bird but is still relevant as a possible answer source. Question 15 asked which tree is present in the Lebanon flag. The answer was cedro (or cedar, in English). Being aware that cedar is a tree was important to the process.
When the question belongs to a cluster and it is not the first from that group the query is fed with more terms, in order to include the implicit topic. The system goes back to that cluster’s first question and gets their search terms and answer into the Lucene query. 3.3
The Solver Module is the core piece in our methodology. It is responsible for finding a list of answers for a query. Each answer has a weight and a support note: sentence or expression justifying the answer and it’s document identifier, as we can see in figure 6 for the question 17: O que ´e um barrete fr´ıgio ?
The search for plausible answers is done on the Lucene selected documents by two tools: the logic solver and the ad-hoc solver.
The semantic analyzer used before for the query will now produce a DRS list for the selected
texts. This list is seen as a small and question dedicated knowledge base: the facts list. The logic
solver is a logic-programming based module that performs a pragmatic interpretation of the query
DRS over the full system knowledge base (the ontology and the facts list). It tries to find the best
explanations for the question logic form to be true. This strategy for interpretation is known as
“interpretation as abduction” [
The inference process is done with the Prolog resolution algorithm, which tries to unify the referents from the query with referents from documents, in the facts list, with help from the semantic information given by the ontology.
The ad-hoc solver is an answer generator for specific cases where the possible solution can be directly detected in the text. The system verifies each case specific conditions for the query and text expressions. When the conditions are verified, that ad-hoc case gives one answer. Verifying the conditions might include a term semantic test for equivalence or “IsA” relation with another term, which is done by ontology analysis.
Figure 7 has a list of answers for question 34:
Qual o di^ametro de Ceres ? This is a Factoid question about a measure. The ad-hoc solver identified the term diˆametro (diameter) and searched for numerical answers, including the unit of measure (km, metros ).
The logic and ad-hoc found results are then merged to a final and weight sorted list. The answer merging process checks for repeated values and joins their support data, assigning the highest weight to that result. When the system finds more than one result for a question the QA@CLEF answer is the one with the maximum weight. 4
In this QA@CLEF’s edition, the Universidade de E´vora’s group registered for the monolingual
Portuguese task, as did in previous participation [
A correct answer was found for 84 questions, which corresponds to an accuracy score of 42%. This value represents a relative increment of 68% from our department last participation global accuracy (25%).
Analyzing the results by question category, we can say that most of the errors were in the 90 wrong NIL returned values, where the system could not find an answer. Then, the List and Temporally Restricted questions represented a challenge and the obtained accuracy for these cases was around 20%. In the Factoids category the system had an accuracy close to the overall value, it was 39.62%. The best relative accuracy result was achieved in the Definition question type: 61.29%. Table 1 shows the accuracy values in more detail.
Right
Wrong
Unsupported
Inexact
Accuracy Question Type
Nil Temporally Restricted
Definition
Lists
Factoids All Questions
#
The overall Confidence Weighted Score over all assessed questions is 39.048/200 or 0.19524. Comparing the current overall accuracy with the obtained in our department previous participation we believe this system produced good results. However, it needs some improvements as explained in the next section. 5
In this paper we describe our Question Answering System for QA@CLEF-2007. Compared with the system we used in 2005, the Senso system has a different methodology and is based in a different ontology. The results obtained are quite satisfactory and better since our last participation.
A preliminary analysis on the incorrect answers showed that some questions had no candidate documents where to search for an answer. This means that the Lucene query used for document retrieval failed in those cases.
Our semantic analyzer also had some problems with DRS generation, while analyzing the morphosyntactical representation of non-trivial sentences. Other problems were related to incorrect pragmatic analysis, in the logic solver, due to ontology limitations and some lack of precision on the semantic information taken from the text sentences.
The Lucene search engine indexes all text collections and gives the system a list of documents that may have an answer and need detailed analysis. This was important to avoid problems with time constraints, because some of the hard work is now done only over the selected documents. However, we need to correct the way the Lucene text search query is built, to fetch the answer candidate documents where it currently cannot do it.
We also intend to improve the Senso ontology. Since many operations in our methodology depend on it’s content, it should be manually revised and extended. Along with this, some disambiguation tool would help for better precision when a sentence concept is being related with an ontology existent term.
In a future QA@CLEF participation we intend to apply our system to other languages besides the Portuguese monolingual task.