The approach followed by the University of Evora team in order to build a system able to participate in the QA-CLEF task is described. The system is based in two steps: for each question, a rst information retrieval task selects a set of potentially relevant documents; then, each of these documents is analysed trying to obtain their semantic representation and the answer to the initial query. The proposed approach was applied to the test set of the QA-CLEF for the Portuguese language and the obtained results were quite interesting and motivating and allowed the identi cation of strong and weak characteristics of the system.
For each question, a documents.
rst information retrieval task selects a set of potentially relevant Then, for each of these documents that where analysed in the preparatory phase trying to extract the facts they conveyed, the user query is interpreted on each selected text knowledge base. When an answer to the query is obtained, the process stops and the system outputs the answer and then identi es the document from where the answer was obtained.
Our question answer system needs to have a preliminary information retrieval task, de ning a smaller set of potentially relevant documents due to computational complexity problems. In fact, our main approach is to deeply analyse the set of documents and to obtain a partial semantic representation of their content. Then, each query would be transformed into its semantic form and an inference process would try to obtain the answer to the query. However, this approach showed many scalability problems due to the large number of documents and associated data and it was necessary to strongly reduce its cardinality.
Section 2 describes the preparatory phase where the set of documents are preprocessed in order to build the IR indexes and the knowledge base for each text.
Section 3 describes the proposed architecture for the question-answer system and section 4 describes each of the architecture modules. A preliminary evaluation is presented in section 5 and some conclusions and future work is discussed in section 6. 2
Pre-processing the set of target documents
There exists an important pre-processing phase of the target collection of documents in order to obtain the input data necessary for our question answer system.
In this phase two main tasks are done:
Semantic/Pragmatic Interpretation { gives rise to a set of knowledge bases, Text facts collection, where each knowledge base has the facts conveyed by each text.
Information retrieval indexing { creates the les that index the full set of documents with a reference to the knowledge base associated to each document, Sino Index Files. The other tasks of this phase are:
Portuguese Parser - each text of the collection is analysed by the Portuguese parser PALAVRAS
[
We've chosen to keep the rst syntactic analysis for each sentence; this option is one of our sources of problems.
Semantic Interpretation { each syntactic structure is rewritten into a First-Order Logic
expression. The technique used for this analysis is based on DRS's (Discourse Representation
Structures)[
This technique identi es triggering syntactic con gurations on the global sentence structure, which activates the rewriting rules. We always rewrite the pp's by the relation 'rel(prep,A,B)' postponing its interpretation to the semantic pragmatic module.
The semantic representation of sentence is a DRS build with two lists, one with the new sentence rewritten and the other with the sentence discourse referents.
One of the most important requirements of the proposed QA system is to have a knowledge base of facts inferred from the analysis of the set of target documents and an ontology containing the concepts referred in the documents.
Ontology { From the output of the DRS's generation and from an existent top ontology
of concepts, a new ontology containing the concepts referred in the documents was created
[
This step showed to be very problematic, due to the large number of concepts referred in the
documents and to the complexity and di culty of nding correct relations between them.
The obtained ontology was created in the OWL (Ontology Web Language) format and in
a logic programming framework, ISCO [
However, this task showed to be very complex in its computational time and space and the obtained knowledge base was very large and it created many problems to the inference processes.
Accordingly, it was decided to rst decrease the set of relevant documents to each query (via IR techniques) and, then, to create a set smaller knowledge base.
The knowledge base referred in gure 1 was build with a set of facts extracted from the target text collection and with rules and facts that we import from other applications. 2.1
In order to obtain the set of facts of each text sentence we need to use the ontology in order to obtain the meaning of each text sentence.
The semantic/pragmatic module receives the sentence rewritten (into a First Order Logic form) and tries to interpret it in the context of the document database information (ontology).
In order to achieve this behaviour the system tries to nd the best explanations for the sentence
logic form to be true in the knowledge base for the semantic/pragmatic interpretation. This
strategy for interpretation is known as \interpretation as abduction" [
The knowledge base for the semantic/pragmatic interpretation is built from the Ontology. The inference in this knowledge base uses abduction, restrictions (GNU Prolog Finite Domain (FD) constraint solver).
The knowledge base rules contains the information for the interpretation of each term in the sentence logic form as a prolog term.
As an example consider the sentence: \O gato do Joa~o comeu o rato do Manuel/John's cat ate Manuel's mouse." is transformed into a DRS-like term showing the 4 referents and their relations: drs([def-A-m-s, def-B-m-s,
def-C-m-s, def-D-m-s], [cat(A), rel(of,A,B), name(B,'Joa~o'), comer(A,C), mouse(C), rel(of,C,D), name(D,'Manuel')]).
The semantic interpretation module using the ontology will rewrite this DRS into: drs([def-A-m-s, def-B-m-s,
def-C-m-s, def-D-m-s], [cat(A), owns(B,A), person(B), name(B,'Joa~o'), eats(A,C), mouse(C), owns(D,C), person(D), name(D,'Manuel')]).
The interpretation of rel(of,A,B) as owns(A,B) is possible due to the existence of the relation owns that relates persons and animals.
Other important step in this task is to create new individuals (new identi ers) for discourse referents when they are not instantiated during the interpretation.
This last step is a source of problems for our QA-system since it is possible to have di erent identi ers for the same individual if this task fails to identify the sentences entities. The opposite can also happen, this task may unify individuals that are di erent.
The option of building a knowledge base with the facts extracted for each documents helps us to deal with this problems, there are fewer entities.
A problem that we still have to solve is the way how we choose the best meaning for a sentence. 2.2
SINO [
Documents are indexed by a specialized search engine for the Portuguese language { SINO
[
The architecture is composed by several independent modules. Figure 2 shows a graphical view of their relations.
In the next sub-sections a brief description of each module is presented. 3.1
Each query is processed using the same natural language tools used to analyse the full set of documents, The Portuguese syntactic Parser Palavras and the DRS's generation. After obtaining the Query DRS there are two task that are performed concurrently:
The Semantic Pragmatic Interpretation of the query. The query semantic representation is obtained taking into account the ontology of concepts and a knowledge base with some general world knowledge.
The query preprocessing and the interrogation of the IR system.
After obtaining the query DRS, it is transformed into a search term to the IR engine { SINO. This step is needed because it was computationally impossible to handle inferences over the complete knowledge base created in the pre-processing phase of the documents. So, we use an information retrieval system to obtain a set of relevant documents to make inferences only over the knowledge base created with the information conveyed by these documents. The IR queries are created from the semantic representation, DRS, of each query and their structure will be described in the next section.
Using the IR queries, the search engine obtains an ordered set of relevant documents. This set is used to create a smaller knowledge base of DRS containing only the information conveyed by these relevant documents.
In this way it is possible to strongly decrease the complexity of the knowledge base and it is possible to handle inferences over it.
Finally the Process answer task receives the set of relevant knowledge bases where the query semantic/pragmatic representation should be evaluated and tries to infer the answer to the query.
The inference process is based in a logic programming framework, ISCO [
In this section a more detailed description of the most relevant modules of the architecture is presented. 4.1
This module tasks: Palavras+Drs generation and the Pragmatic Interpretation follow an approach similar with the semantic-pragmatic interpretation of the documents sentences and it uses the same natural language tools: the PALAVRAS parser, the DRS generation and semantic-pragmatic interpreter.
After the DRS generation we are able to identify the referents which are the focus of each query and the kind of query performed.
For instance, the query: \Quem comeu o rato do Manuel/Who ate Manuel's rat?" is transformed into the DRS-like term: drs([who-A-X-Y, def-B-m-s, def-C-m-s], [eat(A,B), mouse(B), rel(de,B,C), name(C,'Manuel')]).
After obtaining the query DRS, the semantic-pragmatic interpretation using the ontology of concepts created in the pre-processing phase gives rise to the nal query representation: For the above example it will be: drs([who-A-X-Y, def-B-m-s, def-C-m-s], [eat(A,B), mouse(B), owns(C,B),person(C), name(C,'Manuel')]).
This nal query representation will be evaluated in each knowledge base selected by the last sino query. 4.2
After obtaining the query DRS, it is transformed into a search term to the IR engine { SINO.
The approach followed was to create three IR query terms for each natural language query and to order the set of documents retrieved. The overall idea is to create a very restrictive query, a very general one, and one in the middle. The created IR queries are boolean ones and they are obtained from the DRS of each query: ``Em que cidade se encontra a
prisa~o de San Vittore?'' cidade AND encontrar AND prisa~o AND (San AND Vittore) cidade AND (encontrar OR prisa~o OR (San AND Vittore)) cidade OR encontrar OR prisa~o OR (San AND Vittore)
The rst query is the more speci c, obtained from the boolean AND of each term; the second query is obtained from the boolean AND of the head of the query with the OR of the other terms; and the third query is the more general, obtained from the OR of each term.
From the ordered set of documents, the rst 50 are selected and they are the basis for the creation of each query-related knowledge base. 4.3
This module receives the three IR queries and it retrieves the correspondent relevant documents. As it was already described, the IR engine used was an extension of the SINO engine from the AustLII institute.
SINO retrieves the relevant documents (using the boolean operators) and it orders the selection using a ranking function. This ranking function gives higher priority to documents with more word hits or with hits in the title. It is important to point out that rst documents are ordered accordingly with the kind of query: rst the documents retrieved from the more speci c query and with last priority the documents retrieved from the more general one. Inside each set of retrieved documents, the order is obtained through the SINO ranking function.
After having a ordered list of relevant documents, the rst 50 were selected as the basis to create a knowledge base of facts relevant to the query. The reason why the rst 50 were chosen is related with the goal of reducing the computational complexity and assuming a good performance of the SINO engine. 4.4
This module is the responsible for nding the correct, exact answer to each query. It receives the semantic-pragmatic interpretation of each query (in a DRS-like format) and a logic-programming based knowledge base built from the set of the most relevant 50 documents of each query.
The inference process is done via the use of the Prolog resolution algorithm, which tries to unify the referent in the query with facts extracted from the documents. This uni cation takes into account the information associated with the referents, such as, genre and number. Moreover, the inference process uses the "kind of" question, such as, where/when/who, to identify the feature that is queried about. For instance, if the query is about a place of a speci c entity, "Em que cidade se encontra a prisa~o de San Vittore?", the system tries to nd a feature of that entity that is a place and it is not referred in the query.
As it can be seen from this description, the proposed system relies on the quality of the inferred ontology and in a good semantic-pragmatic interpretation of sentences and queries.
As a consequence of our approach, the system has always a con dence value for each answer of 1: if it nds an answer, then it is sure about it! 5
The proposed system was applied to the set of 200 questions (in fact they were 199, because one question was not considered by the judges).
It obtained 47 correct answers, 18 inexact and 134 wrong with an overall accuracy of correct answers of 23.62% and a con dence-weighted score of 0.21619. If the accuracy is calculated over the correct and the inexact answers, then its value is 32.66%.
We believe these are quite interesting results, that show the potential of the proposed approach. However, they also show the main problem of the system: it gave 127 "nil" answers and only 9 of them were correct.
The most important question now is: what happened in the 118 questions that had no answer from the system?
A preliminary evaluation showed that there were two main causes of problems: the information retrieval system the ontology
The information retrieval system, which was used to decrease the complexity of the knowledge base, quite often was not able to nd the relevant documents. In fact, this problem can be clearly seen in the results of the IR task of CLEF'04, in which SINO showed very low recall values. This problem can be overcome by changing the SINO queries to better ones or by solving the complexity problems that made impossible the construction of a large, unique knowledge base. We intend to explore both possibilities.
The second problem was the quality of the ontology. In fact, the inference process relies heavily on the ontology. For instance, it is important to know what are places, persons, dates, synonyms. In the example presented previously, if the ontology does not have information relating "cidade" with the class of "places", then the system would not be able to answer the query. We will also continue to develop new strategies for constructing and merging ontologies.
This proposal represents a rst approach for a question answering system for the Portuguese language.
Our system uses natural language processing techniques to create a knowledge base from the information conveyed by the target documents. Queries are analysed by NLP tools and inferences are done over the knowledge base trying to nd a correct answer. The inference process is done using a logic programming framework and the Prolog resolution.
The initial idea of creating a unique, large knowledge base with the facts extracted from all the documents was not feasible due to computational complexity problems. These problems led to the creation of an IR pre-analysis of the queries to decrease the complexity of the knowledge base. However, the IR engine showed some recall problems and lead to the incapacity of the QA system to answer many queries.
The ontology used was also a major problem and it was the origin of many other wrong answers. As the QA@CLEF task uses general domain documents, this is a very complex problem: how to obtain a good general purpose ontology?
As future work, we intend to try to develop new strategies for (partially) overcome these problems. Working with new implementation strategies it may be possible to have an unique knowledge base and using existent ontologies and Wordnets may improve the quality of the nal ontology.
Finally, we also intend to explore the problem of inter-sentence anaphoric references and to be able to identify the correct referents in the documents.