Esfinge is a general domain Portuguese question answering system. It tries to apply simple techniques to large amounts of text. Esfinge participated last year in the monolingual QA track, but the results were compromised by several basic errors. This year, participation was intended to correct the basic errors of last year and work for the first time in the multilingual QA track. The sphinx in the Egyptian/Greek mythology was a demon of destruction that sat outside Thebes and asked riddles to all passers-by. She strangled all the people unable to answer [1], but the times have changed and now Esfinge has to answer questions herself. Fortunately, CLEF's organization is much more benevolent when analysing the results of the QA task. performance Esfinge (http://acdc.linguateca.pt/Esfinge/) is a question answering system developed for the Portuguese which is based on the architecture proposed by Eric Brill [2]. Brill suggests that it is possible to get state of the art results, applying simple techniques to large quantities of data. Esfinge starts by converting a question into patterns of plausible answers. These patterns are queried in several text collections (CLEF text collections and the Web) to obtain snippets of text where the answers are likely to be found. Then, the system harvests these snippets for word N-grams. The N-grams will be later ranked according to their frequency, length and the patterns used to recover the snippets where the N-grams were found (these patterns are scored a priori). Several simple techniques are used to discard or enhance the score of each of the Ngrams. Finally the answer will be the top ranked N-gram or NIL if neither of the N-grams passes all the filters. During last year participation, several problems compromised the results. The main objectives for this year were to correct these problems, and to participate in the multilingual tasks. This year, in addition to the European Portuguese text collection (Público), the organization also provided a Brazilian Portuguese collection (Folha). This new collection helped Esfinge, since one of the problems encountered last year was precisely that the document collection only had texts written in the European variant and some of the answers discovered by the system were in the Brazilian variant, therefore difficult to justify [3].
2.1
IMS Corpus Workbench [
Onde ([^\s?]*) ([^?]*)\??/"$2 $1"/20 It means that for a question including the word Onde (Where), followed by some words, a possible pattern for an answer will be the words following the one immediately after Onde, followed by the word after Onde in a phrase pattern.
As an example, take the question Onde fica Lillehammer? (Where is Lillehammer located?). This generates the pattern Lillehamer fica with a score of 20, that can be used to search for documents containing an answer to the question.
Passage extraction. The patterns obtained in the previous module are submitted to Google. Then, the system extracts the document snippets {S1, S2 … Sn} from Google’s results pages.
It was detected in the experiments made with the system that certain types of sites may compromise the
quality of the returned answers. To overcome this problem it was created a list of address patterns which are not to be
considered (the system does not consider documents stored in addresses that match these patterns). This list
includes patterns such as blog, humor, piadas (jokes). These patterns were created manually, but in the future it
may be rewarding to use more complex techniques to classify web pages [
Another improvement over last year experiment was that if no documents are recovered from the Web, the
system tries to recover them from CLEF’s document collection. When searching in the document collection, the
stop-words without context are discarded. For example in the query “o” “ditador” “cubano” “antes” “da”
“revolução” (the Cuban dictator before the revolution), the words o and da are discarded while in the query “o
ditador cubano antes da revolução” (phrase pattern) they are not discarded. Last year the 22 most frequent
words in the CETEMPúblico corpus [
When the system neither recovers documents from the Web, nor from CLEF’s document collection, one last
try is made by stemming some words in the search patterns. The system uses the morphological analyser jspell
[
If documents are retrieved using any of the previous techniques, at the end of this stage the system has a set of
document passages {P1, P2 … Pn} hopefully containing answers to the question. If no documents are retrieved,
the system stops here and returns the answer NIL (no answer found).
N-grams harvesting. The distribution of word N-grams (from length 1 to length 3) of the first 100 document
excerpts recovered on the previous module is computed. The system uses the Ngram Statistics Package (NSP)
[
Then, the word N-grams are ordered using the following formula:
N-gram score = ∑ (F * S * L), through the first 100 snippets resulting from the web search where: F = N-gram frequency S = Score of the search pattern which recovered the document L = N-gram length
At the end of this stage, the system has an ordered set of possible answers {A1, A2 … An}. Named entity recognition/classification in the N-grams. This module was developed for this year participation, hoping that the use of a named entity recognition (NER) system could improve the results (at least for some types of questions).
An extra motivation for using a NER system was the HAREM (Evaluation Contest of Named Entity
Recognition Systems for Portuguese) [
SIEMES detects and classifies named entities in a wide range of categories. Esfinge used a sub-set of these categories: Human, Country, Settlement (includes cities, villages, etc), Geographical Locations (locations with no political entailment, like for example Africa), Date and Quantity.
Esfinge uses a pattern file that associates patterns of questions with the type of expected result. The following pattern is included in that file:
Quant(o|a)s.*/VALOR TIPO="QUANTIDADE
This pattern means that a question starting with Quantos (how many – masculine form) or Quantas (how many – feminine form) should have a QUANTIDADE (quantity) type answer.
What the system does in this module is to check whether the question matches with any of the patterns in the “question pattern”/”answer type” file. If it does, the 200 best scored word N-grams are submitted to SIEMES. Then the results returned by SIEMES are analysed to check whether the NER system recognizes named entities classified as one of the desired types. If such named entities are recognized, their ranking in the list of possible answers will be enhanced.
The NER system is used in the “Who” questions in a slightly different way. First it is used to check whether there is a person in the question and if that happens, the NER system is not invoked on the candidate answers (example: Who is Fidel Ramos?). There are some exceptions to this rule however and some special patterns to deal with them too (example: Who is John Lennon's widow?). When there is not a person in the question, the NER system is always invoked to find instances of persons for the Who questions. N-gram filtering. In this module the list of possible answers is submitted to a set of filters (by ranking order), namely: • A filter that discards words contained in the questions. Ex: the answer Satriani is not desired for the question
Quem é Joe Satriani? (Who is Joe Satriani?) and should be discarded.
• A filter that discards answers contained in a list of ‘undesired answers’. This list was built with the help of
Esfinge’s log file. The frequency list of all the solutions provided by Esfinge to the 2004 CLEF QA track
questions was computed (not only the best answer, but all the answers that managed to go through all system’s
filters). With this frequency list and some common sense, the list of ‘undesired answers’ was built. The words in
this list are frequent words that do not really answer questions in isolation (like pessoas/persons, nova/new,
lugar/place, grandes/big, exemplo/example). Later some other answers were added to this list, as a result of
the tests performed with the system. The list includes now 92 entries.
• A filter that uses the morphologic analyser jspell [
This filter is only used if the system could not predict the type of answer for the question (using the “question pattern”/”answer type” file) or if SIEMES was not able to find any answer of the desired type. Jspell returns a set of possible PoS tags for each token. Esfinge considers some PoS as “interesting”: adjectives (adj), common nouns (nc), numbers (card) and proper nouns (np). All answers whose first and final token are not classified as one of these “interesting” PoS are discarded. • A filter that checks whether the system can find a document supporting the answer in the collection. This filter is only used if the system retrieved documents from the Web. When the system cannot retrieve documents from the Web, it retrieves them from CLEF’s document collection, and since the N-grams are extracted from these documents there is no need for this filter. It searches the document collection for documents containing both the candidate answer and a pattern obtained from the question reformulation module.
N-gram composition. The motivation to use this very simple module arose from the analysis of last year’s results and some additional tests performed in the system. Sometimes the answers returned by the system were fragments of the right answers. To minimize this problem, a very simple composition algorithm was implemented this year. When an answer passes all the filters in the previous module, the system does not return that answer immediately and stops like in last year. Instead it checks whether there are more candidate answers containing the answer which was found. Each of these candidate answers are submitted to the filters described in the previous module and if one of them succeeds to pass all the filters, this candidate answer becomes the new answer to be returned as result.
Final answer. The final answer is the candidate answer with the highest score in the set of candidate answers which are not discarded by any of the filters described above. If all the answers are discarded by the filters, then the final answer is NIL (meaning that the system is not able to find an answer in the document collection). Run 2 The difference in this run was that the Web was not used as a resource. The answers were only searched in CLEF’s document collection. Consequently, another difference to the algorithm used for the first run was that it was not necessary to check whether there was a document in the collection supporting the answers found since the document collection was the only source used to find them. 2.3
In this experiment the questions were translated using the module Lingua::PT::Translate freely available at CPAN. This module provides an easy interface to Altavista's Babelfish translating tool.
After the translation this experiment followed the algorithm described for the PT-PT monolingual task in run 1 (the run which seemed to have the best results). The results sent to the organization are presented and discussed in this section together with the error analysis performed for one of the runs and some considerations about CLEF 2005 set of questions motivated by this error analysis. To create the tables, the question set was divided in categories that intended to check how well the various strategies used by Esfinge perform. For example the category “People” includes all questions where the system expects to have the name of a person as answer such as the “Who” questions, in which the NER system is invoked to find names of persons in the recovered documents. This assumption is usually correct, but there are some exceptions however. An example is the question Who was Barings taken over by? in which the answer is not a person, but a bank. Other interesting categories are “Places”, “Quantities” and “Dates” where the NER system is also used to find instances of those categories in the recovered texts. The category “What is the name of X” does not include some matching patterns, in which it is easy to infer that the answer will be of type person. What is Nick Leeson's wife's name? is a good example. The same applies for the categories “Name X” and “Which X” in which some questions are not included and are instead placed in another category because the type of answer is easy to infer. Examples of these kinds of questions are Name a city with 650,000 inhabitants and Which country is Alexandria in?
“(Que|Qual) X” - “Which X” Place “Quem (é|foi|era) <HUM>” - “Who (is|was) <HUM>” Quantity Date “Que é X” – “What is X” Como se chama|chamou|chamava X – What is X called Mencione/Indique/Nomeie X - Name X
Total
“(Que|Qual) X” - “Which X” Place “Quem (é|foi|era) <HUM>” - “Who (is|was) <HUM>” Quantity Date “Que é X” – “What is X” Como se chama|chamou|chamava X – What is X called Mencione/Indique/Nomeie X - Name X Totals 47 36
From table 1 it is possible to conclude that the runs submitted for the Portuguese source/Portuguese target task obtain similar results. The run that used the Web (Run 1) got slightly better results, as last year. One can also see that the results in Run 1 are more homogenous than the ones in the second run. Some results are consistently bad, like definitions not involving people (What is X) and not obvious naming (Name X), but that is not surprising since Esfinge does not have special features to deal with definitions. The results of the second run for the questions of type “People” and “Date” are better both comparing to the other types of questions and to the same type of questions in the first run.
Comparing with last year’s results (right columns in the table), one can see that the results improved consistently in almost all types of questions.
200 26 1 The official result is 46 right answers, but during the evaluation of the results I found two more right answers. The system’s log file was used to investigate the causes for the wrong answers. The system registers in this file all the analysed word N-grams for each of the questions. When word N-grams are rejected by some of the filters, this information is also recorded in the log file.
In Table 3, a detailed error analysis for the first run is provided. For some of the questions, it was possible to detect more than one reason for failure. In these cases, both reasons were counted.
From this evaluation, it is possible to create sets of questions with the same type of problems that can be used to debug and improve the system. 3.2
“(Que|Qual) X” - “Which X” Places “Quem (é|foi|era) <HUM>” - “Who (is|was) <HUM>” Quantities Dates “Que é X” – “What is X” Como se chama|chamou|chamava X – What is X called Mencione/Indique/Nomeie X - Name X Totais 36 33 27 18 15 15 5 4 200 From the results in Table 4 it is possible to conclude that most of the questions with right answers are the ones where the NER system was not used (14 out of 25). However, an error analysis similar to the one performed for the PT-PT task will be needed to take more solid conclusions. The error analysis is not only useful to find the reasons motivating system errors. Here and there one is confronted with some interesting cases. I will describe two of them.
The question Who is Josef Paul Kleihues? doesn’t have an answer in the document collection according to the organization, but is this really true? There is a document with the following text (freely translated from the Portuguese original):
People from Galicia like good architecture. In Santiago de Compostela, besides the “Centro Galego de Arte Contemporânea” designed by Siza Vieira, it was built in the historical center a gym designed by the german Josef Paul Kleihues.
One of Esfinge’s runs returned the answer Arquitectura (architecture) giving as support the text from where the previous excerpt was extracted. One may question which answer would be more useful for a hypothetical user.: NIL or the answer provided by Esfinge? I found another curious example in the question Which was the largest Italian party?. On one of the runs Esfinge returned the answer Força Itália supporting it with a document stating that Força Itália is the largest Italian party (it was true at the time the document was written). The organization considered this answer wrong, however, because they wanted an Italian party that was the largest in the past, but was no longer the largest.
In my opinion the answer provided by the system was acceptable, because the question is being asked in 2005, so one can ask which was the largest Italian party, and one can support an answer with a document from 1994 saying that the largest Italian party is X.
Although I can understand the point of view of the organization, I think that this kind of question is confusing and polemic even for humans, therefore not particularly useful to evaluate Q&A systems. 4
The error analysis (condensed on table 3) provided an insight on the problems affecting the system’s performance.
Some effort was invested in the problems that seemed easier to solve. Namely on the “Error in tokenization”, “Problems with the NER system” and “Missing patterns in the file question pattern/answer type”. The results of the system after this improvement using the same strategy as in Run 1 are presented in table 5. On that table it is also possible to check how each part of the system helps global performance: the results obtained either without using the NER system or without using the morphological analyser are presented. One can see that (in different types of questions) both this components are helping the system.
“(Que|Qual) X” - “Which X” Places “Quem (é|foi|era) <HUM>” - “Who (is|was) <HUM>” Quantities Dates “Que é” – “What is” Como se chama|chamou|chamava 25% 31% 48 24% Applying the system to the 2004 questions after the improvements and using the same strategy as in Run 1 provides the results presented in table 6. The cause for the better results this year could be the possibility that this year’s questions were easier than last year’s, but this table shows that the system performs better with last year’s questions as well.
“(Que|Qual) X” - “Which X” Place “Quem (é|foi|era) <HUM>” - “Who (is|was) <HUM>” Quantity Date “Que é X” – “What is X” Mencione/Indique/Nomeie X - Name X Total 43 42 41 17 23 15 15 3 199 # questions 2004
# Esfinge’s best 2004 8
% Esfinge’s best 2004 19%
# Run 4
%
Run 4 7 7 2 4 0 1 1 30 17% 17% 12% 17% 0% 7% 33% 15% 15 9 17 1 4 4 4 1 55 35% 21% 41% 6% 17% 27% 27% 33% 28% The results show that Esfinge improved comparing to last year: the results are better both with this year’s and last year’s questions. Another conclusion is that the two tested strategies perform better with different types of questions, which suggests that both are still worthwhile to experiment and study further.
The experiments performed to check how each part of the system helps global performance shown that (in different types of questions) both the NER system and the morphological analyser improve the system's performance. 6
I thank Diana Santos for reviewing previous versions of this paper, Alberto Simões for the hints on using the
Perl Modules “jspell” , “Lingua::PT::PLNbase” and Lingua::PT::Translate, Luís Sarmento, Luís Cabral and Ana
Sofia Pinto for supporting the use of the NER system SIEMES and Paul Rayson for supporting the use of
CLAWS Web Tagger [
This work is financed by the Portuguese Fundação para a Ciência e Tecnologia through grant POSI/PLP/43931/2001, co-financed by POSI.