This paper accounts for Priberam's participation in the monolingual question answering (QA) track of CLEF 2007. In previous participations, Priberam's QA system obtained encouraging results both in monolingual and cross-language tasks. This year we endowed the system with syntactical processing, in order to capture the syntactic structure of the question. The main goal was to obtain a more tuned question categorisation and consequently a more precise answer extraction. Besides this, we provided our system with the ability to handle topic-related questions and to use encyclopaedic sources like Wikipedia. The paper provides a description of the improvements made in the system, followed by the discussion of the results obtained in Portuguese and Spanish monolingual runs. 1 M-CAST - Multilingual Content Aggregation System based on TRUST Search Engine - was an European Commission cofinanced project (EDC 22249 M-CAST), whose aim was the development of a multilingual infrastructure enabling content producers to access, search and integrate the assets of large multilingual text (and multimedia) collections (http://www.mcast.infovide.pl).
Priberam has participated in the CLEF campaigns since 2005, where its QA system was evaluated in both
monolingual [
This year, the CLEF QA track presented two novelties with direct consequences in the evaluation of QA systems. First, the organisation introduced topic-related questions, that is, questions clustered around a common topic that might present anaphoric links between them. Second, it also added the open domain encyclopaedia Wikipedia as a target document collection to the already existent newspaper corpora.
Although its overall architecture was maintained, Priberam's QA system was subjected to several changes, both in the Portuguese and in the Spanish modules, the most relevant one being the introduction of syntactical question processing. These modifications, together with the introduction of the two simultaneous changes by the CLEF organisation, had an impact in the performance of our system, as we will show in the analysis of the results.
This paper is organised as follows: in section 2, we present the major adjustments made to the system, such as the work done in the syntactical processing of the question, and the necessary adaptations to deal with topicrelated questions as well as with the Wikipedia encyclopaedic source; in section 3 we analyse and discuss the results of both monolingual runs; finally in section 4 we present the conclusions and future work. 2 2.1
As reported in [
In the former version of the QA system, the question analysis stage categorised questions according to a
previously defined question typology, by matching a set of question patterns [
In the present version, and taking advantage of the company's linguistic technology developed for FLiP2, QPs were enhanced with syntactical information. Currently, when a question is submitted, its syntactic structure is captured in a parsing stage that also determines the syntactic function of the question pivots. Parsing the question enables determining its main syntactic constituent, which we have called the object, as well as its secondary syntactic constituents, in case they exist3. For instance, in question 8 of the Portuguese test set “De que estado brasileiro foi governador Adhemar de Barros?” [Of which Brazilian state was Adhemar de Barros governor?], the object is Adhemar de Barros and the secondary constituents are governador and estado brasileiro. This information is then used to validate the category assigned to each question and to allow a more exact, syntactically based answer extraction. By differentiating pivots through syntactic tags, we also expect to decrease the amount of errors in the document retrieval stage, especially when it comes to long questions where there are too many pivots. In the answer extraction stage, we enhanced QAPs to include information about each pivot’s syntactic specifications, which the system tries to match with the answer pivots. The syntactic information captured by parsing may also be useful for answer validation, an aspect that we intend to exploit in the future. 2.2
Priberam's QA system was adapted to handle topic-related questions. According to the QA@CLEF 2007 guidelines, these are clusters of questions related to the same topic and possibly containing anaphoric references between one question and the others. Although questions belonging to the same cluster are identified as such, no information is given about the topic, which has to be inferred from the first question/answer pair. A natural way to address this problem would consist of two steps: (i) detecting the topic, which requires deciding which named entity, event, object, or natural phenomena should be extracted from the first question/answer pair, and (ii) detecting the anaphora in each of the following questions and using the detected topic as the co-referent. 2 Ferramentas para a Língua Portuguesa, Priberam’s proofing tools package for Portuguese. FLiP includes a grammar checker, a spell checker, a thesaurus and a hyphenator that enable different proofing levels – word, sentence, paragraph and text – of European and Brazilian Portuguese. An online version is available at http://www.flip.pt/online. 3 This is still an ongoing work to be detailed in future publications.
Since this clearly emulates an interactive task, our approach requires that the system processes one question at a time, which means that the topic detection is performed regardless of any information contained in the subsequent questions of the cluster. This requirement introduces a major problem to one-topic based approaches, i.e. those in which only one topic is considered, because it is often difficult to infer just from the first question which will be the topic of the whole set of questions. For example, consider the question 58 of the Spanish test set, “¿Quién diseñó el procesador Zilog Z80?” [Who designed the Zilog Z80 processor?], whose answer is “Federico Faggin”. Only after parsing the subsequent question, “¿De cuántos bits era este procesador?” [How many bits was this processor?] could the system have inferred that the topic was “Zilog Z80”, and not “Federico Faggin”. To overcome this inherent difficulty, we adapted the system to be able to accept several topics, when more than one is possible. This was done by implementing the following procedure: • • •
The system parses the first question of a cluster and follows the usual procedure to extract an answer (if any is available); The system calls the method GetTopic, which collects the noun phrases (nouns, proper nouns and named entities) of the extracted answer and the noun phrases of the question which were considered object pivots (see section 2.1), merging these two groups into a single list of topic pivots; In subsequent questions of the same cluster, the system first parses the question, obtaining its pivots, and then calls the method SetTopic, which appends to the question pivots the list of topic pivots collected in the previous step.
“¿Quién diseñó el procesador Zilog Z80?” diseñar, procesador, zilog z80 Federico Faggin federico faggin, zilog z80
Pivots after SetTopic bits, procesador + federico faggin, zilog z80
Table 1 – Procedure for topic-related questions.
Notice that our approach excludes anaphoric analysis of the questions. Naturally, the system will sometimes perform poorly as a consequence of an eventual excess of topic pivots. We are considering as future work a more sophisticated approach that involves anaphora resolution and through which we can do some sort of topic disambiguation by choosing only the co-references (i.e. the topics) that best suit the question. In the above example, the expression “este procesador” could only refer back to “Zilog Z80” and not to “Federico Faggin”; hence the topic pivot “Federico Faggin” could have been discarded if an anaphora resolution method had been used to disambiguate the topic. 2.3
Unlike the collections of newspaper articles, the Wikipedia collection has a rich structure (links, categories, disambiguation pages, etc.) that suggests using strategies capable of extracting knowledge from structured data. However, as a first approach, we did not use such sophisticated strategies; instead, we indexed the Wikipedia articles as natural language text and did some minor adaptations.
The indexation module ignores all the metadata included in tables and boxes. We did not index disambiguation pages, discussion pages, or any internal Wikipedia pages whose title starts with “Wikipedia:”. We converted links of the form “[[<Article title>|<Link text>]]” into strings of the form “<Link text> (<Article title>)” (i.e., putting the title of the linked article between parentheses) and indexed it as natural text. This strategy allows answering some short definition questions (e.g. acronyms) by identifying the link text with the article title without making any change in the system modules.
We developed a simple scheme of anaphora resolution for Wikipedia articles. Since many anaphoric references in these articles have the article title as their referent, we adopted the following procedure: every time we parse a sentence during the sentence retrieval stage and find a null subject or a personal pronoun subject, we replace it by the article title and parse the sentence again (of course, this approach excludes other possible referents besides that expressed in the article title, which is a limitation). Consider the question 136 of the Spanish test set “¿Cuánto mide de alto la Pirámide del Sol de Teotihuacan?” [How high is the Pyramid of the Sun at Teotihuacan?]. The following sentence appears in the article titled “Pirámide del Sol (Teotihuacan)” and the parser detects that it has a null subject: “Tiene 65 m de altura.”. So a new sentence is composed, “Pirámide del Sol (Teotihuacan) tiene 65 m de altura”, and the answer 65 m is successfully extracted. Unfortunately, as we will see in section 3, some QA@CLEF evaluators considered these answers unsupported (even though the article title is provided) and this had a significant impact on the evaluation of our system in the Spanish run. 2.4
Apart from adapting the system to the new requirements of the 2007 QA@CLEF edition, we did some
internal changes in the processing modules, specifically with respect to the way QPs, APs and QAPs are parsed
(see [
Finally, the Spanish language resources were enhanced with the improvement of the lexicon – by means of the introduction of many new words, inflections and derivations –, the recognition of named entities and the inclusion of a Spanish thesaurus. 3
In the tables below, the sets of questions were classified according to three question categories: factoid (FACT), definition (DEF) and list (LIST), with the judgments used for evaluation (R=Right, W=Wrong, X=Inexact, U=Unsupported), as defined in the CLEF 2007 guidelines.
Whereas for Portuguese (PT) the results shown in Table 2 are those provided by the QA@CLEF evaluators4, for Spanish (ES) we present instead our internal evaluation. This is justified by the fact that we strongly disagree with the Spanish QA@CLEF assessors that have considered 20 answers as unsupported (U) instead of right (R), having a negative impact of 10% in the accuracy of our system. Most of these answers are directly related to the procedure described in the previous section 2.3; in other languages (e.g. Portuguese) the assessors considered such answers as R in similar situations. It seems to us that such a discrepancy between our judgement and that of the Spanish assessors is relevant enough to be discussed in the present paper. We illustrate with two examples: (i) the question 136 referred in section 2.3, “¿Cuánto mide de alto la Pirámide del Sol de Teotihuacan?” [How high is the Pyramid of the Sun at Teotihuacan?], for which our system extracted the answer 65 m from the Wikipedia article “Pirámide del Sol (Teotihuacan)” and provided the snippet “Tiene 65 m de altura”, and (ii) the question 179 “¿Cuántos récords mundiales batió?” [How many world records did he achieve?], with the topic Johnny Weissmüller, for which our system answered 67 and provided the snippet “Ganó 52 campeonatos naciones [sic] de Estados Unidos y estableció un total de 67 récords mundiales”, from the Wikipedia article “Johnny Weissmüller”. Both answers were considered unsupported by the Spanish 4 The results for Portuguese include 5 X answers that the Portuguese CLEF assessors marked with M, an unknown classification according to the CLEF guidelines, and the total of answers accounts for the 200 we provided instead of the 195 accounted in the CLEF organisation statistics. assessors, but there seems to be no clear support of that in the QA@CLEF guidelines which are open to different interpretations, by stating that: “Each exact answer must be supported by: • the DOCID of the document in the news collection or by the filename of the dumped Wikipedia (November 2006) page, from which it has been retrieved; • portion(s) of text, which provide enough context to support the correctness of the exact answer.
Supporting texts may be taken from different sections of the relevant documents, and which must sum up to a maximum of 700 bytes. Unnecessarily long snippets, i.e. those that do not meet this requirement, might be judged as non-supporting.” Under our criteria, the Spanish results considered in Table 2 are significantly higher than CLEF's official evaluation, which has determined an overall accuracy of 44.5%.
A
R
W
X
U PT 63 25 4 92
The general results of Table 2 show that there is a significant decrease of overall accuracy in PT and an increase in ES when comparing with the performance of the system in the last CLEF campaign. In order to compare the current results with the results of previous editions of QA@CLEF, we analysed separately the accuracy of non-topic-related questions, taking into account just those questions that could be directly answered without the need to analyse relations between previous questions or answers.
An analysis of the question clusters in the Portuguese and Spanish sets brings some noticeable differences within their number and size. While Portuguese had 25 clusters including 75 questions, Spanish had 20 clusters including 50 questions. In addition to this, while the majority of Spanish clusters had two questions (13 twoquestion, 4 three-question and 3 four-question clusters), the Portuguese set had as many four-question as twoquestion clusters (10 two-question, 5 three-question and 10 four-question clusters). These figures show a higher complexity of the Portuguese set of questions, which could have led to its lower results. Beside this, the Portuguese set also presented a few elliptic questions, such as questions 33, “Onde?” [Where?], 68, “E Régulo?” [And Regulus?], 104, “Por quem?” [By whom?] and 184, “Quando?” [When?].
The inclusion of Wikipedia articles in this year’s target collections increased the level of difficulty, since previously only newspaper corpora were used. The huge volume of information from the combined corpora (news collection and Wikipedia) was in some cases a handicap to reach the best answers. Syntactic processing helped us, though, to better tune the categorisation of the questions and to structure the information provided by the question pivots.
With regard to non-topic-related questions, the system responded quite satisfactory, both in PT and in ES runs. Besides that, general improvements in the Spanish modules raised the accuracy of the ES run. On the other hand, the accuracy of topic-related questions is by far lower in both languages. Once more, the above mentioned differences between both language question sets can be seen as a significant reason for the stronger impact on Portuguese results.
In the ES run, the reasons for wrong answers do not represent a big change in comparison with last year’s
percentages. As in CLEF 2006 results [
Some of the wrong answers classified as Other are due to dubious clusters of questions that seem not to respect the guidelines, since topics do not always come from the first question/answer pair, as illustrated in Table 4. Notice the change of topic from “Bill Gates” (question ids 185 and 186 of the ES set) to “Universidad de Harvard” (question ids 187 and 188).
There are some examples too, in both languages, of answers that apparently can only be extracted from tables or boxes, such as the coordinates of Guarda for the PT question 87 “Qual é a latitude e longitude da Guarda?” [What are the latitude and longitude of Guarda?] or the distribution company for the ES question 172 “¿Cuál es la distribuidora de la película El Planeta de los Simios que se estrenó en 1968?” [Who was the distributor for the 1968 movie Planet of the Apes?]. In some cases, the documents were retrieved, but the answer was not extracted.
To sum up, in the analysis of this year's results, we find it somewhat complicated to evaluate the introduction of two simultaneous major changes in the same CLEF campaign: is the problem in the way the system handles the test set or in the way the system reads the text collection? Gradual changes are easier to assess and are more helpful for the participants, allowing them to correct errors and try different approaches, hence contributing to the development of better and more robust systems. Besides, any comparative analysis of the system’s performance in multiple languages is made impossible due to the multiple criteria of the different assessors while interpreting the guidelines. 4
Due to the changes that occurred in this year’s QA@CLEF task, we cannot directly compare the accuracy of the Priberam’s QA system with previous CLEF campaigns. The system was adapted to handle the Wikipedia collection and to answer topic-related questions recurring to simple approaches given the short amount of time available to prepare our participation; we believe that more sophisticated strategies would lead to a much better performance, but would also require extra time.
We can say, however, that Priberam’s system achieved a more accurate question categorisation, hence decreasing the number of wrong candidate answers, and this was due to the introduction of syntactical parsing during question processing. We expect to further improve the syntactical analysis and to extend it to the answer extraction module. The answer syntactic analysis will allow the system to more precisely match the pivots of the question with their counterparts in the answer, taking into account their syntactic functions. A further development of anaphora resolution will also be one of our goals in the future. We expect to broaden the approach applied this year in Wikipedia article titles, by using our syntactic parsing engine to deal with coreferences also in text sentences.
Finally, we intend to evaluate again the cross-language performance of our system, as this was left out in this campaign. In particular, it would be interesting to evaluate how the system performs with topic-related questions in a cross-language environment.
Priberam Informática would like to thank Synapse Développement, TiP, University of Economics of Prague (UEP), as well as the CLEF organisation and Linguateca. We would also like to acknowledge the support of the European Commission in the M-CAST (EDC 22249 M-CAST) project.