This report describes the work done by the RFIA group at the Departamento de Sistemas Inform´aticos y Computaci´on of the Universidad Polit´ecnica of Valencia for the 2006 edition of the CLEF Question Answering task. We participated in three monolingual tasks: Spanish, Italian and French. The system used is a slightly revised version of the one we developed for the past year. The most interesting aspect of the work is the comparison between a Passage Retrieval engine (JIRS) specifically aimed to the Question Answering task and a standard, general use search engine such as Lucene. Results show that JIRS is able to return high quality passages.
introduction of another kind of “definitions”, that we named object definitions. This forced us to change slightly the class ontology we used in 2005. This year we did not participate to the cross-language tasks.
In the next section, we describe the structure and the building blocks of our QA system. In section 3 we discuss the results of QUASAR in the 2006 CLEF QA task. 2
The architecture of QUASAR is shown in Fig.1.
Given a user question, this will be handed over to the Question Analysis module, which is composed by a Question Analyzer that extracts some constraints to be used in the answer extraction phase, and by a Question Classifier that determines the class of the input question. At the same time, the question is passed to the Passage Retrieval module, which generates the passages used by the Answer Extraction (AE) module together with the information collected in the question analysis phase in order to extract the final answer. 2.1
This module obtains both the expected answer type (or class) and some constraints from the
question. Question classification is a crucial step of the processing since the Answer Extraction
module uses a different strategy depending on the expected answer type; as reported by Moldovan
et al. [
The different answer types that can be treated by our system are shown in Table 1. We introduced the “FIRSTNAME” subcategory for “NAME” type questions, because we defined a pattern for this kind of questions in the AE module. We also specialized the “DEFINITION” class into three subcategories: “PERSON”, “ORGANIZATION” and “OBJECT”, which was introduced this year (e.g.: What is a router? ). With respect to CLEF 2005, the Question Classifier does not use a SVM classifier.
L0 NAME
Each category is defined by one or more patterns written as regular expressions. The questions that do not match any defined pattern are labeled with OTHER. If a question matches more than one pattern, it is assigned the label of the longest matching pattern (i.e., we consider longest patterns to be less generic than shorter ones).
The Question Analyzer has the purpose of identifying the constraints to be used in the AE phase. These constraints are made by sequences of words extracted from the POS-tagged query by means of POS patterns and rules. For instance, any sequence of nouns (such as ozone hole) is considered as a relevant pattern. The POS-taggers used were the SVMtool2 for English and Spanish, and the TreeTagger3 for Italian and French.
There are two classes of constraints: a target constraint, which is the word of the question that should appear closest to the answer string in a passage, and zero or more contextual constraints, keeping the information that has to be included in the retrieved passage in order to have a chance of success in extracting the correct answer. For example, in the following question: “D´onde se celebraron los Juegos Ol´ımpicos de Invierno de 1994? ” (Where did the Winter Olympic games of 1994 take place? ) celebraron is the target constraint, while Juegos Ol´ımpicos de Invierno and 1994 are the contextual constraints. There is always only one target constraint for each question, but the number of contextual constraint is not fixed. For instance, in “Qui´en es Neil Armstrong? ” the target constraint is Neil Armstrong but there are no contextual constraints.
We did not implement 2http://www.lsi.upc.edu/ nlp/SVMTool/ 3http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/DecisionTreeTagger.html Passages with the relevant terms (i.e., without stopwords) are found by the Search Engine using the classical IR system. This year, the module was modified in order to rank better the passages which contain an answer pattern matching the question type. Therefore, this module is not as language-independent as in 2005 because it uses informations from the Question Classifier and the patterns used in the Answer Extraction phase.
Sets of unigrams, bigrams, ..., n-grams are extracted from the extended passages and from the user question. In both cases, n will be the number of question terms. With the n-gram sets of the passages and the user question we will make a comparison in order to obtain the weight of each passage. The weight of a passage will be heavier if the passage contains greater n-gram structures of the question.
For instance, if we ask “Who is the President of Mexico? ” the system could retrieve two passages: one with the expression “...Vicente Fox is the President of Mexico...”, and the other one with the expression “...Giorgio Napolitano is the President of Italy...”. Of course, the first passage must have more importance because it contains the 5-gram “is the President of Mexico”, whereas the second passage only contains the 4-gram “is the President of ”, since the “is the President of Italy ” 5-gram is not in the original question. To calculate the weight of n-grams of every passage, first the greatest relevance of n-gram in the passage is identify and we assign to this a weight equal to the sum of all term weights. Next, other n-grams less relevant are searched. These n-grams are not composed by terms of found n-grams. The weight of these n-grams will be the sum of all their weight terms divided by two, in order to avoid that its weight will be the same of the complete n-gram. The weight of every term comes fixed by (refeqtermweight): log(nk) wk = 1 − 1 + log(N ) . (1)
Where nk is the number of passages in which the associated term to the weight wk appears and N is the number of system passages. We make the assumption that stopwords occur in every passage (i.e., nk takes the value of N ). For instance, if the term appears once in the passage collection, its weight will be equal to 1 (the greatest weight). Whereas if it is a stopword its weight will be the lowest.
Depending on the style used to submit a question, sometimes a term unrelated to the question can obtain a greater weight than those assigned to the Named Entities (NEs), such as names of persons, organizations and places, dates. The NEs are the most important terms of the question and it does not make sense to return passages which do not contain them. Therefore, the equation (1) has been changed in order to give more weight to the NE than the rest of question terms and so to force its presence in the first passages of the ranking. In order to identify a NE a natural language processing is not used. We assumed that in most questions the NEs start with either an uppercase letter or they are a number. Once the terms are weighted, these are normalized for the sum of all terms are equal to 1.
JIRS can be obtained at the following URL: http://leto.dsic.upv.es:8080/jirs. 2.3
The input of this module is constituted by the n passages returned by the PR module and the constraints (including the expected type of the answer) obtained through the Question Analysis module. A TextCrawler is instantiated for each of the n passages with a set of patterns for the expected type of the answer and a pre-processed version of the passage text. For CLEF 2006, we corrected some errors in the patterns and we also introduced new ones.
Some patterns can be used for all languages; for instance, when looking for proper names, the pattern is the same for all languages. The pre-processing of passage text consists in separating all the punctuation characters from the words and in stripping off the annotations of the passage. It is important to keep the punctuation symbols because we observed that they usually offer important clues for the individuation of the answer (this is true especially for definition questions): for instance, it is more frequent to observe a passage containing “The president of Italy, Giorgio Napolitano” than one containing “The president of Italy IS Giorgio Napolitano” ; moreover, movie and book titles are often put between apices.
The positions of the passages in which occur the constraints are marked before passing them to the TextCrawlers. A difference with 2005 is that now we do not use the Levenshtein-based spell-checker to compare strings in this phase now.
The TextCrawler begins its work by searching all the passage’s substrings matching the expected answer pattern. Then a weight is assigned to each found substring s, depending on the positions of s with respect to the constraints, if s does not include any of the constraint words. If in the passage are present both the target constraint and one or more of the contextual constraints, then the product of the weights obtained for every constraint is used; otherwise, it is used only the weight obtained for the constraints found in the passage.
The Filter module takes advantage of a mini knowledge base in order to discard the candidate answers which do not match with an allowed pattern or that do match with a forbidden pattern. For instance, a list of country names in the four languages has been included in the knowledge base in order to filter country names when looking for countries. When the Filter module rejects a candidate, the TextCrawler provide it with the next best-weighted candidate, if there is one.
Finally, when all TextCrawlers end their analysis of the text, the Answer Selection module selects the answer to be returned by the system. The following strategies apply: • Simple voting (SV): The returned answer corresponds to the candidate that occurs most frequently as passage candidate. • Weighted voting (WV): Each vote is multiplied for the weight assigned to the candidate by the TextCrawler and for the passage weight as returned by the PR module. • Maximum weight (MW): The candidate with the highest weight and occurring in the best ranked passage is returned. • Double voting (DV): As simple voting, but taking into account the second best candidates of each passage. • Top (TOP): The candidate elected by the best weighted passage is returned.
This year we used the Confidence Weighted Score (CWS) to select the answer to be returned
to the system, relying on the fact that in 2005 our system was the one returning the best values
for CWS [
The snippet for answer justification is obtained from the portion of text surrounding the first occurrence of the answer string. The snippet size is always 300 characters (150 before and 150 after the answer) + the number of characters of the answer string. 3
We submitted two runs for each of the following monolingual task: Spanish, Italian and French.
The first runs (labelled upv 061 ) use the system with JIRS as PR engine, whereas for the other
runs we used Lucene, adapted to the QA task with the implementation of a weighting scheme that
privileges long passages and is similar to the word-overlap scheme of the MITRE system [
With respect to 2005, the overall accuracy increased by ∼ 3% in Spanish and Italian, and by ∼ 7% in French. We suppose that the improvement in French is due to the fact that the target collection was larger this year. Spanish is still the language in which we obtain the best results, even if we are not sure about the reason: a possibility is that this can be due to the better quality of the POS-tagger used in the analysis phase for the Spanish language.
We obtained an improvement over the 2005 system in factoid questions, but also worse results in definition ones, probably because of the introduction of the object definitions.
The JIRS-based systems performed better than the Lucene-based ones in Spanish and French, whereas in Italian they obtained almost the same results. The difference in the CWS values obtained in both Spanish and French is consistent and weights in favour of JIRS. This prove that the quality of passages returned by JIRS for these two languages is considerably better. 4
We obtained a slight improvement over the results of our 2005 system. This is consistent with the small amount of modifications introduced, principally because of the new rules defined for the 2006 CLEF QA task. The most interesting result is that JIRS demonstrated to be more effective for the QA task than a standard search engine such as Lucene in two languages over three. Our further works on the QUASAR system will concern the implementation of a specialized strategy for definition questions, and probably a major revision of the Answer Extraction module.
We would like to thank R2D2 CICYT (TIC2003-07158-C04-03) and ICT EU-India (ALA/95/23/2003/077054) research projects.