This paper describes the prototype developed by the Language Technologies Laboratory at INAOE for Spanish monolingual QA evaluation task at CLEF 2005. Our approach is centered in the use of lexical features in order to identify possible answers to factual questions. Such method is supported by an alternative one based on pattern recognition in order to identify candidate answers to definition questions. The methods applied at different stages of the system and prototype architecture for question answering are described. The paper shows and discusses the results achieved with this approach.
QA systems traditionally perform a question processing stage in order to know in advance the type (semantic
class) of the answer expected by a given question and thus, reduce the searching space to only those information
fragments related to the semantic class found. Our prototype implements this stage following a direct approach
involving the next steps:
1. Question is parsed with a set of heuristic rules in order to get its semantic class.
2. Question is tagged with the MACO POS tagger [
The first step is responsible of identify the semantic class of the expected answer. In the experiments performed with the training data set, we found that when the number of classes was minimal (just 3 classes: date, quantity and proper noun) it was possible to achieve similar results in precision to those achieved when we use more than five classes, for instance person, organization, location, date, quantity and other. Steps 2 and 3 produce information that is used later in searching stage to match questions and candidate answer context, contributing to the weighted schema. 4
This year we experiment with a hybrid1 approach for document processing that has allowed simplifying greatly
this stage. The processing of target documents is composed of two parts, first the whole document collection is
tagged with MACO[
Due to the document processing stage, searching stage is also performed in two steps. As we mention, the first step is to retrieve the relevant passages for the given question. This step is performed by JIRS, taking as input the question without previous processing.
JIRS is a PSR specially suited for question answering systems. JIRS ranks the retrieved passages based on the computation of a weight for each passage. The weight of a passage is related to the lager n-gram structure of the question that can be found in the passage itself. The larger the n-gram structure, the greater the weight of the passage. The next example illustrates this concept.
Assume that the user question is “Who is the president of Mexico?” and that two passages were obtained with the following texts: “Vicente Fox is the president of Mexico…” (p1) and “The president of Spain visited Mexico in last February…” (p2).
The original question is divided into five sets of n-grams (5 is the number of question terms without the question word Who) the following sets are gathered: 5-gram: ''is the President of Mexico''. 4-gram: ''is the President of'', ''the President of Mexico''. 3-gram: ''is the President'', ''the President of'', ''President of Mexico''. 2-gram: ''is the'', ''the President'', ''President of'', ''of Mexico''.
1-gram: ''is'', ''the'', ''President'', ''of'', ''Mexico''.
Next, the five sets of n-grams from the two passages are gathered. The passage p1 contains all the n-grams of the question (the one 5-gram, the two 4-grams, the three 3-grams, the four 2-grams and the five 1-grams of the question). Therefore the similarity of the question with this passage is 1.
The sets of n-grams of the passage p2 contain only the “the President of” 3-gram, the “the President”' and “President of” 2-grams and the following 1-grams: “the”, “President”, “of” and “Mexico”. The similarity for this passage give us a lower value than for p1 because the second passage is very different with respect to the original question, although it contains all the relevant terms of the question.
Previous evaluation of JIRS also demonstrates that it is possible to achieve coverage of over 60% for the first
20 passages. That is, the possible answer to a given question is found between the first 20 passages retrieved by
JIRS for over 60% of the training set. We refer the reader to [
Once the relevant passages are selected, the second step requires the POS tagged form of each passage in order to gather the representation used to extract the answer. Due to some technical constraints we were unable to finish the implementation for the alignment of the tagged collection and the JIRS index before test set release. Therefore the tagging of relevant passages was performed online with the disadvantage of a couple of extra hours for such processing.
Tagged passages are represented in the same way that in [
Answer extraction is performed according to the type of question, factual or definition. Next subsections detail the processes involved in answering each one. 6.1
The system makes no difference between factual and temporal restricted factual questions in order to extract their possible answer. Given the set of retrieved passages and their representations (named entities and their contexts) the system computes a weight for each candidate answer (named entity) based on two main factors: a) the activation and deactivation of some features at different steps of the system, and b) the coefficient gather by the formula 1.
The features listed in table 1 allow us to configure the system in order to change its behavior, for instance, deactivate the question classification step, allowing to the final answer selection to rely on no more information 2 The semantic classes used rely on the capability of the named entity classifier used in our experiments. that just statistical computations. The opposite case could be, deactivate frequency features and let the final answer selection to rely on the matching between question and candidate answers context.
ω
A = tq ∗ NEq n ∩ NE NEq
A +
Cq ∩ C
Cq
A + FA (Pi ) + 1 −
FA (P)
Pi k − 1
i=1..k; k=number of passages retrieved by JIRS
Where ω A is the assigned weight for a candidate answer; tq is 1 if the semantic class of the candidate answer is the same that the question’s one and 0 in other case; n is a normalization factor based on the number of activated features, NEq is the set of named entities in the question and NEA is the set of named entities in the context of the candidate answer; Cq is the question’s context and C A is the candidate answer’s context; FA (Pi ) is the frequency of occurrence of the candidate answer in the passage i; FA (P) is the total frequency of occurrence P of the candidate answer in the passages retrieved by JIRS; and 1 − i is an inverse relation for the passage k −1 ranking.
Activate question classification step Defines the number of classes to use in question and named entity classification.
Define the elements included as part of a name entity context. They could be: named entities, common names, verbs, adjectives, adverbs, etc.
Number of elements at left and right of a named entity to include in the context.
Defines if the passages without question’s named entities will be allowed.
Intersection Number of times that a named entity appears as candidate answer in the same passage.
Position of passage as retuned by JIRS.
Number of phrases in the passage retrieved.
Once the system computes the weight for all candidate answers, these are ranked by decreasing sort order, taking as answer the one with the greatest weight. Section 7 describes some experiments performed with training data set and the results achieved with both, training and test sets. 6.2
Our system uses an alternative method to answer definition questions. This method makes use of some
regularities of language and some stylistic conventions of news letters to capture the possible answer for a given
definition question. A similar approach was presented in [
The process of answering a definition question considers to main tasks. First, the definition extraction, which detects the text segments that contains the description or meaning of a term (in particular those related with the name of a person or an organization). Then, the definition selection, where the most relevant description of a given question term is identified and the final answer of the system is generated.
The language regularities and the stylistic conventions of news letters are captured by two basic lexical patterns. These patterns allow constructing two different definition catalogs. The first one includes a list of pairs of acronym-meaning. The other one consists of a list of referent-description couples.
In order to extract the acronym-meaning pairs we use an extraction pattern based on the use of parentheses.
In this pattern, w1 is a lowercase non stopword, <meaning> is a sequence of words starting with an uppercase letter (that can also include some stopwords), and <acronym> indicates a single word also starting with an uppercase letter.
By means of this pattern we could identify pairs like [PARM – Partido Auténtico de la Revolución Mexicana]. In particular this pair was catch from the following paragraph: “El Partido Auténtico de la Revolución Mexicana (PARM) nombró hoy, sábado, a Álvaro Pérez Treviño candidato presidencial de ese organismo para las elecciones federales del 21 de agosto de 1994”.
In contrast, the extraction of referent-description pairs is guided by the occurrence of a special kind of appositive phrases. This information was encapsulated in the following extraction pattern.
w1 w2 <description> , <referent> ,
Where w1 may represent any word, except for a preposition, w2 is a determiner, <description> is a free sequence of words, and <referent> indicates a sequence of words starting with an uppercase letter or being in the stopwords list.
Applying this extraction pattern over the below paragraph we could find the pair [Alain Lombard - El director de la Orquesta Nacional de Burdeos].
“El director de la Orquesta Nacional de Burdeos, Alain Lombard, ha sido despedido por el Ayuntamiento de esta ciudad, que preside Alain Juppé, tras un informe que denuncia malos funcionamientos y gastos excesivos”. (i) (ii)
The main quality of the extraction patterns is their generality. However, this generality causes the patterns to often extract non relevant information, i.e., information that does not indicate a relation acronym-meaning or concept-description. For instance, when using the extraction pattern (i) to analyze the following news we obtain the incorrect definition pair [Ernie - AFS]. In this case the resultant pair does not express an acronym-meaning relation; instead it indicates a person-nationality association.
Ernie Els (AFS) se mantiene en cabeza de la lista de ganancias de la "Orden de Mérito" de golf, con más de 17 millones de pesetas, mientras que el primer español es Miguel Angel Martín, situado en el puesto decimoséptimo, con 4.696.020.
Given that the catalogs contains a mixture of correct and incorrect relation pairs, it is necessary to do an additional process in order to select the most probable answer for a given definition question. The proposed approach is supported on the idea that, on the one hand, the correct information is more abundant than the incorrect one, and on the other hand, that the correct information is redundant.
Thus, the process of definition selection considers the following two criteria: 1. The more frequent definition in the catalog has the highest probability to be the correct answer. 2. The largest and therefore more specific definitions tend to be the more pertinent answers.
The following example illustrates the process. Assume that user question is “who is Félix Ormazabal?”, and that the definition catalog contains the records showed below. Then, the method selects the description “diputado general de Alava” as the most probable answer. This decision is based on the fact that this answer is the more frequent description related to Félix Ormazabal in the catalog.
Félix Ormazabal: Joseba Egibar: Félix Ormazabal: candidato alavés: Félix Ormazabal: diputación de este territorio: Félix Ormazabal: presidente del PNV de Alava y candidato a diputado general: Félix Ormazabal: nuevo diputado general Félix Ormazabal: diputado Foral de Alava Félix Ormazabal: través de su presidente en Alava Félix Ormazaba : diputado general de Alava Félix Ormazabal: diputado general de Alava
Félix Ormazabal: diputado general de Alava
This section discusses some training experiments and the decision criteria used to select the configuration of the experiments evaluated at QA@CLEF2005 monolingual track for Spanish. Given that we have used the same modules for answering definitions in all our runs for monolingual QA, including those described in “INAOEUPV Joint Participation at CLEF 2005: Experiments in Monolingual Question Answering”, the discussion on these results has been documented in that paper. Thus the rest of this document is intended to discus the results on factual question answering.
As we mention earlier, the approach used in our system is similar to the one used last year [
Despite the fact that our results (for factual questions) were over 10% better than last year, we believe that the approach described is near to its limits of accuracy. A shallow analysis of the results shows that the proposed system is suited for questions with some stylistic characteristics whose answer is commonly found in the near context of some reformulation of the question into the passages. While for others, more elaborated factual questions is unable to identify the right answer. That is the case of questions whose expected answer is an object or some abstract entity which can not be identified a priori by a shallow NLP or without a knowledge base.
Another point to note is that in some cases, the statistical factor given by the frequency of occurrence of a candidate answer becomes a secondary aspect that could yield to a wrong selection of an answer.
A detailed analysis of these results will help us to take the next direction in our research. 2322.5 2424.5 19 19
20
We have begun some experiments in order to get the right configuration for each question online, that is, to select automatically the appropriate configuration for a given question based on question’s attributes. Another direction in our research is to include more features that allow us to perform a better selection and discrimination of candidate answers, more over, that allow to consider objects and abstract entities that are currently excluded by the methodology.
This work has presented an approach for QA in Spanish centered in the use of lexical features for factual questions resolution that is complemented with a pattern matching approach for definition question resolution. The results achieved in the monolingual track for Spanish have improved our last year performance by over 10% on factual questions and over 30% on definition questions. It is important to note that the approach was able to answer over 30% of temporal restricted factual questions without additions or modifications to the proposed approach.
After a shallow analysis of these results we have begun to work in two directions: first the inclusion of other features that allow us to respond questions whose answer is not necessarily expressed as a reformulation of the question into the target documents. Currently our work in this direction is based on the study of a syntactic analysis of the retrieved passages, and in the inclusion of external knowledge. The second direction of research is the automatic selection of features online in order to get the best performance of the system given a question. Acknowledgements. This work was done under partial support of CONACYT (Project Grants U39957-Y and 43990), SNI-Mexico, and the Human Language Technologies Laboratory of INAOE. We also like to thanks to the CLEF as well as EFE agency for the resources provided.