This report describes the participation of the NLEL Lab. from the Universidad Politecnica of Valencia to the RespubliQA task at CLEF 2010. The system designed for this participation is based on the one used in our previous participation, with some modi cations required in order to adapt it to the new guidelines. The system participated to both the \Paragraph Selection" (PS) and \Answer Selection" (AS) subtasks.
The participation to the PS sub-task was centered around the JIRS n-gram
based passage retrieval system [
The architecture of JIRS is shown in Figure 1. The user question is passed to a search engine that returns relevant snippets of a documents collection in which relevant terms from the question occur. The n-gram extraction module will return all the n-grams of size 1 to n, where n is the number of terms of the question. This process is done both for the question and for each of the snippets 1 http://sourceforge.net/projects/jirs/ retrieved by the search engine. Once obtained the n-grams of the question and the snippets, a comparison is made to calculate a similarity value between them. This similarity value is used to sort the list of passages that will eventually be returned to the user. The similarity between the question and the retrieved passages is defined in Equation 1.
Sim(p; q) = 8x2Q ∑ h(x; P )
1 d(x; xmax) ∑n i=1 wi
Where, Sim(p; q) is the function that measures the similarity of n-grams sets of the question q with respect to the n-grams sets of the passage p. P is the n-gram set of the heaviest passage p (i.e., the one with most weight) whose terms are in the question; Q is the set of j-grams that are generated from the question q and n is the total number of terms in the question. There are three special and particular terms functions: { wi is the weight of the i -th term of the question which is determined by: wi = 1
log(ni) 1 + log(N ) Where ni is the number of sentences in which the term ti occurs and N is the number of sentences in the collection; { the function h(x; P ) measures the weight of each n-gram and is defined as: h(x; Pj ) = { ∑jk=1 wk 0 if x 2 Pj otherwise Where wk is the weight of the k-th term (see Equation 2) and j is the number of terms that compose the analyzed n-gram; (1) (2) (3) 1 { and the factor d(x;xmax) that is a distance factor which reduces the weight of the n-grams that are far from the heaviest n-gram. The function d(x; xmax) determines numerically the value of the separation according to the number of words between a n-gram and the heaviest one. That function is defined as shown in Equation 4 : d(x; xmax) = 1 + k ln(1 + L) (4) Where k is a factor that determines the importance of the distance in the similarity calculation and L is the number of words between a n-gram and the heaviest one (see Equation 3). 3
In order to cope with the AS sub-task guidelines, which require that beyond
retrieving a paragraph containing the answer to a question in natural language,
systems are required to demarcate also the exact answer, we had to fit JIRS
with an answer extraction system. This system is based on the QUASAR AE
module described in [
This module obtains both the expected answer type (or class) and some constraints from the question. The different answer types that can be treated by our system are shown in Table 1.
Each category is defined by one or more patterns written as regular expressions. For instance, the Italian patterns for the category “CITY” are: .*(che|quale) .*citta .+ and (qual|quale) .*la capitale .+ . The questions that do not match any defined pattern are labeled with OTHER. If a question matches more than one pattern, it is assigned the label of the longest matching pattern (i.e., we consider longest patterns to be less generic than shorter ones).
The Question Analyzer has the purpose of identifying patterns that are used as constraints in the AE phase. In order to carry out this task, the set of different n-grams in which each input question can be segmented are extracted, after the removal of the initial quetsion stop-words. For instance consider the question: “Where is the Sea World aquatic park? ”, then the following n-grams are generated: [Sea] [World] [aquatic] [park] [Sea World] [aquatic] [park] [Sea] [World aquatic] [park] [Sea] [World] [aquatic park] [Sea World] [aquatic park] [Sea] [World aquatic park] [Sea World aquatic] [park] [Sea World aquatic park] The weight for each segmentation is calculated in the following way: x2Sq ∏ log 1 + ND log ND log f (x) (5) where Sq is the set of n-grams extracted from query q, f (x) is the frequency of n-gram x in the collection D, and ND is the total number of documents in the collection D.
The n-grams that compose the segmentation with the highest weight are the contextual constraints, which represent the information that has to be included in the retrieved passage in order to have a chance of success in extracting the correct answer. 3.2
The input of this module is constituted by the n passages returned by the PR module and the constraints (including the expected type of the answer) obtained through the Question Analysis module described in Section 3.1. The positions of the passages in which the constraints occur are marked before passing them to the text analyzers (we named them TextCrawlers since they move on text like a spider on its web). One of these analyzers is instantiated for each of the n passages with a set of patterns for the expected type of the answer and a pre-processed version of the passage text.
Each TextCrawler begins its work by searching all the passage’s substrings matching the expected answer pattern. Let us define C the set of constrains extracted in the Question Analysis phase; then a weight w(s) is assigned to each found substring s, inversely proportional to the text distance of s with respect to the constraints ci 2 C. The final weight w(s) is calculated as the product of the distance weights obtained for every constraint in the passage: w(s) = ∏ci2C 1=d(s; ci).
A Filter module is based on a set of patterns compiled by hand in order to discard the candidate answers which do not match an allowed pattern or that do match with a forbidden pattern. When the Filter module rejects a candidate, the TextCrawler provide it with the next best-weighted candidate, if there is one. Finally, when all TextCrawlers end their analysis of the text, the Answer Selection module selects the answer to be returned by the system. The following strategies apply: { Simple voting (SV): The returned answer corresponds to the candidate that occurs most frequently as passage candidate. { Weighted voting (WV): Each vote is multiplied for the weight assigned to the candidate by the TextCrawler and for the passage weight as returned by the PR module. { Double voting (DV): As simple voting, but taking into account the second best candidates of each passage. { Top (TOP): The candidate elected by the best weighted passage is returned.
SV is used for NAME type questions, with DV as a backoff strategy in case of two candidates obtaining the same weight. WV is used for every other type of questions, with TOP as a backoff strategy. 4
For the RespubliQA 2010 competition, the NLE Lab.has decided to participate in five monolingual tasks for passages extraction, the distribution of these tasks with the respective approaches used is: { English task : Monolingual and monolingual - Stem participation; introducing these two units is expected to determine whether use of the Stem technique improves the performance of JIRS or not. { Spanish task : Monolingual and monolingual - BM25 participation; introducing these two units is expected to determine whether the use of the BM25 technique improves the performance of JIRS or not. { French, Italian and German Tasks: We present monolingual and multilingual approaches.
The following sections explain each one of the approaches implemented.
The data had to be preprocessed, due to the format of the collection employed in ResPubliQA competition, a subset of the JRC-ACQUIS and Europarl Multilingual Parallel corpus. The documents cover various subject domains: law, politics, economy, health, information technology, agriculture, food and more.
To be able to use the JIRS system in this task, the documents were analyzed and transformed for proper indexing. Since JIRS uses passages as basic indexing unit, it was necessary to extract passages from the documents. We consider any paragraph included between <p> tags as a passage. Therefore, each paragraph was labeled with the name of the containing document and its paragraph number.
Once the collection was indexed by JIRS, the system was ready to proceed with the search for the answers to the test questions. For each question, the system returned a list with the passages that most likely contained the answer to the question, according to the JIRS weighting scheme. The architecture of the monolingual JIRS -based system is illustrated in Fig. 2. According to the excellent results obtained in the competition RespubliQA 2009 we decided to implement the multilingual approach also in RespubliQA 2010. This approach used the parallel collection to obtain a list of answers in different languages (Spanish, English, Italian, French and German). The idea of this approach is based on the implementation of 5 monolingual JIRS -based systems, one for each language, which process the set of questions in the respective language. For this purpose, we used a parallel sets of questions provided by the competition organisers. The final answer is selected as the one obtaining the best score; if the answer is not in the target language, the identifier of each paragraph (answer) is used to retrieve the aligned paragraph in the target language. The architecture of the multilingual JIRS -based system is illustrated in Fig. 3. 4.3
The Monolingual - Stem approach was inspired by the competition of the year
2009 [
In Table 2 shows the results of the paragraph selection task. In Table 3 shows the results of the answer selection task.
As shown in Table 2, the monolingual approach applied to each of the five
languages returns acceptable results, especially in English and Italian. The
implementation of the systems: multilingual, monolingual - stem and monolingual
- BM25, decreased the overall performance of the system with respect to the
monolingual approach that used the JIRS n-grams density weighting scheme.
This result confirms that the n-grams density weighting scheme of JIRS fits
particularly well the QA task, with respect to term-based weighting scheme, as
observed in [
It is important to note that the multilingual approach was not able to repeat the results obtained in the RespubliQA-2009 competition. An analysis of the results in RespubliQA-2010 showed that the provided corpus is not perfectly aligned, as it can be observed from Tables 4 and 5: a passage with the same ID in the same document can be different for each of the studied languages. This problem is present in both the JRC-AQUIS and Europarl corpora.
Due to the scheme adopted for the multilingual approach it is necessary to work on a corpus with 100% accuracy in alignment; otherwise, the system is not able to obtain good results, as it can be seen in Table 2. Due to the fact that the
Passage Text 139 4. BESTIMMUNG DER PEPSINAKTIVITT 139 7.3 SE IL PALLONE DELL'APPARECCHIO DI. . . 139 7 . OBSERVATIONS 139 3.2 . Acido clorh drico 0,075 N .
139 D nition : L'unite de pepsine est de nie comme. . .
Passage Text 127 4. souhaite vivement entamer des. . . 127 4. Expresses its strong willingness to enter. . . 127 3. Toma nota de la Comunicacion de la. . . 127 5. fordert, dass die gegenwrtige Krise nicht. . .
127 4. esprime la sua forte volonta di avviare. . . answers for the Answer Selection task were extracted from the same passages retrieved in the basic multilingual approach, the results obtained for this task were also poor as shown in Table 3. 6
According to the experiments, the use of techniques such as BM25 and Stem, decrement the performance of JIRS tool for purposes of question answering tasks. It is verified through analysis, that problems with the alignment of the corpus provided poor performance resulting in the multilingual approach used. Additionally, Due to the poor result obtained with the multi-lingual approach, the extraction experiment response has similarly low results. In future work, we plan to implement a filter able to determine the paragraphs alignment of the corpus to improve the performance of multilingual approach.
We thank the TEXT-ENTERPRISE 2.0, MICINN (Plan I+D+i) research project (TIN2009-13391-C04-03).