This paper describes the baselines proposed for the ResPubliQA 2009 task. These baselines are purely based on information retrieval techniques. The selection of an adequate retrieval model that fits the specific characteristic of the supplied data is considered as a core part of the task. Applying a not adequate retrieval function would return a subset of paragraphs where the answer could not appear, and thus the posterior techniques applied in order to detect the answer within the subset of candidates paragraphs will fail. In order to check the ability to retrieve the right paragraph by a pure information retrieval approach, two baselines are proposed. Both of them use the Okapi-BM25[3] ranking function, with and without a stemming preprocess. The main aim was to prove how well can a pure information retrieval system perform on this task.
∗This paper corrects a minor mistake found in the results of the not stemmed baseline. The previous results were slightly lower than those presented here.
1http://langtech.jrc.it/JRC-Acquis.html
Related works can be found on previously international competitions and workshops as
NTCIR7[
Different techniques can be found on the previous works specifically focused on applying classical retrieval models, to the selection of paragraphs or snippets within a document. In general these techniques are adapted to the data characteristics moreover of the straight use of the model.
The proposed baselines can be considered as a first phase within a typical pipeline architecture for question answering. That is, a first selection of paragraphs that are considered relevant for the proposed question are selected. Therefore the focus is to obtain a first set of paragraphs ordered according to their relevance with the question. The precision in terms of retrieving a correct answer for the question within the top k paragraphs, delimits in some sense the overall quality of the full system. In order to retrieve the most relevant paragraphs, the full collection has been indexed by paragraphs removing stopwords and applying a stemmer when there is some available. Both process are performed specifically by language. 2
The selection of an adequate retrieval model is a key part of this task, as only the returned paragraphs in this phase will be analysed to check if any of them contains the answer for the proposed question. In general, retrieval models are built around three basic statistics from the data: frequency of terms in a document; frequency of a term in the collection, where document frequency (DF) or collection frequency (CF) can be applied; and document length.
The ideal ranking function for this task should be adaptable enough to fit the specific characteristics of the data in use. For the ResPubliQA 2009 task the documents are actually paragraphs with an average length of ten terms, and where the frequency of a question term within a paragraph will hardly exceed one. Given the task characteristics, a paragraph candidate to contain the answer to a question will be one where the maximum number of question terms appear (excluding stopwords); with a length similar to the average, avoiding to give too much relevance to term frequency within the paragraph.
The use of the classic Vector Space Model[
R(q,d) = X t en q pf reqt,d · idft plength(d) (1)
A more adequate ranking function for this task is BM25[
The parameter b defines the length normalisation applied and it is computed as in the next equation:
B = (1 − b) + b(
dl
avdl
)
where parameter b ∈ [
tf = f reqt,d
B
Once the normalisation factor has been calculated it is applied to the term frequency. Final score is computed applying a term frequency saturation that uses the parameter k1 allowing us to control the effect of frequency in final score, as it can be seen in next equation: (2) R(q, d) =
tf tf + k1
· idft idft =
N − dft + 0.5
dft + 0.5
where ∞ > k1 > 0. Finally, IDF (Inverse Document Frequency) is tipically computed as next:
Where N is the total number of documents in the collection and dft is the number of documents
in the collection that contains t. An implementation of the BM25 ranking function over Lucene
was developed for this work3. The details of this implementation can be seen in [
R(q,d) = X t en q k1((1 − b) + b · alvdld ) + f reqt
· idft f reqt,d 3
In order to test the precision of our retrieval system we propose the execution of two baselines. In both baselines the paragraph selected in order to answer the question is the one that appears first in the ranking obtained after the retrieval phase. The only difference between both baselines is if a stemming pre-process is carried out or not. In order to proceed with the stemming process for each language4 the Snowball implementation that can be found at http://snowball.tartarus.org/ has been applied. The resources used for each language can be downloaded from the following sites: 1. Bulgarian: 2. German 3. English 4. French • Stoplist: http://members.unine.ch/jacques.savoy/clef/bulgarianST.txt • Stemmer: Not Available • Stoplist: http://members.unine.ch/jacques.savoy/clef/germanST.txt • Stemmer: http://snowball.tartarus.org/algorithms/german/stemmer.html • Stoplist: http://members.unine.ch/jacques.savoy/clef/englishST.txt • Stemmer: http://snowball.tartarus.org/algorithms/english/stemmer.html • Stoplist: http://members.unine.ch/jacques.savoy/clef/frenchST.txt • Stemmer: http://snowball.tartarus.org/algorithms/french/stemmer.html 3http://nlp.uned.es/~jperezi/Lucene-BM25/ 4Except Bulgarian for which not available stemmer could be found.
• Stoplist: http://members.unine.ch/jacques.savoy/clef/italianST.txt • Stemmer: http://snowball.tartarus.org/algorithms/italian/stemmer.html • Stoplist: http://members.unine.ch/jacques.savoy/clef/portugueseST2.txt • Stemmer: http://snowball.tartarus.org/algorithms/portuguese/stemmer.html • Stoplist: http://members.unine.ch/jacques.savoy/clef/roumanianST.txt • Stemmer: http://snowball.tartarus.org/algorithms/romanian/stemmer.html
• Stoplist: http://members.unine.ch/jacques.savoy/clef/spanishSmart.txt • Stemmer: http://snowball.tartarus.org/algorithms/spanish/stemmer.html The parameters values are equivalent for both baselines and have been fixed as next: 1. b: 0.6. Those paragraphs with a length over the average will obtain a slightly higher score. 2. k1: 0.1. The effect of term frequency over final score will be minimised.
Both parameters have been fixed to these values after a training phase with the English development set supplied by the organisation. The obtained results are shown and described in detail in the next section: 4
After the execution of both baselines the obtained results can be observed in Table 1, where the obtained results, the paragraphs average length and the best result obtained for each language. Where no data appears is due to no run was submitted for the specific language. The data for both baselines is the number of right paragraphs returned for the 500 questions. Between brackets an average of the P@1 measure per question appears. That is the number of questions answered correctly divided by the total of questions.
Some preliminary conclusions can be extracted from the obtained results. First, it is clear that the best results have been obtained for the English language (.53). This can be easily explained by the fact that the parameters were fixed using the English development set. It is expected that a window for improvement could be found fixing the parameters specifically for each language.
As it can be observed in Table 1 a general behaviour is the fact that best results are achieved with the use of stemming.
Moreover, it appears that for those languages with more lexical variability (Spanish, French) the improvement with the use of stemming is clearly higher in relation with lexically more simple languages as English, where less variation appears.
The lowest performance can be observed for languages like Bulgarian or German, we believe that it is due to the fact of the higher complexity of these languages. The performance for these languages could be increased with the use of a lemmatiser instead of a stemmer.
Finally, no correlation has been found between the performance and the paragraph average length for each language.
This work has been partially supported by the Spanish Ministry of Science and Innovation within the project QEAVis-Catiex (TIN2007-67581-C02-01), the TrebleCLEF Coordination Action, within FP7 of the European Commission, Theme ICT-1-4-1 Digital Libraries and Technology Enhanced Learning (Contract 215231), the Regional Government of Madrid under the Research Network MAVIR (S-0505/TIC-0267), the Education Council of the Regional Government of Madrid and the European Social Fund.