This report presents the work carried out at NLE Lab for the QA@CLEF-2009 competition. We used the JIRS passage retrieval system, which is based on redundancy, with the assumption that it is possible to find the response to a question in a large enough document collection. The retrieved passages are ranked depending on the number, length and position of the question n-grams structures found in the passages. The best results were obtained in monolingual English, while the worst results were obtained for French. We suppose the difference is due to the question style that varies considerably from one language to another.
In this task the systems receive in input natural language questions about knowledge of European law, and these, in turn, should return a paragraph containing the response from the documents collection. This constitutes an important difference with respect to previous QA tasks where the answer had to be extracted by the system. For this reason we employed just the JIRS system instead of the complete QUASAR QA system we developed for previous participations.
The document collection is composed by the JRC-Acquis corpus3, containing the complete EU legislation, including texts between the years 1950 to 2006 (in total 10,700 documents); these documents have been aligned in parallel and were made available to the competition in the following languages: Bulgarian, Dutch, English, French, German, Italian, Portuguese, Romanian and Spanish. The corpus is encoded in XML format according to the TEI guidelines4. Each document have a title and is subdivided in a series of phrases, each one marked with the "<p>" tag.
The test set is composed of 500 questions that must be analyzed by the systems to return a paragraph that contains the answer to the formulated question. 3
Nowadays, many passage retrieval systems are not targeted to the specific problem of finding answers, due to the fact that they only take into account the keywords of the question to find the relevant passages. The information retrieval system JIRS is a based on n-grams (an n-gram is a sequence of n adjacent words extracted from a sentence or a question.) instead than keywords. JIRS is based on the premise that in a large collection of documents, an n-gram associated with a question must be found in this collection at least once.
JIRS starts searching the candidate passage with a standard keyword search that retrieves an initial set of passages. These passages are ranked later depending on the number, position and length of the question n-grams that are found in the passages. E.g.: suppose you have a publications database of a newspaper, using the JIRS system and based on these documents you will find the answer to the question: “Who is the president of Colombia?”; The system could retrieve the following two passages: "... Álvaro Uribe is the president of Colombia ...” and “...Giorgio Napolitano is the president of Italy...”. Of course, the first passage should have more relevance as it contains the 5-gram “is the president of Colombia”, while the second passage contains only the 4-gram “is the president of”. To calculate the n-grams weight of each passage, first of all you must identify the most relevant n-gram and assign to it an equal weight to the all weights sum of the terms. The weight of each Where nk is the number of passages in which the term appears and N is the total number of passages in the The similarity between a passage d and a question q is determinated by: Where ℎ( , ) returns a weight for the j-gram x with respect to the set of j-grams (Dj) in the passage: = 1 − log ( )
1+log ( ) ,
= =1 ∈ ℎ ,
=1 ∈ ℎ , ℎ , = 0 =1
∈
ℎ
(
A more detailed description of the system JIRS can be found in [5].
The data had to be preprocessed, due to the format of the collection employed in the ResPubliQA@CLEF-2009 competition, a subset of the JRC-ACQUIS Multilingual Parallel Corpus, this Corpus containing the total body of European Union (EU) documents, of mostly legal nature. In particular, the subset is constituted by documents of 9 out of 22 languages. It consists of approximately of 10,700 parallel and aligned documents per language. The documents cover various subject domains: economy, health, information technology, law, agriculture, food, politics and more.
To be able to use the JIRS system in this task, the documents were analysed and transformed for proper indexing. Since JIRS uses passages as basic indexing unit, it was necessary to extract passages from the documents. We consider any paragraph included between <p> tags as a passage. Therefore, each paragraph was labelled with the name of the containing document and its paragraph number.
Once the collection was indexed by JIRS, the system was ready to proceed with the search for the answers to the questions provided by the competition. For each question, the system returned a list with the passages that most likely contained the answer to the question, according to the JIRS weighting scheme. In an additional experiment, we used the parallel collection to obtain a list of answers in different languages (Spanish, English, Italian and French), choosing as the best answer the one that obtained the best score by JIRS and subsequently taking the identifier of each paragraph (answer) for retrieving the aligned paragraph in the target language. 5
We submitted four “pure” monolingual runs for the following languages: English, French, Italian and Spanish, and in an additional experiment we exploited the parallel corpus to produce a monolingual Spanish run. This experiment consisted in searching the question in all languages, and selecting the passage with the highest similarity; finally, the returned passage was the Spanish alignment of this best passage. In Table 1 we show the official results for the submitted runs. From Figure 1 we can see that the result obtained in English were particularly good, while in French and Spansish the percentage of wrong answers is very high. We did not expect this behaviour for the Spanish language, since JIRS was developed specifically for the Spanish QA task. On the other hand, we expected the French to be the language in which the system obtained the worst results, because of the results of the system at previous QA competitions.
The difference between the best results (for English) and the worst ones (in French) is of 22%. This may reflect the different way of formulating questions in each language. The abandonment of the Question Classification and Answer Extraction phases, with respect to our previous participation in the CLEF QA tasks, did not result particularly useful. In many cases, the right answer was available in a passage ranked lower, and this could have been detected by checking the question type and determining if the passage contained a candidate answer. We plan to re-introduce these modules for further participations, at least for the validation of the returned passage, that is, to check if it contains the answer or not.
The work of the first author has been possible thanks to a scholarship funded by Maat Knowledge in the context of the joint project with the Universidad Politécnica de Valencia “Módulo de servicios semánticos de la plataforma G”. We also thank the TEXT-ENTERPRISE 2.0 TIN2009-13391-C04-03 research project.