This paper presents our bilingual question-answering system MUSCLEF. We underline the di culties encountered when shifting from a mono to a cross-lingual system, then we focus on the evaluation of three modules of MUSCLEF: question analysis, answer extraction and fusion. We nally present how we re-use di erent modules of MUSCLEF to participate to AVE (Answer Validation Exercise).
This paper presents our cross-lingual question answering system, called MUSCLEF. This year we participated to the French-English cross-language task for which we submitted two runs. Like the past two years, we used two strategies: the rst one consists in translating only a set of terms selected by the question analysis module, this strategy being implemented in a system called MUSQAT; the second one consists in translating the whole question and then applying our mono-lingual system named QALC.
To our knowledge, none of the systems participating to CLEF continues to use the rst strategy (term translation), which seems indeed to give lower results than the rst one. Nevertheless, we think this approach remains interesting for several reasons: some languages may not dispose of good enough translation tools so this approach would be the only means to build cross-language systems; translation tools when they exist are not e cient for all types of questions; nally, our system MUSQAT could be used when the translation of a question is too di cult to obtain.
The paper is organized according to the following plan: rst we describe the architecture of MUSCLEF (section 2), then we underline some di culties when shifting from a mono to a crosslingual system (3), after we focus on evaluation and give results obtained by three particular modules of MUSCLEF (4), we give also the general results of our participation to CLEF (5). Lastly, before concluding, we present how we re-used di erent modules of MUSCLEF to build a rst system for the Answer Validation Exercise (6). 2
QALC, our mono-lingual system, is composed of four modules described below, the rst three of them begin classical modules of question answering systems: the rst module analyzes the question and detects characteristics that will enable us to nally get the answer: the expected answer type, the focus, the main verb and some syntactic features; the second module is the processing of the collection: a search engine, named MG 1, is applied; then the returned documents are reindexed according to the presence of the question terms. Next a module recognizes the named entities and each sentence is weighted according to the information extracted from the question; the third module is the answer extraction which applies two di erent strategies depending on whether the expected answer is a named entity or not; the fourth module is the fusion. Indeed our system QALC is applied on the Web as well as on the closed collection of the CLEF evaluation, then a comparison of both set of answers is done; this way, we increase the score of answers that are present in both sets.
To build MUSCLEF, our cross-lingual question anwering system, we added several modules to QALC, corresponding to both possible strategies to deal with cross-lingualism: question translation and term-by-term translation. In MUSCLEF, the rst strategy uses Reverso 2 to translate the questions then our mono-lingual system QALC is applied. The second strategy, that we named MUSQAT, uses di erent dictionnaries to translate the selected terms (a description and an evaluation of this translation are given section 3).
Finally, we apply the fusion module to the di erent sets of answers: a rst one corresponds to MUSQAT, a second one corresponds to the application of QALC on the translated questions, both these sets of answers coming from the CLEF collection of documents, and a third one corresponds to the application of QALC on the translated questions using the Web. MUSCLEF is presented Figure 1, where the rst line of modules corresponds to our mono-lingual system QALC and the second line contains the modules necessary to deal with cross-lingualism.
French questions
English questions
Question analysis
Answer type Focus Semantically linked words Syntactic relations Main verb Terms
English translation
Web / Collection Search engine English terms
Document processing
Reindexing and ranking Selection
Named entity tagging Answer extraction
Sentence weighting Answer extraction 2 lists of ranked answers
Fusion
English answers 1MG for Managing Gigabytes, http://www.cs.mu.oz.au/mg/ 2http://www.reverso.net
Document selection Document processing
Five sentences Five short answers First short answer
QALC 94.4 93.3 67.5 40 28
QALC + Reverso MUSQAT 88.3 84.4 87.7 82.2 58.5 50.5 39.5 36.5 26 23
3.1
After CLEF 2005 evaluation, CLEF organizers gave the original set of question written in good English to the participants, from which all sets of question were derived. Thanks to this new set of questions we could compare the behaviour of our di erent implementations: mono-lingual QALC, cross-lingual QALC (using Reverso), and cross-lingual MUSQAT. The results are given table 1. The results of document selection and document processing were calculated for 180 questions instead of 200 because of the 20 NIL questions. Each number in this table represents the percentage of questions for which a good document/sentence/answer is returned.
Concerning the rst three lines, we observe a big di erence between the mono-lingual and the cross-lingual systems (from to 6 to 17 %). This di erence is due to missing translations: for instance acronyms or proper names (which original alphabet can be di erent from ours) are often not correctly translated. In the last two lines, the di erences are more surprising (and we could not explain them yet): the mono-lingual system lost 40% of good answers during answer extraction, while the best cross-lingual system, QALC+Reverso lost 32.5%, and MUSQAT lost 27.7%.
On the same data of CLEF 2005, [
In this section, we present how in MUSQAT, we proceeded to the term and multi-term translation and to the validation of this translation. The translation is achieved using two dictionaries, MagicDic 4 and FreeDict 5, both being under GPL licence. Thus, the system MUSQAT gets several translations for each French word, which can be either synonyms or di erent translations when the term is polysemic.
The evaluation made last year (reported in [
For example, to the French bi-term cancer du sein corresponded the three following translations: breast cancer, chest cancer and bosom cancer. In the retained document only the rst translation is present, this lead us to discard the terms chest, bosom and their corresponding bi-term.
3It is the best of their cross-lingual systems with a percentage of 39.5 of good answers, while their Portugueseto-French system gets 36.5 and their Italien-to-French system gets 25.5.
4http://magic-dic.homeunix.net/ 5http://freedict.org/en/ 6Fastr was developed by Christian Jacquemin, it is a transformational shallow parser for the recognition of term occurrences and variants , http://www.limsi.fr/Individu/jacquemi/FASTR/
We hoped this way to decrease the noise due to the presence of wrong translations. Unfortunately, this rst experience in translation validation was not convincing for we obtained nearly the same results in MUSQAT with or without it. (22% of good answers without the validation, 23.5% with it).
Undoubtedly this approach needs to be enhanced but also evaluated on larger corpora. Indeed, we only evaluated it on the corpus of CLEF 2005 questions, on which we obtained the following gures: from the 199 7 questions, we extracted 998 bi-terms from 167 questions and 1657 non empty mono-terms; only 121 bi-terms were retrieved in documents, which invalidated 121 mono-terms and reduced the number of questions with at least one bi-term to 98. The number of invalidated mono-terms (121) is certainly not high enough in this rst experiment to enable MUSQAT to reduce the noise due to wrong translations. 3.3
Yet, after the translation and its validation, some terms are absent from these dictionaries, and thus remain untranslated. A module was developed to try and translate these terms using the Web.
Description Since some terms are absent from our dictionaries, we decided to look for them in Web resources. These resources can be on-line dictionaries like Mediaco, Ultralingua or other dictionaries from Lexilogos 8, but not necessarily: for example, we also use the free encyclopedia Wikipedia, and the web site for European languages and cultures Eurocosm.
Many of the terms that remain untranslated are multi-word terms, which require a special strategy because it is not always possible to search directly for multi-words expressions in the Web resources. The translation is thus composed of three steps. First, all the multi-word terms are cut into single words. Then we browse the Web to get pages from all the on-line resources that contain these words. Each page is mapped into a common format which gives for each term its translations (there can be several ones). Finally, for each term of the original list, we look for all exact matches in the Web pages, and the most frequent translation is chosen.
Table 2 shows an example of a mapping for the French term "voiture" and table 3 the frequency of each of its translations. To avoid incorrect translations, we only consider the translations of the exact term. For the term \voiture", the most frequent translation is \car" and thus this translation is chosen.
French term voiture voiture voiture d'enfant
... voiture voiture voiture de fonction
... voiture
... cle de voiture voiture
Translation
car carriage baby-carriage ...
car automobile company car ... car ...
car key automobile
Translation
car automobile
coach carriage # of occurrences 3 2 1 1 7199 instead of 200 because one has been thrown out by the process 8www.lexilogos.com
For each corpus, about 30% of the originally untranslated terms were translated by this module. Most of the terms that can't be translated are actually incorrect multi-word terms in French, mostly because the words are lemmatized, which leads to incorrect terms like \second guerre" (instead of \seconde guerre") or \seigneur de anneau" (instead of \seigneur des anneaux").
The question analysis module determines several characteristics of the question among which its category, expected answer type (named entity or not) and focus. We conducted a corpus study in order to validate our choice concerning these characteristics, and the focus in particular, on the corpus of English questions and collection. For the focus, we found that 54% of the correct answers contain the focus of the question, while only 32% of the incorrect answers do (against 20% and 11% for an non-empty word chosen by chance in the question), which tends to validate the choice we made for the focus.
The performance of this module was evaluated in [
In MUSQAT and QALC, we use the same method to extract the nal short answer from the candidate sentence. And in both these systems, this last step of the question-answering process entails an important loss of performance. Indeed, in MUSQAT and QALC the percentage of questions for which a candidate sentence containing the correct answer is ranked rst is around 35%, and as seen in section 3 the percentage of questions for which a correct short answer is ranked rst falls to around 25%. During this step, we lose about one third of good answers.
In [
Since we now have three sets of results to merge, we proceeded in two steps: we rst merged the results of QALC+Reverso and MUSQAT, which gave us our rst run. And, as a second run, we merged our rst run and the set obtained with QALC+web system.
Those tests were done on the CLEF 2005 data, and we can see table 5 that neither the rst fusion nor the second enabled us to increase our results. Nevertheless, as we can see it section 5, on CLEF 2006 results the second fusion using the web gave better results since we obtained 25 % of good answers with the web and 22 % without.
[
Concerning the rst fusion, both systems (QALC+Reverso and MUSQAT) giving similar
results, it is not surprising that the fusion does not increase the number of good answers. However,
our fusion algorithm (described in details in [
In order to build the Answer Validation system, we used our QA system, applied to the hypotheses and justi cations rather than to the questions and the collection, and we added a decision module. Our goal was to obtain the information needed to decide whether the answer was entailed by the text proposed to validate it.
First the initial corpus le goes through a formatting step transforming it into a le which may be treated by our system, then the QA system is used to extract needed information from it, like tagged hypothesis, tagged justi cation snippet or terms extracted from the question for example. They are written in a pseudo-xml le passed to the decision algorithm. We also get the answer our QA system would have extracted from the proposed justi cation, which is used to see if the answer to judge is likely to be true.
Then, the decision algorithm proceeds in two main steps. During the rst one, we try to detect quite evident mistakes, such as the answers which are completely enclosed in the question, or which are not part of the justi cation.
The second step proceeds to more sophisticated veri cations : (a) verifying the adequate type of the expected named entity if there is one; (b) looking the justi cation for terms judged as important during the question analysis; (c) con rming the decision with an extern-justi cation module using the latest version of Lucene to execute a number of coupled queries on the collection, like proximity queries (checks if a number of terms can be found close to one another within a text); the top results of each couples queries are compared in order to decide whether the answer is likely to be true or not; (d) comparing the results that our answer-extraction module (part of our QA system) would provide from the justi cation text.
The results obtained by these di erent veri cations are combined to decide if the answer is justi ed or not and to give a con dence score to this decision. Some errors have been corrected after submitting our results to the AVE campaign (which were rather bad, with very few positive answers). We proceeded to a partial evaluation on our positive answers. We found 363 \YES" among more than 3,000 hypothesis-snippet pairs. About 80% of them are \good" ones. When we only consider the \YES" with a con dence score of 1, we obtained 146 answers, with 90% of good answers. So our algorithm has a good precision, but we have not evaluated the recall result. 7
Our cross-lingual system MUSCLEF presents the particularity to use three strategies in parallel: question translation, term-by-term translation and the use of another source of knowledge (limited actually to the Web). The three sets of answers are nally merged thanks to a fusion algorithm proceeding on two set of answers at the same time. The term-by-term strategy gives lower results than the most widely used strategy consisting in translating the question into the target source then applying a mono-lingual strategy. Nevertheless, we think it remains interesting from the multilingualism point of view, and we try to improve it by using of di erent techniques of translation (use of several dictionaries and on-line resources) and validation.