For our second participation to the Question Answering task of CLEF, we kept last year's system named MUSCLEF, which uses two translation strategies implemented in two modules. The multilingual module MUSQAT analyzes the French questions, translates \interesting parts", and then uses these translated terms to search the reference collection. The second strategy consists in translating the question in English and applying QALC our existing English module. Our purpose in this paper is to analyze term translations and propose a mechanism for selecting correct ones. The manual evaluation of bi-terms translation leads us to the conclusion that bi-term translations found in corpus can con rm mono-term translations.
Cass robust parser1 to analyze the English questions that were translated using Reverso2. As a new type of questions, the temporally restricted questions, was introduced in this year's campaign, we have adjusted question analysis to the category of the question. When a temporal restriction was to be found, we tried to detect it, and to classify it according to the three following types: date, period, and event. The answering strategy was then adapted to the type of temporal constraint.
For querying the CLEF collection and retrieving passages we used MG3. Retrieved documents are then processed: they are re-indexed by the question terms and their linguistic variants, reordered according to the number and the kind of terms found in them, so as to select a subset of them. Named entity recognition processes are then applied. The answer extraction process relies on a weighting scheme of the sentences, followed by the answer extraction itself. We apply di erent processes according to the kind of expected answer, each of them leading to propose weighted answers.
The rst run we submitted corresponds to the strategy implemented in MUSQAT: translation of selected terms. For the second run, we added a nal step consisting in comparing the results issued from both strategies: the translated questions and the translated terms. This module named fusion in Figure 1, computes a nal score for each potential answer, its principle is to boost an answer if both chains ranked it in the top 5 propositions, even with relatively low scores.
French Questions
English questions
Question analysis Answer type Focus Semantically linked words Syntactic relations Main verb Terms
English Translation
Collection
Search engine English terms
Document processing
Reindexing and ranking Selection
Named entity tagging Answer extraction
Sentence weighting Answer extraction
Fusion
English
Answers 2 lists of ranked
answers
The automatic indexing of documents is performed by FASTR, a transformational shallow parser
for the recognition of term occurrences and variants. Terms are transformed into grammar rules
and the single words building these terms are extracted and linked to their morphological and
semantic families. The morphological family of a single word w is the set M(w) of terms in the
CELEX database
For instance, the following pattern, named NtoSemArg, extracts the occurrence making many automobiles as a variant of the term car maker :
VM('maker') RP? PREP? ART? (JJ|NN|NP |VBD|VBG)0-3 NS('car') where RP are particles, PREP prepositions, ART articles, and VBD, VBG verbs. VM('maker') is any verb in the morphological family of the noun maker and NS('car') is any noun in the semantic family of car.
Relying on the above morphological and semantic families, auto maker, auto parts maker, car manufacturer, make autos, and making many automobiles are extracted as correct variants of the original term car maker through the set of metarules used for the QA-track experiment. Unfortunately, some incorrect variants are extracted as well, such as make those cuts in auto produced by the preceding metarule. 4
Di erent methods can be used to achieve term translation and we considered the easiest one, which consists in using a bilingual dictionary to translate the terms from the source language to the target language. This simple method presents two drawbacks: it is impossible to directly disambiguate the various meanings of the words to be translated, and the two languages must be of equivalent lexical richness. To give an idea of the ambiguities we may encounter in a QA context, we studied the corpus of 1893 questions in English of TREC. After analysis, we kept 9000 of the 15624 words used in this corpus. The average of the number of meanings was 7.35 in WordNet. The extrema were 1 (example: neurological ) and 59 (example: break ). Around the average value, we found common words such as prize, blood, organization. Hence, we could not consider a dictionary giving only one meaning for a word. Moreover we needed to de ne a measure of the value of a translation in our QA context.
With these constraints, we studied the di erent dictionaries we could use: the online dictionaries (such as Reverso4, Systran5, Google6, Dictionnaire Terminologique7 or FreeTranslation 8), and the dictionaries under GPL licences (such as Magic-Dic9, Unidic or FreeDict10). The online dictionaries are generally complete. But they resolve the ambiguity and they only give one translation per word. Another limitation was the fact that we could not modify these dictionaries, and that we had to deal with some technical constraints such as the limited number of requests we may adress and the access time. Concerning the GPL dictionaries, they are obviously less complete, but they can be modi ed, they are very fast and for most of all, they give several translations for a request, as classical bilingual dictionaries. Among the GPL dictionaries, we chose Magic-dic, because of its evolutivity: terms can be added by any user, but they are veri ed before being integrated, and FreeDict. For example the query for the French word porte to Magic-Dic gives the following results (we only give an excerpt): porte bagages - luggagerack, luggage rack porte cigarette - cigarette holder porte clefs - key-ring porte plume - fountain pen 4http://translation2.paralink.com 5http://babel.altavista/translate.dyn 6http://www.google.com/language tools 7http://granddictionnaire.com 8http://www.freetranslation.com 9http://magic-dic.homeunix.net/ 10http://www.freedict.de/ porte parole, locuteur - spokesman porte - door, gate
To prevent Magic-dic uncompleteness, and because it has been proved that the use of several dictionaries gives better results than a unique one, we used this year two dictionaries and merged their translations. FreeDict had added 424 di erent translations of the 690 words. However, these new translations are mainly other synonyms rather than new translations of unknown words. 4.1
The multilingual module MUSQAT We illutrate the strategy de ned in our multilingual module MUSQAT on the following example: \Quel est le nom de la principale compagnie aerienne allemande?", which is translated in English \What is the name of the main German airline company?".
The rst step is the parsing of the French question that provides a list of the mono-terms and all the bi-terms (such as adjective/common noun) which are in the question, and eliminates the stop words. The bi-terms are useful, because they allow a disambiguisation by giving a (small) context to a word. In our example, the bi-terms (in their lemmatized form) are: principal compagnie, compagnie aerien, aerien allemand ; and the mono-terms: nom, principal, compagnie, aerien, allemand.
With the help of the dictionnaries, MUSQAT attempts to translate the bi-terms (when they exist), and the mono-terms. All the proposed translations are taken into account. All the terms are grammatically tagged. If a bi-term cannot be directly translated, it is recomposed from the monoterms, following the English syntax. For our example, we obtained for the bi-terms: principal company/main company, air company, air german; and for the mono-terms: name/appellation, principal/main, company, german. When a word does not exist in the dictionnaries, we keep it as it whithout any diacritic, which is often relevant for proper nouns. Then, all the words are weighted relative to their existence in a lexicon that contains the vocabulary found in Latimes of the Trec collection, so that each word is weighted according to its speci city within this corpus. If a word is not found in this lexicon, we search with MG if documents contain it (or rather its root because MG indexation was made using stemming). If it is not the case, MUSQAT eliminate it from the list of translated terms. By this way, MUSQAT discarded 72 non-translated words (on 439 non-translated mono-terms, the remaining ones often being proper nouns). As we form boolean requests, it was important not to keep inexisting words.
English terms plus their categories (given by the Tree Tagger) were then given as input to the other modules of the system, instead of the original words. The translation module did not try to solve the ambiguity between the di erent translations. We account on the document retrieval module to discard irrelevant translations. This module has been improved this year: it always selects passages (the collection was preliminary splitted), but in a very smaller number. It rst generates boolean requests, based on proper nouns, numbers and speci city of the words. It aims at retrieving 200 passages maximum, and makes the smaller request with the more speci c terms so as to obtain a minimum number of passages, set to 50. Each term of the request is made of the disjonction of the di erent translations. If the boolean query leads to retrieving too few or too much documents, passage retrieval is made thanks to a ranked research with a query that hold all the terms. If di erent terms are synonyms, relevant documents are then retrieved with these synonyms. If a word is incoherent within the context, we suppose its in uence is not su cient to generate noise. This hypothesis can only be veri ed if the question is made of several words. 5
We manually evaluated the bi-term translations for the 200 questions of CLEF04 given by this module. Table 1 presents the results of this evaluation. The system found 375 bi-terms. Among them, 135 are correct translated bi-terms (OK) such as CERN member. 24 are bi-terms contextually false i.e. for which one word is not a good translation in the context of this bi-term, such as accretion hormone instead of growth hormone to translate hormone de croissance. 74 bi-terms are due to an erroneous bi-term constitution (False Bi-Terms), such as able animal in question asking to Give an animal able to.... Finally, 142 bi-terms are (a) completely erroneous translations (False Translation), such as overground escort instead of main company (110) or (b) the translation was absent from the dictionnary (Absent Translations), such as olympique, where the French word has been kept instead of the English termolympic (32).
It is obvious on this table that a lot of terms are wrong for di erent reasons. We decided to con rm those that must be kept by considering their presence or absence in the selected documents. To do so, we used FASTR results to evaluate the bi-terms or their variants which are retrieved in the documents. Table 2 shows the results of this evaluation. The second column gives the results obtained by FASTR without considering the semantic variations. The third column includes these semantic variations. The last column indicates the percentage of bi-terms FASTR con rms, taking into account the semantic variations.
False Transl Absent Transl Total
The correct bi-terms are mostly con rmed by FASTR. The contextually false bi-terms obtain a rather high percentage of con rmation due to the semantic variations which lead to recognize correct synonyms of non accurate translated terms. The false bi-terms can be considered as co-occurrences rather than bi-terms. As co-occurrences, they are retrieved by FASTR in the documents and just a few false translations are retrieved. 6
We also proceeded to a similar evaluation of the terms extracted from the questions translated last year by Systran.
As a rst step we proceeded to an evaluation of the question translations themselves. We evaluated the syntactic quality of the translations, and classi ed them in correct, false, or quite correct. Table 3 recapitulates these results.
200 100
We also evaluated the terms extracted from these translated questions by our monolingual system QALC. We use the same notations than in table 1. Results are given Table 4.
These results are quite interesting: despite the moderate quality of the translations, QALC is able to identify good terms from these questions. We can also notice that we obtain a smaller number of terms following this procedure because there is only one translation by word. 7
Table 5 gives the results that our system obtained at the CLEF04 and CLEF05 campaigns, with the di erent strategies: (a) with the translation of the terms (MUSQAT), (b) with QALC applied on the translated questions and searching the collection. The evaluation was made by an automatic process that looks for the answer patterns in the system answers, applying regular expressions. These results were computed with 178 answer patterns that we built for the 200 questions of CLEF04 and 188 for the CLEF05 questions.
The rst line indicates the number of correct answers found in the 5 rst sentences given by MUSQAT (using term translation) and QALC. The second line, \NE answers", gives the number of correct answers on questions the system categorized as waiting for a Named Entity (the total is 107 in CLEF04 for MUSQAT and 97 for QALC and 91 in CLEF05 for MUSQAT and 66 for QALC). Our total number of questions of this category is far beyond the real number in CLEF05. The third line, \non NE answers", concerns the other questions (the complement to 178 in CLEF04 and to 188 in CLEF05). Results are presented when the system just gives one answer and when it gives 5 answers. The last line indicates the best o cial result of our system on the 200 questions. The o cial score of MUSQAT was 22 (11%) in CLEF04 and 28 (14%) in CLEF05, thus we can observe that merging answers obtained by di erent strategies enables a signi cative gain. We also can notice that if our CLEF05 system better selects sentences, it is less performant on extracting the answers, specially on named entity answers.
According to the manual evaluation results of bi-terms translations, we have tested an automatic process for ltering Magic-dic translations on CLEF04 questions. So, if a bi-term or a variant form was found in the selected documents, we kept it as a valid translation and we kept its lemmas as valid mono-term translations. When a validated translation existed for a term, the non-validated translations were taken out. When no translation of a bi-term was found in corpus, we assumed that mono-term translations were wrong and we kept Systran translations. In order to improve the coverage of our translation, we added Systran translation for terms absent from Sentences 5 rst ranks NE Rank 1 answers 5 rst ranks Non NE Rank 1 answers 5 rst ranks Total Rank 1 % 5 rst ranks Fusion (o cial results) 38 (19 %) 38 (19 %) the dictionary. By this way, we took o 253 bi-terms in 112 questions, and added 37 translations, with 12 bi-terms, which concerns 35 questions. The last improvement consisted in adding Systran translations that were di erent from Magic-dic translations (138 terms in 96 questions) to the ltered terms. This last set of terms was compound of 1311 translations for 836 terms in 200 questions (522 terms with 1 translation, 199 with 2 translations, 81 with 3 translations, 25 with 4 translations, 6 with 5 translations and 3 with 6 translations).
We tested MUSQAT with this new selection. Results are shown Table 6. We see that MUSQAT nds relevant documents for 7 supplementary questions (increase of 4%).
MUSQAT extracts 7 supplementary correct answers in the top 5 short answers, with 29 answers in rank 1. MUSQAT obtains here slightly better results than QALC with Systran translations, both for short and long answers. We also measured the number of questions for which the selection process based on FASTR indexing provides documents containing the answer pattern. In the original MUSQUAT, it was possible to nd the answer for 80% of questions. Term selection allows to improve this value to 85%. These improvements are not signi cative enough so we had not incorporated them in this year's version, even if we think that this kind of translation validation is worth being tried. So we plan to realize bi-term validation on a larger corpus. Concerning the absence of translations, we began to increase manually our dictionary from lexicons and gazetteers we use for named entities recognition, specially for acronyms and location names, and we plan to use a bilingual aligned corpus.