This paper presents our ¯rst participation in the bilingual English-Spanish track at CLEF QA 2006. The Multilingual BRUJA system is presented, a Question Answering (QA) system that works with questions in several languages and also collections in several languages. The BRUJA system is currently in its ¯rst phase of develop, so we have only run one o±cial experiment with questions into English and the collection into Spanish. The results obtained shown that the prototype and its answer extraction phase, have to be ¯nished and improved. An overall accuracy of 20.53% in not a good result and the system is in progress.
The goal of this bilingual task is to answer a set of questions, where there is a questions language (English) and a di®erent collection language (Spanish), so it is necessary to translate the set of questions.
We present the BRUJA system (in Spanish, Bu¶squeda de Respuestas University of JA¶en),
a prototype of a complete multilingual QA system, based on NLP tools [
We have combine di®erent modules, in order to evaluate the system in di®erent points, that are explained in the following sections.
Next Section describes the system architecture and some details of each module. In Section 3 we explain the main experiments and the results. Finally, conclusions and further works are presented in Section 4. 2
2.1
The BRUJA system is a prototype of a complete multilingual Question Answering system, that works with questions into Spanish, English or French, and the collection datasets are also into these three languages.
Basically, when a new question arise it is translated to the other languages and the original questions and its translations are launched over its collection index. Then it is necessary to merge the monolingual lists of relevant documents or passages and one multilingual relevant list is returned, like an usual Cross Language Information Retrieval system (CLIR).
In some steps we use English as the pivot language, and we apply online machine translators if it is necessary.
In the ¯gure 1 we can see the architecture of the BRUJA system.
In the following sections we describe each module in detail. 2.2
When a new question arise we detect its language, and if it is di®erent from English we translated it to English. We use SYMTRAM (our Machine Translation system that works with di®erent online machine translators and implements some heuristics).
We do that because our preprocessing methods work with English questions, in order to improve the result of this phase. We make the preprocessing phase using the GATE architecture (REF).
For the main process and for the next one, the Question Classi¯cation, some lexical, syntatic and semantic features and the keywords are extracted.
After that, we run our Question Classi¯cation (QC) subsystem, that classify the question in
a general class (ABBR, DESC, ENTY, HUM, LOC or NUM). Other experiments give us a high
con¯dence in its module, with results around a 90% of F-score [
As data of this ¯rst phase we obtain relevant features, such as the focus of the questions, and the general class of the question.
In the Information Retrieval subsystem we ¯rst preprocess the collections, using also GATE, and then we index these collections. In order to improve the results of this step we index documents using LEMUR (REF) and passages using the IR-n (REF) system developed at the University of Alicante.
After that we combine the results with a simple voting system. We sum the score of each individual relevant docid.
The idea of the multilingual system is that it would have to join the lists of relevant passages from the di®erent languages, to obtain one multilingual list. Because the merging module has not ¯nished yet we have only worked with the Spanish list. 2.4
Before to extract the answers we try to improve the passage selection. By default we take only the 10 ¯rst passages for each question. The idea is to decrease the time consumption of the general system.
In this phase we apply some heuristics that depends of the question class.
For instance if the class of a question is LOC we expect that the answer is a location. In this case we take the ten ¯rst passages that contains any location. To do that we apply some Named Entity Recognition methods.
Finally, for the answers ¯le, if the score is below 0.5, we consider that it is an incorrect answer, and NIL is written. 3
This section describes the result obtained with the simple run sent and the evaluation. The proposed system was applied to the set of 200 questions, although only factual questions and some de¯nitional have been used.
Table 1 shows the results for our run.
The results shown that the answer extraction module don't work properly, and only a low percent of factual questions have good answers.
For Factoid questions a simple manual analysis of the experiment gave us some reasons about the results obtained.
² Some questions have not real relevant passages. This happen, for instance, when the focus words have not identi¯ed well, and the others keywords appear in the relevant passage. ² In the cases where relevant passages contains the possible answer we count on the goodness of the Named Entity Recognition system, and sometimes it fails or didn't recognize the entities.
For De¯nition questions the same manual analysis give us some reasons. The main one is that there are a lot of patterns, and the use of only some of them is not enough. 4
For our ¯rst participation in QA@CLEF track we proposed a prototype of a multilingual QA system, that works with English and Spanish questions, to search Spanish relevant documents.
For this prototype only the answers extraction was in its ¯rst phase, so the results are obviously not good, but with the experimentation made we will evaluate the multilingual system in di®erent points.
As future work we will ¯nish and tuning each module. The next important task is to evaluate the experiments made this year and develop the answer extraction module based on logic and machine learning. 5