the characteristics of this evaluation changed signican tly. Participants were proThis campaign, the QA@CLEF-2004 proposed new diÆculties and therefore, nized the rst Multiple Language Question Answering task (QA@CLEF-2003) get document collection. This way the evaluation proposed from monolingual guided to the evaluation of QA systems in several languages. This evaluation Our participation was restricted to the Spanish monolingual task. The novCross-Language Evaluation Forum (CLEF) are characterized for Campaigns1 For each language, the organisation provided 200 questions requiring factual elty in the experiments developed was the use of documents in languages dieren t vided with document collections and question sets in seven European languages: spective of European languages integration. Particularly last year, CLEF orgaParticipants had to choose a language for questions and another for the tartasks (when question and document languages were the same) to dieren t comfostering investigation in multilingual information access systems from the perSpanish, Portuguese, Italian, Dutch, German, French and English. binations of bilingual QA (when selected languages were dieren t). the development and evaluation of QA systems from a multilingual perspective. was very important since it fostered the development of a series of resources for collection. Systems should return only one response per question. or denition answ ers whose answer was not guaranteed to occur in the document and (2) the possibility of using Web documents in other languages to support monolingual Spanish QA.
As this system is described in detail in [3] we only present here their main hancements have been added: (1) the inclusion of a dictionary-based NE tagger characteristics and enhance the new modules included. Our system is organized Our system is based on the QA system described in [3] where two main eninto the following main modules: extract the nal answ er. Figure 1 shows system architecture. detect and extract the useful information they contain. Passage retrieval modQuestion analysis processes questions formulated to the system in order to ule retrieves relevant passages from the Spanish EFE document collection and also from the Internet in the selected language (Spanish or English). Finally, the answer selection module processes relevant passages in order to locate and and keyword selection. The former detects the type of information that the quesoped set of lexical patterns. Answer types have increased and now the system tion expects as answer (a date, a quantity, etc) and the latter selects those quescontain the answer. These processes are performed by using a manually develtion terms (keywords) that will allow locating the documents that are likely to currently copes with seven possible answer types: NUMBER, DATE, LOCATION, PERSON, ORGANIZATION, DEFINITION and OTHER.
Question analysis module carries out two processes: answer type classic ation dieren t perspectives: (1) using Web Spanish documents and (2) using English Web documents to support monolingual Spanish QA. This way we could be able support monolingual QA. from Spanish in order to obtain evidences for supporting answers obtained from This paper is organised as follows: Section 2 describes the main characteristics of our QA system. Afterwards, we present and analyse the results obtained sions and discuss directions for future work. at QA@CLEF-2004 Spanish monolingual task. Finally, we extract initial concluCLEF Spanish corpora. Particularly, we performed monolingual task from two to investigate on using English (or by extent, other languages) documents to 3 http://www.systransoft.com/ 2 http://www.google.com/ from the EFE document database.
In parallel, the same keyword list (without being lemmatised) is translated to been translated by using online translation services. SysTran3 their corresponding lemmas are used for retrieving the 50 most relevant passages the language the system is required to use for Web search (in this case Spanish or question analysis stage are processed using MACO Spanish lemmatiser [1] and English) and posed to Google Internet search engine. The system selects the 50 entire Spanish EFE document collection. In this case, keywords detected at engines: IR-n [2] and IR-n system performs passage retrieval over the Google2.
Passage retrieval stage is accomplished in parallel using two dieren t search best short summaries returned in Google main retrieval pages. Keywords have
Answers this purpose (aliv041eses). Nevertheless, performance dierences are near inResult analysis shows that evidence obtained through English Internet docsignican t (32.5% { 31.5%). These results contradicted our initial hypotheses document processing: several translation problems that seriously aected the process of English W eb uments (aliv042eses) performs better than using Spanish Web documents for nican tly Spanish monolingual QA. After a shallow error analysis we detected since we thought that English web documents would probably help more sig3 Results Table 1. Spanish monolingual task results All these translation problems aect passage retriev al and answer extraction stages. First, an incorrect translation of content words in questions supposes the aliv042eses 31.11 45.00 32.50 Accuracy (%) aliv041eses 30.56 40.00 31.50 Run Factoid Denition Overall to candidate list obtained from the Spanish collection. This process is explained in detail in [3] Table 1 shows the results obtained for each run. tem described above and using Spanish Web retrieved documents while second run performed QA process by activating English Web retrieval (aliv042eses).
We submitted two runs. First run (aliv041eses) was obtained applying the sys{ Titles translation. Titles, such as names of books or lms should not be Usually, proper nouns referring to people or companies have no translation { Proper noun translation. Proper noun translation is an unresolved problem. { Abbreviation translation. Abbreviations usually refer to language-dependent be excluded from translation processes. refer to in the original language. question. viations and acronyms we need to know the whole expression or terms they or cities (Londres vs. London) dier depending on the language. glish documents that have no semantic relation with the original Spanish translated. The basic problem here resides in detecting these expressions to { Keyword translation. The lack of context when translating question keywords produces non-adequate translations. This implies sometimes retrieving Enexpressions. From this point of view, if we want to correctly translate abbre(eg. Bill Clinton). On the other hand, names of countries (Espan~a vs. Spain) monolingual QA is possible and worthwhile if we are able to solve correctly the be investigated will be directed to adopt translation techniques that minimize evidences obtained form corpora in dieren t languages, in this case, English. the currently detected errors. multilingual question answering.
Moreover we argue that surely, this problem is the main bottleneck towards translation problems described before. Consequently, main line of future work to This work is a rst attempt to perform monolingual QA in Spanish by using As we have previously seen, using corpora in other languages to support the main long-term objective of developing a whole system capable of performing This work has been partially supported by the Spanish Government (CICYT) with grant TIC2003-07158-C04-01. retrieval of useless documents that do not support the original question. And supporting candidate answer selection if proper nouns, abbreviations an titles are not correctly translated. second, it makes impossible to take advantage of evidences in other languages for