This paper describes IHARDETSI, a question answering system for Basque. We present the results of our rst participation in the QA@CLEF 2008 evaluation task. We participated in three subtasks using Basque, English and Spanish as source languages and Basque as target language. We approached the Spanish-Basque and EnglishBasque cross-lingual tasks with a machine translation system that process a question in the source language (i.e. Spanish, English), translates into the target language (i.e. Basque) and, nally, the obtained Basque question is sent to Ihardetsi system. We submitted four runs, one for Basque-Basque subtask, one for English-Basque subtask and two for Spanish-Basque subtask.
we use a Spanish-Basque and English-Basque machine translation systems [
This paper is structured as follows. The next section presents the corpus processing. Section 3 describes the system architecture. Section 4 introduces the results and a preliminary analysis of the kind of errors that the system made. Conclusions and directions of future work to solve main problems follow in section 5. 2
This year's document collection consists of two di erent collections: a dump of the Wikipedia 2006 articles and Egunkaria newswire collection from 2000 until 2002.
The document collection has been lemmatized before indexing it. Due to Basque is an agglutinative language, a given lemma makes many di erent word forms, depending on the case (genitive, locative, etc.) or the number (singular, plural, inde nite) for nouns and adjectives, and the person (me, he, etc.) and the tense (present, past, etc.) for verbs. For example, the lemma lan ("work") forms the in ections lana ("the work"), lanak ("works" or "the works"), lanari ("to the work"), lanei ("to the works"), lanaren ("of the work"), lanen ("of the works"), etc. This means that looking only for the exact word given or the word plus an "s" for the plural is not enough for Basque. And the use of wildcards, which some search engines allow, is not an adequate solution, as these can return occurrences of not only conjugations or in ections of the word, but also derivatives, unrelated words, etc. For example, looking for lan* would also return all the forms of the words lanabes ("tool"), lanbide ("job"), lanbro ("fog"), and many more.
Before the Wikipedia was analysed it needed to be parsed to clean the text, getting out html tags. So, we created a XML parser that extracts page title, paragraphs, and lists creating a simple XML document which is very similar to the XML of the newspaper collection.
The entire document collection was lemmatized, part-of-speech tagged and named entity recognised. The named entity recogniser for Basque captures entities such as PERSON, ORGANIZATION, LOCATION and the numerical and temporal expressions are captured by the lemmatizer/tagger.
And nally, the document collection was indexed by lemma using Swish-e search engine and the retrieval unit is the passage. 3
The system relies on NLP tools, which perform a linguistic analysis, both on the question and on the corpus.
A XML con guration le governs the running of these components. The con guration le is a declarative document where all the features involved in a run are described. The set of features is divided into two categories: 1. General requirements. It includes speci cations such as the corpus to be used, the location of the list of questions to be answered, and the metrics and conditions for the evaluation. 2. Descriptors of the QA process itself. This subset of features represents the characteristics of the answering process. Mainly, it determines which modules act during the answering process, describes them and speci es the parameters of each module. In that way, the process is controlled by means of the con guration le, and di erent processing options, techniques, and resources can be easily activated/deactivated and adapted. These descriptors constitute the documentation support of the system.
The principles of versatility and adaptability have guided the development of the system. The system is based on web services, integrated using SOAP communication protocol. Some tools previously developed in the IXA group are used as autonomous web services, and the QA system becomes a client that calls these services when it needs them. This distributed model allows to parameterize the linguistic tools, and to adjust the behaviour of the system during the development and testing phases.
Morfeus
Eihera BasqueWN
EU questions
Question type
Expected answer type Topic related information
EN questions EN-EU MT
EU questions
ES questions ES-EU MT
EU questions
Question Analysis procNeLsPsing claQsuseifsictiaotnion Query terms extraction and expansion Query terms
Passage Retrieval Query generation Passages Passage retieval Answer Extraction Candidate extraction Answer selection Answer Wikipedia PARSER Wikipedia parsed Egunkaria
NLP Processing Wikipedia procesed Egunkaria processed Index Creation Wikipedia + Egunkaria index
The communication between the web services is done using XML documents. This model has
been adopted by other systems ([
A Machine Translation engine named Matxin [
In the English to Basque translation we have used an early version of the English to Basque engine based on the same technology. The quality was poor and in a similar shallow test with factoid questions we detected that the translation of some question types were wrong, specially when the question marker was composed of two words that appeared as non-contiguous (i.e. Where is he from? ). To face this problem a heuristic was applied after the translation process in order to repair bad translations of question markers. The heuristic was implemented using a few number of conditional rules, which work on the original and the translated sentences.
In the near future we want to evaluate the quality of the translation of questions and to improve it, using if it is possible a corpus-based approach. 3.2
The main goal of this module is to analyse the question and to generate the information needed for the next tasks. Concretely, a set of search terms are extracted for the passage retrieval module, the question type (factoid, list or de nition) and the expected answer type along with some lexical information is passed to the answer extraction module. To achieve this goal, our question analyser performs the following steps:
Linguistic processing: The question analysis uses a set of general purpose tools like the
morphological analyser, Morfeus [
Question classi cation: For identifying the question type, the question focus and the expected answer type, a set of rules has been de ned after the examination of a Basque question set.
The question focus is the word or the word sequence that de nes or disambiguates the question, i.e. it pinpoints what the question is searching for or what it is about. For example in the question Which river is in the south of this country?, the focus is river and in question What is the North Pole?, the focus is North Pole.
Next step is to identify the expected answer type. Our system's answer type taxonomy distinguishes the following classes: PERSON, ORGANIZATION, DESCRIPTION, LOCATION, QUANTITY, TEMPORAL, ENTITY, and OTHER. The assignment of a class to analysed question is performed using the question stem, the syntactic construction and the type of the question focus. The question focus type is used to detect the expected answer type using BasqueWN1 semantic le to the categories PERSON / ORGANIZATION / LOCATION / QUANTITY / TEMPORAL.
Query terms extraction and expansion: All nouns, verbs, adjectives and abbreviations of the question constitute the set of search terms. They are lemmatized and arranged in descending order by their Inverse Document Frequency (IDF) value in the corpora. Optionally, the search terms can be expanded using synonymy, hyponymy and hypernymy information. To do this, the system uses a service which consults the lexical-semantic database BasqueWN. 3.3
The retrieval unit is a passage and not the entire document. The corpus is indexed by lemma using swish-e2 search engine. The corpus is batch-processed (see section 2): all words are lemmatized, and complex lexical units and entities are marked.
This module takes as input:
1. the search terms selected by the question analysis module
2. the search terms selected by the question analysis module for the rst question of a topic (if
the question is not the rst)
3. the rst three answers of the rst question of a topic (if the question is not the rst)
1It is the Basque version of EuroWordNet. This resource is integrated in the Multilingual Central Repository
(MCR), which is a multilingual lexical database developed in the Meaning project [
2http://swish-e.org
and produces a set of queries. For each group a set of queries are created using relaxation
techniques [
Two tasks are performed in sequence: Candidate Extraction and Answer Selection. The candidate extraction consists of extracting all the candidate answers from the highest scoring passages. The answer selection consists of choosing the best three answers.
Candidate Extraction. The process is carried out on the set of passages obtained in the previous step. First, all candidate answers are detected from each retrieved passage and a set of windows are de ned around them. The selected window for each candidate answer is the smaller one which has all the query terms, or taxonomically related terms, in. Then, the candidate answer score is computed like this: scoreCA =
Pn i=1 wi n (1) where n is the window size and wi is the i word weight. wi is 1 for search terms, 0.8 for the synonyms of the search terms, 0.5 for hyponyms and hypernyms, and 0.3 for other question terms.
The candidate answers extraction process then addresses each question type in a di erent manner, as follows: { Question type is Factoid: the answer selection depends on entities in the most of the cases except when the expected answer type is Entity or Other. For Entity and Other types we select all the entities and nouns near the question focus. Although the numerical and temporal expressions are marked in the processed corpus (see section 2), a grammar has been applied to mark even more. { Question type is De nition: a set of rules have been de ned to extract de nition from retrieved text passages. { Question type is List: an attempt has been done similar to Factoid questions but it was asked to be a list of answers being in the same sentence.
Answer Selection. In order to select the best answers from the set of candidates, the same answers that appear in di erent passages must be combined. We try to map as identical those answers that refer to the same entity. The formula used to compute the nal score of each answer is as follows:
Pp f inal scoreCA = i=1 wi
N where p is the number of identical answers and N is the number of candidate answers. (2) 4
This section describes the results we obtained in our CLEF-2008 participation. We submitted four runs, one for Basque to Basque monolingual QA task, one for English to Basque cross-lingual QA task, and two runs for Spanish to Basque cross-lingual QA task. The methodology we employed targeted precision at the cost of recall, therefore we always choose NIL answers for those questions we could not reliably locate a candidate answer in the retrieved passage.
RIGHT WRONG
INEXACT UNSUPPORTED
TOTAL ACCURACY
OVERALL 26 163 11 0 200 13%
FACTOID 23 113 9 0 145 15.862% At it was expected the best results were obtained for the monolingual task. Table 1 illustrates the results achieved by our system in the monolingual run.
It is clear that the best results were achieved for factoid questions. It is due to the fact that we focused on this type of questions in the development of the system. There were 145 factoid questions and 50 had a correct or inexact answer in the proposed three answers, 22 had a NIL answer (incorrect) and 73 had an incorrect answer. Analysing these 73 questions we detected that for 17 the correct passage was detected but the system did not extract the correct answer.
The system answered NIL for 57 questions but only 4 of them were correct. Analysing the reasons for this we can group them in 5 groups:
The expected answer type detection failed: 6 questions.
No passage was retrieved: 14 questions The passage had the answer but the system could not extract the answer: 13 question Retrieved passage had not the answer: 16 questions
Some other reasons: 4 questions
After an analysis of the results of the NIL questions, we realized that some questions did not get any documents due to a bug in the system. Once it was corrected we performed a new run for Basque questions; 10 more questions were answered (9 DEFINITION questions and 1 FACTOID question) and for 8 questions the answer was changed (losing two correct answers). The answered new factoid question (i.e. Where is Ocotal? ) was answered correctly and for the nine de nitional questions 4 were answered correctly and 2 more had the correct answer in the second place. 4.2
Three cross-lingual runs, two for Spanish-Basque and one for English-Basque, have been performed. The aim of the second run for Spanish-Basque was to test if the semantic expansion (see 3.2 section) of the question could compensate the lost of precision in the translation process.
The results of the three runs are shown in Table 2.
OVERALL
FACTOID DEFINITION
LIST TEMPORAL RESTRICTED
The results are quite poor. The loss of precision respect to the monolingual system is more than 50%.
Very similar results are obtained for the basic Spanish-Basque and for the English-Spanish runs (in both there are 11 right answers, 7 right answers in 2nd or 3rd place and 7 inexact in the rst place). Due to better quality of the Spanish-Basque translator we hoped better results for this run. Anyway, it is necessary a wider evaluation of each MT engine when translating questions.
Although the results are similar in average, the right results do not correspond always to the same questions. Only ve of the eleven right answer are common.
The semantic expansion in the second run for Spanish-Basque do not achieve better results. A slight smaller precision is observed, because some right answer are lost. In compensation to this, new right or inexact answer appear but not in the rst place. With this gures we can think that at least a higher number of \passages\ are recovered, but it is not true, because the number of recovered \passages\ remains at same level (about 40 of 200). 5
The development stage of our monolingual Basque to Basque QA system has been described in this paper, as well as our participation in the QA@CLEF campaign. Thanks to this track we have had the opportunity of testing our system. Although the results might look no good, our general conclusion is very positive taking into account that it was our rst participation. We can not directly compare our system results with the results of other languages systems due to the particularities of Basque language. However, we have been able to extract some of the strengths and weakness of each module of the system, which we will take into account for future improvements. Besides we study the possibility of adding a fourth module to deal with topicrelated questions. 6
We would like to thank Gorka Labaka and Mikel Lersundi who have collaborated in this research. Arantxa Otegi's work is funded by a PhD grant from the Basque Government. Part of this work has been funded by the Basque Government (Department of Language Policy and Department of Industry Anhitz project, Etortek IE06-185) and the Education Ministry (KNOW, TIN200615049-C03-01).