This paper presents a question answering (QA) system called Tikka. Tikka's approach to QA relies heavily on the semantic annotation of text documents and on the usage of answer extraction patterns. In this way, Tikka applies to QA pattern-based techniques traditionally used in named entity recognition and information extraction. In the experiments presented in this paper, Tikka's performance is evaluated in the following tasks: monolingual Finnish and French and bilingual Finnish-English QA. Its performance in the monolingual tasks is near the average when it is compared with the QA systems' performance that participated in the monolingual French task. In the monolingual Finnish task, Tikka was the only participating system.
GAZETTEER CLASSIFI− CATION
RULES EXTRACTION PATTERNS BILINGUAL DICTIONARY SYNTACTIC PARSER
SEMANTIC ANNOTATOR QUESTION
CLASSIFIER TOPIC AND TAR− GET EXTRACTOR TRANSLATOR QUERY TERMS AND QUESTION CLASS, TOPIC AND TARGET
TEXT DOCUMENTS GAZETTEER PATTERN PROTOTYPES DOCUMENT RETRIEVER PARAGRAPH SELECTOR SEMANTIC ANNOTATOR
PATTERN INSTANTIATOR AND MATCHER ANSWER SELECTOR
The question analysis component of the QA system consists of five software modules: 1) the syntactic parser for Finnish, 2) the semantic annotator, which is detailed in Section 4, 3) the question classifier, 4) the topic and target extractor and 5) the translator, which is described in the system description of the previous version of Tikka, that participated in QA@CLEF 2004 [1]. All these modules, along with the databases that they use, are illustrated in Figure 1.
Table 1 shows through an example how question analysis is performed. First, a natural language question is given as input to the system, for example: D FI EN Mika¨ on WWF? 1. Next, the Finnish question is parsed syntactically and the French question is annotated semantically. Then both questions are classified according to the expected answer type, and the topic and target words are extracted from them. The expected answer types are determined by the multinine corpus, and they are: LOCATION, MEASURE, ORGANIZATION, OTHER, PERSON and TIME. The target words are extracted or inferred from the question and they further restrict the answer type, e.g. age, kilometers and capital city[2].The topic words are words extracted from the question that in a sentence containing the answer to the question carry old information. For example, in the question What is WWF?, WWF is the topic because in the answer sentence WWF is the World Wide Fund for Nature., WWF is the old information and the World Wide Fund for Nature is the new information. The old and new information of a sentence are contextually established [9]. In our case, the question is the context. In Tikka, topic words are useful query terms along with the target words, and they are also used to fill slots in the answer pattern prototypes.
1D stands for a definition question and FI EN means that the source language is Finnish and the target language
is English. In English, the question means What is WWF?
(
(
(
D FI EN Mik¨a D FI FI Mik¨a on WWF? on WWF ? 1 Mik¨a mik¨a subj:>2 &NH PRON SG NOM 2 on olla main:>0 &+MV V ACT IND PRES SG3 3 WWF wwf &NH N
N/A
D FR FR Qu’est-ce que la WWF? N/A Qu’est-ce que <organization> la WWF</organization>? Organization Topic: WWF Target: N/A
N/A The answer extraction component consists of five software modules: 1) the document retriever, 2) the paragraph selector, 3) the semantic annotator, 4) the pattern instantiator and matcher and 5) the answer selector. All these modules, along with the databases that they use, are illustrated in Figure 1. The dotted arrows that go from the document retriever back to itself as well as from the answer selector back to the document retriever illustrate that if no documents or answers are found, answer extraction starts all over. 3.1
An Example
(
(
22 docs retrieved 22 docs inspected 70 paragraphs selected
Query terms: WWF, Topic: WWF, Target: N/A 76 docs retrieved, 313 docs retrieved, 30 docs inspected 10 docs inspected 99 paragraphs 39 paragraphs selected selected
See Table 4 0 patterns 0 matches nothing to choose from 0 NIL 12 instantiated patterns match 7 different answers chooses the answer with the highest score, 18 0.25 AAMU19950818-000016 Maailman Luonnon S¨a¨ati¨o 18 instantiated patterns match 4 different answers chooses the answer with the highest score, 8 0.75 ATS.940527.0086 le Fonds mondial pour la nature ((<[a-z]+>[^<>]+<\/[a-z]+> )+)\( (<[a-z]+>)?TOPIC(<\/[a-z]+>)? \)Score:9 ((<[a-z]+>[^<>]+<\/[a-z]+> )+)\( (<[a-z]+>)?Wwf(<\/[a-z]+>)? \)Score:9
The text snippet that matched the above pattern is in Table 4. (The patterns are case insensitive.) Only at least partly semantically annotated candidates can be extracted. The score of a unique answer candidate is the sum of the scores of the patterns that extracted the similar answer instances, or more formally: score(answer) =
X patternScore(i),
i in A
(
If document retrieval does not return any documents, or no answer is extracted from the paragraphs, Tikka has several alternative ways in which to proceed, depending on which task it is performing and how many times document retrieval has been tried. This is illustrated in Table 5. As can be seen from the figure, if no documents are retrieved in the first iteration, the parameter settings of the retrieval engine are altered and document retrieval is performed again in the monolingual Finnish and bilingual Finnish-English tasks. However, in the monolingual French task the system halts and returns NIL with a confidence of 1 as an answer. In the monolingual Finnish task, the system halts after the second try, but in the bilingual English-Finnish task, document retrieval is performed for a third time if either no documents are retrieved or no answer is found. Alternatively, in the monolingual Finnish and English tasks, if documents are retrieved, but no answers are found after the first try, document retrieval is tried once more. In all tasks, the system returns a confidence value of 1 for the NIL answer if no documents are found and a confidence value of 0 for the NIL answer if documents are found but no answer can be extracted. 3.2
Document Retrieval The document retrieval module of Tikka consists of the vector space model [10] based search engine Lucene 2 and of the document indices for English, Finnish and French newspaper text built using it. Tikka has one index for the English document collection, two indices for the Finnish document collection and two for the French document collection. In each of the indices, one newspaper article forms one document. The English index (enstem) is a stemmed one. It is stemmed using 2http://lucene.apache.org/java/docs/index.html 17 10 5 0
1 the implementation of Porter’s stemming algorithm [7] included in Lucene. One index (filemma) to the Finnish collection is created using the lemmatized word forms as index terms. The Connexor’s parser is used for the lemmatization. The other Finnish index (fistem) consists of stemmed word forms. The stemming is done by Snowball [8] project’s 3 stemming algorithm for Finnish. A Snowlball stemmer is also used to create one of the indices for French (frstem). The other French index (frbase) is built using the words of the documents as such. This index is case-insensitive.
In the document retrieval phase, Lucene determines the similarity between the query (q) and the document (d) using the formula presented in Equation 2 [4].
similarity(q, d) =
X tf (t in d) · idf (t) · boost(t.f ield in d) · lengthN orm(t.f ield in d), (
Lucene does not use the pure boolean information retrieval (IR) model, but we model the conjunctive boolean query by requiring all of the query terms to appear in each of the documents in the result set. This differs from the pure boolean IR model in that the relevance score for each document is calculated according to Equation 2, and the documents are ordered according to it. The ordering is important in Tikka. This is what the term boolean means in Figure 5. In the same figure, the term ranked means a normal Lucene query where all of the query words are not required to appear in the retrieved documents. 4
Semantic annotation is in many ways a similar task to named entity recognition (NER). NER is commonly done based on preset names lists and patterns [11] or using machine learning techniques [3]. Our method relies on the first method. The main difference between NER and semantic annotation is that the first one aims at recognizing proper names whereas the second aims at recognizing both proper names and common nouns.
In Tikka, French questions and selected paragraphs from the search results are annotated semantically. We have 14 semantic classes that are listed in table 3. To the classes consisting mainly of proper nouns, some common nouns are added in order to be able to analyze the questions correctly. For instance, in the gazetteers of the class organization, there are proper names denoting companies (IBM, Toyota), but also some common nouns referring to organizations in each language (school, bank, union). As can be seen from Table 3, the organization gazetteer in English is significantly shorter than those in other two languages. This is due to the NER from Connexor that is used in addition to our own semantic annotator.
Class person country language nationality capital location organization
The semantic annotator uses a window of two words for identifying the items to be annotated. In that way we can only find the entities consisting of one or two words. The external NER that is used in the English annotation is able to identify person names, organizations and locations. Hence, there are no limitations on the length of entities on these three classes in English. For Finnish, we exploit Connexor’s syntactic parser for part of speech recognition to eliminate the words that are not nouns, adjectives or numerals. For French, the semantic annotator builds solely on the text as it is without any linguistic analysis.
In the text to be annotated, persons are identified based on a list of first names and the subsequent capital word. The subsequent capital words are added to the list of known names in the document. In this way the family names appearing alone later in the document can also be identified to be names of a person. The class location consists of names of large cities that are not capitals and of the names of states and other larger geographical items. To the class measure belong numerals and numeric expressions, for instance dozen. Unit consists of terms such as percent, kilometer. The event class is quite heterogeneous, since to it belong terms like Olympics, Christmas, war and hurricane. Time gazetteer lists time related terms, the names of the months, week days etc. Example annotations for each of the languages can be seen in Table 4. 5
Tikka was evaluated by participating in the monolingual Finnish and French tasks and in the bilingual Finnish-English task. The evaluation results are described in detail in Section 4 (Results) of the QA track overview paper [6]. In each of the tasks, two different parameter settings (run 1 and run 2) for the document retrieval component were tested. These settings are listed in Table 5. The results of the runs are shown in Figure 3. We can observe that the difference between runs is not very big for the French monolingual run. The accuracy of the artificial combination run 4 (C) is not much higher than that of the the French monolingual run 1, which means that almost 4In this case, the artificial combination run represents a run where the system is somehow able to choose for each question the better answer from the two answers provided by the runs 1 and 2. For more information on the combination runs, see the track overview paper [6].
English Finnish French
The <organization>World Wide Fund for Nature</organization> ( <organization>WWF</organization> ) reported that only <measure>35,000</measure> to <measure>50,000</measure> of the species remained in mainly isolated pockets . <organization>Maailman Luonnon S¨a¨ati¨o</organization> ( <ne>WWF</ne> ) vetoaa kaikkiin <country>Suomen</country> metsstjiin , ett ei <person>Toivoa</person> ja sen perhett ammuttaisi niiden <unit>matkalla</unit> toistaiseksi tuntemattomille talvehtimisalueille . <ne>La</ne> dcision de <organization>la Commission</organization> baleini`ere internationale ( <ne>CBI</ne> ) de cr´eer un sanctuaire pour les c´etac´es est ”une victoire historique” , a comment´e <time>vendredi</time> <ne>le Fonds</ne> mondial pour la nature ( <organization>WWF</organization> ) all answers given by the runs are equal. On the contrary, there is a difference of 4 points between the accuracies of the two monolingual Finnish runs, and in addition, as the difference between run 1 and the combination run for Finnish is 3.5 points, we can conclude that some of the correct answers returned by run 2 are not included in the set of correct answers given by run 1. This means that the different parameter settings of the runs produced an effect on Tikka’s overall performance. Between the runs, both the parameters for type of index and the maximum number of documents were altered. In the bilingual Finnish-English task, some difference between the runs can be observed, but the difference is not as big as in he monolingual Finnish task.
Accuracy
20%
10%
26.5
C
23.0
1
Tikka is a QA system that uses pattern-based techniques to extract answers from text. In the
experiments presented in this paper, its performance is evaluated in the following tasks: monolingual
Finnish and French and bilingual Finnish-English QA. Its performance in the monolingual tasks is
near the average when it is compared with the QA systems’ that participated in the monolingual
French task. In the monolingual Finnish task, Tikka was the only participating system.
(
In the future, as the document databases are not very big (about 1.6 GB), the documents could be annotated semantically before indexing. This would speed up the interactive processing time and the semantic classes could be used as fields in index creation. In addition, indexing based on paragraph level instead of document level might raise the ranking of the essential results and it would speed up the processing time of the interactive phase. 7
The authors thank Connexor Ltd 5 for providing the Finnish and English parsers and NE reconizers, Kielikone Ltd 6 for providing the bilingual dictionary and Juha Makkonen for providing the question classifier for Finnish.
5http://www.connexor.com 6http://www.kielikone.fi/en
Available at