This article describes our rst participation at the QAst task of the CLEF campaign 2008. We submitted 4 experiments in total, two for each subtask t1 and t4. These subtasks employed manual speech transcription collections. We used the Stanford Named Entity Recognizer for tagging named entities and the CRFTagger - Conditional Random Fields Part-of-Speech (POS) Tagger for English. The passage retrieval was done with the Xtrieval framework and its Apache Lucene implementation. For the classi cation of the question hand-crafted patterns were implemented. Our experiments achieved an accuracy of about 20%. The rate of returned NIL answers was too high for all of our experiments.
This article describes the design, architecture and evaluation of a prototype extension of the Xtrieval
framework [
We developed our prototype with respect to the restrictions and assumptions of the task. The prototype includes all classical components of a QA system: (a) question classi cation, (b) passage retrieval, (c) answer extraction and (d) natural language processing (NLP) components for named entity recognition (NER) and a part-of-speech (POS) tagger. Since we are newcomers to the topic of QA, we decided to start with two simpler tasks (t1 and t4) on manual speech transcriptions. These collections allow a rather simpler strategy
Collection Preprocessing
NER-Tagger
POS-Tagger
Tagged Collection Named Entity-Extraction
Sentence Extraction
Noun-Extraction Collection Processing
Xtrieval Framework
Question Processing for the identi cation of the right answer to a factual or de nitional question, because no phonetic analysis is necessary.
The remainder of the article is organized as follows. In section 2 we describe our system and its architecture. Section 3 shows the results of our submitted experiments. A summary of the result analysis is given in section 4. The nal section concludes our experiments with respect to our expectations and gives and outlook to future work. 2
The main architecture of our prototype is illustrated in gure 1. We use the Xtrieval framework with Apache Lucene2 as core implementation to retrieve answer candidates from our passage index. Both questions and collections are preprocessed with two kinds of taggers. For the selection of the nal answer(s) we use: (a) the RSV's of the returned passages, (b) the class the current question belongs to and (c) the NE and POS tags from question and collection.
Candidates & Weights
Classification
Answer Selection Answers
Answer Extraction
Answer Ranking
searching the collection for speci c tags. Since many NLP systems exist in the community and also because
we are not experts in the NLP topic, we decided to compare di erent systems. We also investigated whether
the systems were adaptable to our special needs for the task. For a more robust QA system we thought it
might be useful to integrate both a NER and a POS tagger. The prerequisite for both systems was Java
compatibility, because our prototype and the Xtrieval framework are implemented in Java. We did not
bother when the support for Java was provided by a Java API. In the two following subsections we go into
the details of the tagging systems we applied in this work.
In our system we used the Stanford Named Entity Recognizer 3 (SNER) for named entity recognition. It is
used for fast extraction of answers to questions, that imply a certain type of named entity [
In this step the question is analyzed to decide whether the question is a factual or a de nitional one. Therefore we simply apply a hand-crafted pattern to the interrogative. If the question starts with what/who is/are we regard the question as de nitional question. Otherwise we assume the current question is a factual question. 2.3
The types of the named entities are also very important for the analysis of the question itself. The question could be classi ed by using the 10 types of entities de ned in the task guideline. Due to that fact, we try to assign at least one type of entities to the question by using hand-crafted patterns that contain special words. Again to give an easy example one might imagine that a question starting with how many/much/long/old is most likely to be answered with a measure. We de ned a number of these patterns to help estimating what 3http://nlp.stanford.edu/software/CRF-NER.shtml 4http://crftagger.sourceforge.net/ 5http://nlp.stanford.edu/software/tagger.shtml might be a right answer to the question. There are a number of easy types of entities in the group of the 10 entities proposed in the guideline, like numbers or locations. But there are also a number of hard entities like color and shape. When our system is not able to assign a single class to a question, it assigns a number of classes. 2.4
We separated the test collection into phrase for the retrieval stage. For the separation of the collection we simply use punctuation marks and all sentences were indexed with the Xtrieval framework. We assume this is a good solution because if the passages are larger than it might be much harder to extract the correct answer. Contrary, when we had used smaller passages it may happen that a correct answer is split into two di erent passages. 2.5
The answer nding procedure mainly consists of four important step. At rst a query is formulated and feed into the retrieval system. The retrieval system ranks the passages based on their RSV and returns the top passages. Thereafter, the best matching passage is selected and the corresponding answer will be extracted. In the next subsections we go into the details of these four steps. 2.5.1
At rst, we need to formulate a query to the IR system. The query has to be created depending on the type and the content of the question. Therefore, the system extracts nouns with the help of the POS tagger and assigns weights to the di erent types of part of speech, e.g. proper names/nouns (NNP/NNPS), which have to occur in a corresponding passage or a standard noun (NN/NNS) that could occur in the relevant passage. The system automatically forms phrase queries for neighboring nouns. 2.5.2
The ranking of the passages is de ned by the RSV for the passage returned from the IR system. There is only one exception: when the second method for the answer extraction is used (see corresponding section below) the score will be the inverse of the calculated distance value.
The formulated query is fed into the IR system, which queries the passage index. If this step does not return any relevant passages, a fall-back algorithm will be applied. This method simpli es the query by removing possible phrases or changing a mandatory term into an optional term. As a matter of principle we do only consider the rst passage returned by the IR system, because we observed that in the test set the best matching passage contained the correct answer in many cases. This could help us to achieve a high precision, which is very important in a QA scenario from our point of view. 2.5.4
The module for the answer extraction consists of two parts. At rst, we try to nd corresponding answers for special classes of questions by using the POS or NER tags. For example, if the question asks for values or measures, the system will search for adjectives and tries to combine groups of words to build a complete answer like more than ve hundred. If a posed question contains a person, organization or location, the system will use the terms that were tagged by the SNER tagger. Unfortunately, we discovered that quite a large number of entities are tagged wrong, i.e. they are tagged with the wrong class. For example persons can be recognized as organizations and vice versa. Therefore we think we should use an NER tagger that was adapted or trained with the collections we use for the experiments.
If the rst part of the answer extraction did not return any answers or the question was not assigned to a single type a fall-back strategy will be applied. We analyzed some experimental answers on the development data and we observed that the correct answers to some questions are near the terms that occurred in the question. Therefore, we implemented an algorithm to take advantage of this observation. It calculates the distances between all nouns in a sentence that do also occur in the query. We use this measure for the fall-back and return the noun with the lowest overall distance to all query terms. This calculation was only used when the best matching passage returned by the IR system had a RSV that was higher than a certain threshold. This threshold is necessary because otherwise the system would return an answer for every question even for those that are not supported by the collection. 3
The general setup of the system was discussed in the preceding sections. We submitted 4 experiments in total. For both collections (CHIL and EPPS) we tested a con guration (cut1*) that returned only one answer per question. We also submitted another con guration (cut2*), where up to 3 answers per questions were returned. Tables 2 and 3 present the results of the evaluation.
run ID cut1 t1a cut2 t1a cut1 t4a cut2 t4a In table 2 we have the following values: (Q) the total number of question that were processed, (CA) the total count of correct answers, (MRR) the mean reciprocal rank and (ACC) the overall accuracy of the system con guration. The results show that the proposed prototype did only answer a fth of all questions correctly. Interestingly, the accuracy of the rst answer is very high. Another observation can be made by looking at the results for the two di erent collections. We found a higher number of correct answers for the considerably smaller EPPS collection, which might be due to the larger ratio of recognized named entities. But another reason could also be the distribution of the question types over both collections.
run ID cut1 t1a cut2 t1a cut1 t4a cut2 t4a
The following list provides a summary of the analysis of our retrieval experiments for the QAst task at CLEF 2008:
Task 1a (CHIL-manual): Both experimental con gurations of the system achieved an accuracy slightly below 20%. Interestingly, this accuracy was achieved by generating only one answer per question (experiments cut1*). The total number of NIL answers is above 50%, which is an almost inacceptable value.
Task 4a (EPPS-manual): Both experimental con gurations of the system achieved an accuracy slightly above 20%. The total number of NIL answers is above 40%, which is also an almost inacceptable value. 5
The experiments for the CLEF 2008 QAst task allow us to draw some conclusions on how to improve the quality of the used system. At rst, we should improve the ratio of correctly recognized named entities either by training the tagger for the special collection or by using a more general tagger. With this improvement we could probably assign more correct classes to the questions posed, which will boost the total number of correct answers without deteriorating the accuracy of the system. Additionally, we could improve or vary the passage extraction procedure to improve the passage retrieval itself. Last but not least one could try to implement an answer extraction strategy that is adapted to nd more answers to raise the total number of answered question. This could easily be achieved by tuning the RSV based parameter in the answer extraction fall-back strategy to lower values.