This paper describes the participation of the Technical University of Catalonia in the CLEF 2009 Question Answering on Speech Transcripts track. We have participated in the English and Spanish scenarios of QAst. For both manual and automatic transcripts we have used a robust factual Question Answering that uses minimal syntactic information. We have also developed a NERC designed to handle automatic transcripts. We perform a detailed analysis of our results and draw conclusions relating QA performance to word error rate and the di erence between written and spoken questions.
The CLEF 2009 Question Answering on Speech Transcripts (QAst) track [
24.1%. For T2 WERs are 11.5%, 12.7% and 13.7%. Figure 1 shows a text sample extracted from T1 corpus.
This paper summarizes our methods and results in QAst. We have participated in scenarios
T1 and T2 with all transcripts and question sets. Our QA system is based on our previous work
in [
The architecture of our QA system follows a commonly-used schema which splits the process into three phases performed in a sequential pipeline: Question Processing (QP), Passage Retrieval (PR), and Answer Extraction (AE) This QA system is designed to answer to factoid questions, those whose answer is a named entity (NE). 2.1
The main goal of this component is to detect the type of the expected answer. We currently
recognize the 53 open-domain answer types from [
For scenario T2, he have developed an Spanish question classi er using human translated
questions from the corpus of [
This component retrieves a set of relevant passages from the document collection, given the
previously extracted question keywords. The PR algorithm uses a query relaxation procedure that
iteratively adjusts the number of keywords used for retrieval and their proximity until the quality
of the recovered information is satisfactory (see [
When dealing with automatic transcripts, the incorrectly transcribed words may create a problem of word recognition to the IR engine, introducing false positives and false negatives to its input.
To overcome such drawbacks, we have used an IR engine relying on phonetic similarity for the
automatic transcripts. This tool is called PHAST (after PHonetic Alignment Search Tool) and
uses pattern matching algorithms to search for small sequences of phones (the keywords) into a
larger sequence (the documents) using a measure of sound similarity. Then the PR algorithm may
be applied to the words found with PHAST. A detailed description of PHAST can be found in [
Set WER
m
a
b
c
Identi es the exact answer to the given question within the retrieved passages. First, answer
candidates are identi ed as the set of named entities (NEs) that occur in these passages and have
the same type as the answer type detected by QP. Then, these candidates are ranked using a
scoring function based on a set of heuristics that measure keyword distance and density [
As described before, we extract candidate answers from the NEs that occur in the passages retrieved by the PR component. We detail below the strategies used for NERC in both manual and automatic transcripts.
We have taken a machine learning approach to this problem. First we apply learning at word
level to identify NE candidates using a BIO tagging scheme. Then these candidates are classi ed
into NE categories. Each function is modeled with voted perceptron [
Our NERC uses a rich set of lexical and syntactic features which are standard to state-ofthe-art NERCs. This features include: words, lemmas, POS tags, word a xes, ags regarding presence of numerals and capitalization, use of gazetteers, and n-grams of this features within a certain window of words. New features specially designed for automatic transcripts have been added to the sets a, b and c. These features use phonetic transcription of the words: Pre xes and su xes of phones.
Phonetic similarity with words in the gazetteer.
A clustering of the transcriptions of the words has been done by grouping words with similar pronunciation. This clustering reduces the sparseness of the word{based features by mapping the words in several smaller subsets of di erent coarseness.
Features capturing the possibility of splitting or merging adjacent words due to ASR recognition errors.
The addition of this phonetic features improves the results by no more than 2 points of F =1 score in datasets a, b and c.
Given that all there is no speci c datasets for development, and we don't have more automatic transcripts of EPPS data, it is not possible to train our NERC in a dataset other than the test set. Therefore we have relabeled both corpora through a process of cross{validation. Both corpora have been randomly split in 5 segments, a NERC model has been learned for all subsets of 4 segments and the remaining segment has been labeled using this model. Thus we can train our NERC with documents from the same domain but test it on unseen data.
T1B-a 2 T1A-b 2 T1B-b 2 T1A-c 2 T1B-c 2
T2B-a 2 T2A-b 2 T2B-b 2 T2A-c 2 T2B-c 2 MRR 0.29 0.33 0.20 0.24 0.20 0.20 0.20 0.21 UPC participated in 2 of the 3 scenarios, English (T1) and Spanish (T2) ones. We submitted two runs for each task, run number 1 uses the standard NERC described in Section 3 and run number 2 uses the hand{annotated NEs. Each scenario included 100 test questions, from which 20 do not have an answer in the corpora (these are nil questions). In T1 75 question are factoids for 44 in T2. Our QA system is designed to answer only factual questions, therefore the our experimental analysis will refer only to factual questions.
We report two measures: (a) TOPk, which assigns to a question a score of 1 only if the system
provided a correct answer in the top k returned; and (b) Mean Reciprocal Rank (MRR), which
assigns to a question a score of 1=k, where k is the position of the correct answer, or 0 if no
correct answer is found. The o cial evaluation of QAst 2008 uses TOP1 and TOP5 measures [
Tables 2 and 3 summarizes our overall results for factual questions in English and Spanish. It shows MRR, TOP1 and TOP5 scores for each track and run as de ned previously.
Table 4 contains a statistical error analysis of our system covering the QC, PR and AE parts. It deals only with factoid questions with non{nil answers. The meaning of each column is the following. Q: number of factual question. QC: number of questions with answer type correctly detected by QP. PR: number of question where at least on passage with the correct answer war retrieved. QC&PR: number of questions with correct answer type and correct passage retrieval. 1Consider manual transcripts mhaving a WER of almost 0.
T1A-m T1A-a T1A-b T1A-c
T1B-c
T1A-c
T1B-c
T2A-c
T2B-c
T2A-c
T2B-c
C.NE: number of questions where the retrieved passages contain the correct answer tagged as a NE of the right type (speci ed by the QC module), so it is a candidate answer for the AE module. TOP5 non-nil: number of question with non-nil answer correctly answered by our system in the TOP5 candidates. There is an \Avg. Loss" row for each task and run that shows the performance loss (averaged in all transcripts) introduced by each module in relation to the previous step. Note that this numbers have been gathered using an automatic application and some disagreement between in the selection of factoid questions may exist, therefor the TOP5 scores in this table may di er slightly from the o cial QAst scores.
In Table 2 we can see that moving from transcript m to a implies a loss of 10 points in TOP5
score for T1. For T2 this loss is as much as 50 points. But subsequent increases of WER in
transcripts b and c have a low impact in our performance. According to the QAst 2009 overview
paper [
We should note an important loss of performance in T1 scenario when using the question set B. Table 2 shows that TOP5 and TOP1 scores decrease by 50% or more when compared to question set A. Table 4 shows that the number of correctly classi ed questions drops from 49 to 22 in T1 (44% of the original), therefore QA&PR drops about 30%. The AE module has comparable performance with both sets thus this poor performance is due solely to our QC module. In scenario T2 there is a smaller loss in QC (just 30%) and it has di erent repercussion. The speci c distribution of errors among QC and PR leads a higher QC&PR count in all T2 tracks than in T1 tracks, although T1 has 20 more questions than T2, and this yields a smaller performance drop in T2. This must be blamed on our machine learning question classi er. Although this is based on shallow textual analysis, the model doesn't generalize well to spontaneous questions. Probably it has an strong dependency on the usual well{formed question structure. Additionally, we note that the classi cation of written questions is better for T2 question set than T1 question set. This suggests that in this evaluation T1 questions are more domain speci c than the others.
The di erence between runs number 1 and 2 is that number 2 uses hand{tagged NEs instead of our automatic NERC. The results show that it has little impact on performance. In Table 4 we can see that most of the correct answers retrieved by our PR module are annotated with the correct entity. It is shown by the small di erence between QC&PR and C.NE columns. Using hand-tagged NEs improves slightly the results for TOP5 and TOP1, probably because it lters out incorrect candidates and the AE process becomes easier. As we have seen in Table 3, the F =1 score of our NERC models is below 70% but this poor performance doesn't re ect in the nal QA results. We think that hand-tagged NEs doesn't improve the results due to two facts. On one hand, the NERC we have developed is useful enough for this task even having poor F =1 scores, and on the other hand, there is a probable disagreement between the humans who tagged the NEs and the humans who wrote the questions.
One of the main issues of our QAst 2009 system is the poor performance of the AE module. More than 25% of the correctly retrieved and tagged answers aren't correctly extracted in T1, and more than 50% are lost in T2. In fact, AE is the main source of errors in T2, more than the combination of both PR and QC. This is a big di erence with the results achieved in 2008 evaluation, where AE was of high accuracy. It shows that our answer selecting heuristics may be more domain{dependant than we knew and they should be tuned for this task. 5
This paper describes UPC's participation in the CLEF 2009 Question Answering on Speech Transcripts track. We submitted runs for all English and Spanish scenarios. In this evaluation we analyzed the impact of using gold{standard NEs with using a far from perfect NERC.
We have developed a new NERC designed for speech transcripts that shows results competitive with gold-standard NEs when used in Question Answering.
The results achieved in the di erent scenarios and tasks are not the top ones. But there is little degradation due to ASR e ects thus showing that our QA system is highly robust to transcript errors, being this one of the main focuses of the QAst evaluation.
This work has been partially funded by the Spanish Ministry of Science and Technology (TEXTMESS project).