This paper describes the participation of the Technical University of Catalonia in the CLEF 2008 Question Answering on Speech Transcripts track. We have participated in the English and Spanish scenarios of QAst. For the processing of manual transcripts we have deployed a robust factual Question Answering that uses minimal syntactic information. For the handling of automatic transcripts we combine the QA system with a Passage Retrieval and Answer Extraction engine based on a sequence alignment algorithm that searches for \sounds like" sequences. We perform a detailed analysis of our results and draw conclusions relating QA performance to word error rate (WER) in transcripts.
Our QA system is based on our previous work in [
The architecture of our QA system follows a commonly-used schema which splits the process into three phases performed sequentially: Question Processing (QP), Passage Retrieval (PR), and Answer Extraction (AE), as shows Figure 1. These three phases are described in the following sections. 2.1
Question Processing and Classi cation
The main goal of this component is to detect the type of the expected answer. We currently
recognize the 53 open-domain answer types from [
For scenario T5, he have developed an Spanish question classi er using human translated
questions from the corpus of [
Passage Retrieval
This component retrieves a set of relevant passages from the document collection, given the
previously extracted question keywords. The PR algorithm uses a query relaxation procedure that
iteratively adjusts the number of keywords used for retrieval and their proximity until the
quality of the recovered information is satisfactory (see [
When dealing with automatic transcripts, you have to bear in mind that the state of the art in ASR technology is far from perfect. For example, the word error rate (WER) of the meetings automatic transcripts (T1B) is around 38% and the WER of the lectures (T2B) is over 20%, and from 10.6% to 24.1% for the T4B transcripts. Figure 2 shows three real examples of common errors when generating automatic transcripts. From the point of view of passage retrieval, imperfect 1M: \The pattern frequency relevance rate indicates the ratio of relevant documents. . . " 1A: \the putt and frequency illustrating the case the ratio of relevant documents. . . " 2M: \The host system it is a UNIX Sun workstation" 2A: \that of system it is a unique set some workstation" 3M: \Documents must be separated into relevant documents and irrelevant documents by a manual process, which is very time consuming." 3A: \documents must be separated into relevant documents and in relevant document by a manual process witches' of very time consuming" transcripts create a new problem of incorrectly transcribed words that yield false positives and false negative for traditional search methods.
To overcome such drawbacks, we have used an IR engine relying on phonetic similarity for the
automatic transcripts. This tool is called PHAST (after PHonetic Alignment Search Tool) and
uses pattern matching algorithms to search for small sequences of phones (the keywords) into a
larger sequence (the documents) using a measure of sound similarity. A detailed description of
PHAST can be found in [
Answer Extraction
Identi es the exact answer to the given question within the retrieved passages. First, answer
candidates are identi ed as the set of NEs that occur in these passages and have the same type as
the answer type detected by QP. Then, these candidates are ranked using a scoring function based
on a set of heuristics that measure keyword distance and density[
The same measure is used for English and Spanish scenarios. 3
As described before, we extract candidate answers from the NEs that occur in the passages retrieved by the PR component. We detail below the strategies used for NERC in both manual and automatic transcripts.
NERC for English Manual Transcripts.
We have used a modi ed version of the NERC presented in [
NERC for English Automatic Transcripts. We have used a similar framework for the processing of automatic transcripts: we annotated the development corpora and trained speci c NERC models for lectures and meetings. The signi cant di erence is that here we expand the classi ers' feature sets with phonetic attributes. These features are motivated by the fact that even when the ASR incorrectly transcribes NEs the phonetic structure is by and large maintained in the transcript (e.g. in Figure 2 the name \Sun" is recognized as \some"). We used an unsupervised hierarchical clustering algorithm that groups tokens based on the similarity of their phonetic sequences. The stop condition of the algorithm is set to reach a local maximum of the Calinski criterion [1]. Then the cluster of each token is added as a feature (e.g. \Sun" and \some" share the same cluster), which helps the NERC model generalize from the correct to the incorrect transcript. We also added phonetic features that model pre x and su x similarity.
NERC for Spanish. For the Spanish track T5 we have used a previously developed NERC.
It uses a machine learning approach and it has been trained with the CoNLL Spanish corpus. See
details in [
UPC participated in 4 of the 5 scenarios, all but the French one (T3). We submitted two runs for the tasks on automatic transcripts, one using run using the standard QA system for written text (QAm) and another run using the system tailored for automatic transcripts (QAa). See section 2 for the di erences between both. Each scenario included 100 test questions, from which 10 does not have an answer in the corpora (these are nil questions). Around 75% of the questions are of factual types and around 25% are de nitional. Our QA system is designed to answer only factual questions, therefore the our experimental analysis will refer only to factual questions.
We report two measures: (a) TOPk, which assigns to a question a score of 1 only if the system provided a correct answer in the top k returned; and (b) Mean Reciprocal Rank (MRR), which assigns to a question a score of 1=k, where k is the position of the correct answer, or 0 if no correct answer is found. The o cial evaluation of QAst 2008 uses TOP1 and TOP5 measures []. An answer is considered correct by the human evaluators if it contains the complete answer and nothing more, and it is supported by the corresponding document. If an answer was incomplete or it included more information than necessary or the document did not provide the justi cation for the answer, the answer was considered incorrect.
Table 1 summarizes our overall results for factual question only. The cost of moving from
manual transcripts to automatic transcripts (i.e., the di erence between TXA and TXB) is a loss
in TOP1 score of at last 10% for T1, 43% for T2, 50% for T4 and 42% for T5. The performance
of QAa is very similar to QAm. As shown in QAst 2008 Overview paper [
Table 2 shows the distribution of correct answers for all tasks according to the answer type. In scenario T4, a design error prevented our NERC from recognizing entity types Sha, Mat and Col. Therefor there are 20 unanswerable questions from the 78 factual ones. Our system for the Spanish scenario (T5) is limited to answer types Org, Per, and Loc, so the real upper bound for factual questions is 36 instead of 75.
Finally, Table 3 summarizes the error analysis of QP, PR, and AE parts. The meaning of each column is the following. Q: number of factual question. QC: number of questions with answer type correctly detected by QP. PR: number of question where at least on passage with the correct answer war retrieved. C.NE: number of questions where the retrieved passages contain the correct answer tagged as a NE of the right type. U.NE: number of questions where the retrieved passages contain the correct answer but it remains undetected by the NERC. Er.NE: number of questions where the retrieved passages contain the correct answer tagged as a NE with an incorrect type. #Q
QC&PR: number of questions with correct answer type and correct passage retrieval. QC&NE: number of questions with correct answer type and correctly tagged answer in the passages. TOP5 non-nil: number of question with non-nil answer correctly answered by our system in the TOP5 candidates. Due to technical reasons this analysis has not been performed on task T2B.
We can draw several important observations from this error analysis: Question classi cation
performs better for T1 question set than T2 and T4 question sets. This suggests that in this
evaluation T1 questions were more domain speci c than the others. In T5, results are really
disappointing and this suggests that our Spanish classi er may be too domain dependant since
it achieves 74% accuracy in our test data. \PR" is specially degraded in task T4B-C, where
we processed automatic transcripts with the highest WER (24.1%). This proves that passage
retrieval is indeed a ected by a high WER but is robust enough to be used with a good ASR.
Passage retrieval using PHAST performed better than the passage retrieval with classical retrieval
for tasks in T5 and worse for tasks in T4. Since both scenarios have similar domain, we think this
di erence is due to the nature of Spanish and English phonology. Further experiments in [
System QAm QAm QAa QAm QAm QAm QAa QAm QAa QAm QAa QAm QAm QAa QAm QAa QAm QAa bound for answer extraction is limited specially by NERC performance. The average number of factual questions in all runs is 75.3, the average value for PR is 56.61 and the average for \C.NE" is 22.44, so in less than 40% of the passages the answer is correctly tagged allowing its correct extraction in the answer extraction step. \QC&NE" is a theoretical upper bound of the total score of each task. We can see that the performance of our answer extraction process is very good since \TOP5" score is very near this upper bound in all tasks. As a remark, all of the scores in T5 are above the upper bound. This is due to the combination of two factors: rst, a fall-back mechanism in our answer extraction process to help overcome the PER/ORG ambiguity2 in question classi cation, this mechanism allows to answer misclassi ed questions. Second, a double{error situation when the question is misclassi ed and the answer is erroneously tagged but matches the question type.
The impact of transcription errors in QA can be analyzed in detail thanks to the three di erent automatic transcripts for task T4B (WERs of T5B have very close values and our overall performance is far too poor for this analysis). Figure 3 shows graphically the values in table 3 for T4, QAm. The yellow bars show the WER percentage for each transcript (0% for manual reference) and the lines show the evolution of variables \PR", \C.NE", \U.NE", \QC&NE" and \TOP5". The performance of passage retrieval decreases linearly with WER increase. The linear regression ts the data with a Pearson coe cient r = 0:99. Other measures such as \C.NE", \QC&NE" and \TOP5" are also strongly related to WER and its diminishment is more pronounced. All this measures decrease the same amount when going from 0% WER to 10.6% WER than from 10.6%% to 24.1%. In fact \C.NE" values t the non-linear regression curve
C:NE = 43:9 0:94WER with a coe cient r = 0:97. Therefore we can conclude that the passage retrieval performance decreases linearly with WER while NERC performance decreases exponentially with WER.
2In questions such as \Who helped solving the packet loss problem? " is impossible to know if the correct answer is a person name or an organization name. For this question, the answer is the name of a university. 60 50 40 30 20 10 0 10.6% 14%
WER
PR C.NE
U.NE QC&NE
TOP5 24.1% 0% Manual
A
B
C This paper describes UPC's participation in the CLEF 2008 Question Answering on Speech Transcripts track. We submitted runs for all English and Spanish scenarios, obtaining the best results in some tasks. In this evaluation we analyzed the behavior of two systems di ering in that one is tailored for manual transcripts while the other is tailored for automatic transcripts (uses approximate keyword search based on phonetic distances and a NERC enhanced with phonetic features).
Our approximated keyword search algorithm used for passage retrieval obtains mixed results. It can improve standard search for Spanish but makes little di erence for English. We think this because in some document collections it may generated too many false-positive, introducing noise in sets of candidate passages and answers. Nevertheless, we believe that this approach is a good long-term research direction because it can truly address the phenomena speci c to automatic transcripts.
Finally, our results show that automatic speech recognition has critical impact on the performance of NERC but its a ect on passage retrieval is much less severe.
This work has been partially funded by the European Commission (CHIL, IST-2004-506909) and the Spanish Ministry of Science (TEXTMESS project). [1] T. Calinski and J. Harabasz. A dendrite method for cluster analysis. Communications in
Statistics, 3, 1974.