This paper describes the experience of QAST 2008, the second time a pilot track of CLEF has been held aiming to evaluate the task of Question Answering in Speech Transcripts. Five sites submitted results for at least one of the five scenarios (lectures in English, meetings in English, broadcast news in French and European Parliament debates in English and Spanish). In order to assess the impact of potential errors of automatic speech recognition, for each task contrastive conditions are with manual and automatically produced transcripts. The QAST 2008 evaluation framework is described, along with descriptions of the five scenarios and their associated data, the system submissions for this pilot track and the official evaluation results.
Question Answering (QA) technology aims at providing relevant answers to natural language
questions. Most Question Answering research has focused on mining document collections
containing written texts to answer written questions [
This paper provides an overview of the second QAST pilot evaluation. Section 2 describes the principles of this evaluation track. Sections 3 present the evaluation framework and section 4 the systems that participated. Section 5 reports and discusses the achieved results, followed by some conclusions in Section 6. 2
The objective of this pilot track is to develop a framework in which QA systems can be evaluated when the answers have to be found in speech transcripts, these transcripts being either produced manually or automatically. There are five main objectives to this evaluation: • Motivating and driving the design of novel and robust QA architectures for speech transcripts; • Measuring the loss due to the inaccuracies in state-of-the-art ASR technology; • Measuring this loss at different ASR performance levels given by the ASR word error rate; • Comparing the performance of QA systems on different kinds of speech data (prepared speech such as broadcast news (BN) or parliamentary hearings vs. spontaneous in meeting for instance); • Motivating the development of monolingual QA systems for languages other than English. In the 2008 evaluation, as in the 2007 pilot evaluation, an answer is structured as a simple [answer string, document id] pair where the answer string contains nothing more than the full and exact answer, and the document id is the unique identifier of the document supporting the answer. In 2008, for the tasks on automatic speech transcripts, the answer string consisted of the <starttime> and the <end-time> giving the position of the answer in the signal. Figure 1 illustrates this point comparing the expected answer to the question What is the Vlaams Blok? in a manual transcript (the text criminal organisation) and in an automatic transcript (the time segment 1019.228 1019.858). A system can provide up to 5 ranked answers per question. Question: What is the Vlaams Blok? Manual transcript: the Belgian Supreme Court has upheld a previous ruling that declares the Vlaams Blok a criminal organization and effectively bans it .
Answer: criminal organisation
A total of ten tasks were defined for this second edition of QAST covering five main task scenarios and three languages: lectures in English about speech and language processing (T1), meetings in English about design of television remote controls (T2), French broadcast news (T3) and European Parliament debates in English (T4) and Spanish (T5). The complete set of tasks are: • T1a: QA in manual transcriptions of lectures in English. • T1b: QA in automatic transcriptions of lectures in English. • T2a: QA in manual transcriptions of meetings in English. • T2b: QA in automatic transcriptions of meetings in English. • T3a: QA in manual transcriptions of broadcast news for French. • T3b: QA in automatic transcriptions of broadcast news for French. • T4a: QA in manual transcriptions of European Parliament Plenary sessions in English. • T4b: QA in automatic transcriptions of European Parliament Plenary sessions in English. • T5a: QA in manual transcriptions of European Parliament Plenary sessions in Spanish. • T5b: QA in automatic transcriptions of European Parliament Plenary sessions in Spanish. 3
3.1
The data for this second edition of QAST is derived from five different resources, covering
spontaneous speech, semi-spontaneous speech and prepared speech: The first two are the same as were
used in QAST 2007 [
• The CHIL corpus1 (as used for QAST 2007): The corpus contains about 25 hours of
speech, mostly spoken by non native speakers of English, with an estimated ASR Word
Error Rate (WER) of 20%.
• The AMI corpus2 (as used for QAST 2007): This corpus contains about 100 hours of
speech, with an ASR WER of about 38%.
• French broadcast news: The test portion of the ESTER corpus [
The spoken data cover a broader range of types, both in terms of content and in speaking style.
The Broadcast News and European Parliament date are less spontaneous than the lecture and
meeting speech as they are typically prepared in advance and are closer in structure to written
texts. While meetings and lectures are representative of spontaneous speech, Broadcast News and
European Parliament sessions are usually referred to as prepared speech. Although they typically
have few interruptions and turn-taking problems when compared to meeting data, many of the
characteristics of spoken language are still present (hesitations, breath noises, speech errors, false
starts, mispronunciations and corrections). One of the reasons for including the additional types
of data was to be closer to the textual data used to assess written QA, and to benefit from the
availability of multiple speech recognizers that have been developed for these languages and tasks
in the context of European or national projects [
Questions and answer types For each of the five scenarios, two sets of questions have been provided to the participants, the first for development purposes and the second for the evaluation. – Lectures: 10 seminars and 50 questions. – Meetings: 50 meetings and 50 questions. – French broadcast news: 6 shows and 50 questions. – English EPPS: 2 sessions and 50 questions.
– Spanish EPPS: 2 sessions and 50 questions. 1http://chil.server.de 2http://www.amiproject.org – Lectures: 15 seminars and 100 questions. – Meetings: 120 meetings and 100 questions. – French broadcast news: 12 shows and 100 questions. – English EPPS: 4 sessions and 100 questions.
– Spanish EPPS: 4 sessions and 100 questions.
Two types of questions were considered this year: factual questions and definitional ones. For each corpus (CHIL, AMI, ESTER, EPPS EN, EPPS ES) roughly 70% of the questions are factual, 20% are definitional, and 10% are NIL (i.e., questions having no answer in the document collection). The question sets are formatted as plain text files, with one question per line (see the QAST 2008 Guidelines3). The factual questions similar to those used in the 2007 evaluation. The expected answer to these questions is a Named Entity (person, location, organization, language, system, method, measure, time, color, shape and material). The definition questions are questions such as What is the Vlaams Blok? and the answer can be anything. In this example, the answer would be a criminal organization. The definition questions are subdivided into the following types: • Person: question about someone
Q: Who is George Bush?
R: The President of the United States of America. • Organisation: question about an organisation
Q: What is Cortes?
R: Parliament of Spain. • Object: question about any kind of objects
Q: What is F-15?
R: combat aircraft. • Other: questions about technology, natural phenomena, etc.
Q: What is the name of the system created by AT&T?
R: The How can I help you system. 3.2
As in QAST 2007, the answer files submitted by participants have been manually judged by native speaking assessors, who considered the correctness and exactness of the returned answers. They also checked that the document labeled with the returned docid supports the given answer. One assessor evaluated the results, and another assessor manually checked each judgment of the first one. Any doubts about an answer was solved through various discussions. The assessors used the QASTLE4 evaluation tool developed in Perl (at ELDA) to evaluate the responses. A simple window-based interface permits easy, simultaneous access to the question, the answer and the document associated with the answer.
For T1b, T2b, T3b, T4b and T5b (QA on automatic transcripts) the manual transcriptions were aligned to the automatic ASR outputs to find associate times with the answers in the automatic 3http://www.lsi.upc.edu/˜qast: News 4http://www.elda.org/qastle/ transcripts. The alignments between the automatic and the manual transcription were done using time information. Unfortunately, for some documents time information were not available and only word alignments were used.
After each judgment the submission files were modified, adding a new element in the first column:
the answer’s evaluation (or judgment). The four possible judgments (also used at TREC[
The two following metrics (also used in CLEF) were used in the QAST evaluation: 1. Mean Reciprocal Rank (MRR): This measures how well the right answer is ranked in the list of 5 possible answers.. 2. Accuracy: The fraction of correct answers ranked in the first position in the list of 5 possible answers. 4
A total of five groups from four different countries submitted results for one or more of the proposed QAST 2008 tasks. Due to various reasons (technical, financial, etc.), three other groups registered but were not be able to submit any results.
The five participating groups were: • CUT, Chemnitz University of Technology, Germany; • INAOE, Instituto Nacional de Astrofica, Optica y Electrica, Mexico; • LIMSI, Laboratoire d’Informatique et de M´ecanique des Sciences de l’Ing´enieur, France; • UA, Universidad de Alicante, Spain; • UPC, Universitat Polit`ecnica de Catalunya, Spain.
All groups participated to task T4 (English EPPS). Only LIMSI participated to task T3 (French broadcast news). Table 1 shows the number of submitted runs per participant and task. Each participant could submit up to 32 submissions (2 runs per task and transcription). The number of submissions ranged from 2 to 20. The characteristics of the systems used in the submissions are summarized in Table 2. A total of 49 submissions were evaluated with the distribution across tasks shown in the bottom row of Table 2.
The results for the ten QAST 2008 tasks are presented in Tables 3 to 12, according to factual questions, definitional questions, and all questions.
For manual transcriptions, the accuracy ranges from 45% (LIMSI1 on task T3a) down to 7% (UPC1 on task T5a). For automatic transcriptions, the accuracy goes from 41% (LIMSI1 on task T3b and ASR a) to 2% (UPC1 on task T5b and ASR c). Generally speaking, a loss in accuracy is observed when dealing with automatic transcriptions. Comparing the best accuracy results on manual transcription and automatic transcriptions, the loss of accuracy goes from 15% for task T2 to 4% for tasks T3 and T4 tasks. This difference is larger for tasks where the ASR word error rate is higher. cut1 cut2 limsi1 upc1
Acc
Acc 0.16 0.20 0.45 0.38 16.0 17.0 41.0 34.0
All MRR cut1 cut2 inaoe1 limsi1 ua1 upc1 12 12 41 44 32 38 0.21 0.22 0.38 0.42 0.27 0.37 21.0 21.0 33.0 33.0 20.0 34.0
Another observation concerns the loss of accuracy when dealing with different word error rates. Generally speaking higher WER results in lower accuracy (e.g. from 30% for T4b A to 20% for T4b B). Strangely enough this is not completely true for the T5b task where results for ASR C (13.7% WER) are 4% higher than for ASR B (12.7% WER). The WER being rather close, it is probable that ASR C errors had a smaller impact on the named entities present in the questions. 6
In this paper, the QAST 2008 evaluation has been described. Five groups participated in this track with a total of 49 submitted runs, across ten tasks that included dealing with different types of speech (spontaneous or prepared), different languages (English, Spanish and French) and different word error rates for automatic transcriptions (from 10.5% to 35.4%). For the tasks where the word error rate was low enough (around 10%) the loss in accuracy compared to manual transcriptions was under 5%, suggesting that QA in such documents is potentially feasible. However, even where ASR performance is reasonably good, there remain outstanding challenges in dealing with spoken language and the earlier mentioned differences from written language. The results from the QAST evaluation indicate that if a QA system which performs well on manual transcriptions it also performs reasonably well on high quality automatic transcriptions. The performance on spoken language have not yet reached the level of those in the main QA track.
This work has been jointly funded by the Spanish Ministry of Science (TEXTMESS project) and OSEO under the Quaero program.