This paper describes the experience of QAST 2009, the third time a pilot track of CLEF has been held aiming to evaluate the task of Question Answering in Speech Transcripts. Four sites submitted results for at least one of the three scenarios (European Parliament debates in English and Spanish and broadcast news in French). In order to assess the impact of potential errors of automatic speech recognition, for each task manual transcripts and three different ASR outputs were provided. In addition an original method of question creation was tried in order to get spontaneous oral questions resulting in two sets of questions (spoken and written). Each participant who had chosen a task, was asked to submit a run for each condition. The QAST 2009 evaluation framework is described, along with descriptions of the three scenarios and their associated data, the system submissions for this pilot track and the official evaluation results.
Question Answering (QA) technology aims at providing answers to natural language questions.
Current QA technology is focused mainly on the mining of written text sources for extracting the
answer to written questions from both open-domain and restricted-domain document collections
[
Current text-based QA systems tend to use technologies that require text written in accordance
with standard norms for written grammar. The syntax of speech is quite different than that of
written language, with more local but less constrained relations between phrases, and punctuation,
which gives boundary cues in written language, is typically absent. Speech also contains
disfluencies, repetitions, restarts and corrections. Moreover, any practical application of search in speech
requires the transcriptions to be produced automatically, and the Automatic Speech Recognizers
(ASR)introduce a number of errors. Therefore current techniques for text-based QA need
substantial adaptation in order to access the information contained in audio documents, and probably
to analyse oral questions. Preliminary research on QA in speech transcriptions was addressed in
QAST 2007 and QAST 2008, pilot evaluation tracks at CLEF in which systems attempted to
provide answers to written factual and definitional written questions by mining speech transcripts
of different scenarios [
This paper provides an overview of the third QAST pilot evaluation. Section 2 describes the principles of this evaluation track. Sections 3 and 4 present the evaluation framework and the systems that participated, respectively. Section 5 reports and discusses the achieved results, followed by some conclusions in Section 6. 2
The aim of this third year of QAST is to provide a framework in which QA systems can be evaluated in a real scenario, where the answers of both spontaneous oral questions and written questions have to be extracted from speech transcriptions, these transcriptions being manually and automatically generated. There are five main objectives to this evaluation: • Motivating and driving the design of novel and robust QA architectures for speech transcripts; • Measuring the loss due to the inaccuracies in state-of-the-art ASR technology; • Measuring this loss at different ASR performance levels given by the ASR word error rate; • Measuring the loss when dealing with spontaneous oral questions • Motivating the development of monolingual QA systems for languages other than English. In the 2009 evaluation, as in the 2008 evaluation, an answer is structured as a simple [answer string, document id] pair where the answer string contains nothing more than the full and exact Spontaneous oral question: When did the bombing of Fallujah eee took take place? Written question: When did the bombing of Fallujah take place? Manual transcript: (%hesitation) a week ago President the American (%hesitation) occupation forces (%hesitation) m() m() m() marched into Fallujah and they (%hesitation) bombarded (%hesitation) m() murdered and have been persecuting everyone in the city . Answer: a week ago answer, and the document id is the unique identifier of the document supporting the answer. For the tasks on automatic speech transcripts, the answer string consisted of the <start-time> and the <end-time> giving the position of the answer in the signal.
A total of six tasks were defined for this third edition of QAST covering three scenarios: English questions related to European Parliament sessions in English (T1a and T1b), Spanish questions related to European Parliament sessions in Spanish (t2a and T2b) and French questions related to French Broadcast News (t3a and T3b). The complete set of tasks is: • T1a: QA of English written questions in the manual and automatic transcriptions of European Parliament Plenary sessions in English (EPPS English corpus). • T1b: QA of manual transcriptions of English spontaneous oral questions in the manual and automatic transcriptions of European Parliament Plenary sessions in English (EPPS English corpus). • T2a: QA of Spanish written questions in the manual and automatic transcriptions of European Parliament Plenary sessions in Spanish (EPPS Spanish corpus). • T2b: QA of manual transcriptions of Spanish spontaneous oral questions in the manual and automatic transcriptions of European Parliament Plenary sessions in Spanish (EPPS Spanish corpus). • T3a: QA of French written questions in manual and automatic transcriptions of broadcast news for French (ESTER corpus) • T3b: QA of manual transcriptions of French spontaneous oral questions in manual and automatic transcriptions of broadcast news for French (ESTER corpus) 3
3.1
The QAST 2009 data is derived from three different resources, each one corresponding to a different
language (English, Spanish and French):
• English parliament (EPPS EN): The TC-STAR05 EPPS English corpus [
European Parliament and Broadcast News data are usually referred to as prepared speech. Although they typically have few interruptions and turn-taking problems when compared to actual spontaneous speech, many of the characteristics of spoken language are still present (hesitations, breath noises, speech errors, false starts, mispronunciations and corrections). 3.2
For each of the three languages, two sets of manually transcribed spontaneous oral questions and their respective written questions have been created and provided to the participants, the first for development purposes and the second for the evaluation: • Development sets (released on the 25th of March 2009): – EPPS EN: 50 transcribed questions and their respective written questions. – EPPS ES: 50 transcribed questions and their respective written questions.
– French BN: 50 transcribed questions and their respective written questions. • Evaluation sets (released on the 1st of June 2009): – EPPS EN: 100 transcribed questions and their respective written questions. – EPPS ES: 100 transcribed questions and their respective written questions. – French BN: 100 transcribed questions and their respective written questions. For each language, both the development and evaluation sets were created from the whole document collection (i.e. the 6 European Parliament sessions for English and Spanish, and the 18 Broadcast News shows for French). In other words, there was no collection split between a development data set and an evaluation data set as was done last year.
As for last year, two types of questions were considered: factual questions and definitional ones. The expected answer to a factual question is a named entity. There were 6 types of factual question this year, each corresponding to a particular category of named entities: • Person: names of humans, real and fictional, fictional or real non-human individuals.
Ex: Mirjam Killer, John, Jesus, etc. • Organisation: names of business, multinational organizations, political parties, religious groups, etc.
Ex: CIA, IBM, but also named entities like Washington when they display the characteristics of an organisation. • Location: geographical, political or astronomical entities.
Ex: California, South of California, Earth, etc. • Time: a date or a specific moment in time, absolute and relative time expressions.
Ex: March 28th, last week, at four oclock in the morning, etc. • Measure: measures of length, width or weight, etc. Generally, a quantity and a unit of measurement.
Ex: five kilometers, 20 hertz, etc. But also ages, period of time, etc.
This is less than the 10 categories used for the 2007 and 2008 evaluations. Some categories have not been considered this year because no occurence were found in the collected set of sponteaneous questions (Color, Shape, Language, System, Material).
The definition questions are questions such as What is the CDU? and the answer can be anything. In this example, the answer would be political group. This year, the definition questions are subdivided into three types: • Person: question about someone.
Q: Who is George Bush?
R: The President of the United States of America. • Organisation: question about an organisation.
Q: What is Cortes?
R: Parliament of Spain. • Other: questions about technology, natural phenomena, etc.
Q: What is the name of the system created by AT&T?
R: The How can I help you system.
For each language a number of ’NIL’ questions (i.e., questions having no answer in the document collection) have been selected. The distribution of the different types of questions across the three collections is shown in Table 1.
Type T1 (English) T2 (Spanish) T3 (French)
Factual 75% 55% 68%
Definition 25% 45% 32%
NIL 18% 23% 21%
The question sets are formatted as plain text files, with one question per line (see the QAST 2008 Guidelines1). The procedure to generate the questions is described in the following section. 3.2.1
Question generation A novel feature in QAST 2009 was the introduction of spontaneous oral questions. The main issue in the generation of this kind of questions was how to obtain spontaneity. The solution adopted was to set up the following procedure for question generation: 1. Passage generation: a set of passages was randomly extracted from the document collection.
A single passage was composed by the complete sentences included in a text window of 720 characters. 2. Question generation: human question generators were randomly assigned a number of passages (varying from 2 to 4). They had to read each passage and then to formulate one or more questions based on the passage they just read about information not present in it. 3. Question transcription: precise transcriptions of the oral spontaneous questions were made, including hesitations, etc.
Ex: (%hesitation) What (%hesitation) house is the pres() the president elect being elected to? 4. Question filtering: some questions were filtered out from the set of generated questions because their answer types were not allowed or because they did not have answer in the document collection. The resulting questions were usable questions. 1http://www.lsi.upc.edu/˜qast: News 5. Written question generation: the usable questions were re-written by removing speech disfluencies, correcting the syntax and simplifying the sentence when necessary.
Ex: What house does the president run? 6. Question selection: the final set of development questions and test questions were selected by ELDA from the usable questions.
• definition: person, organisation, object and other • factoid : person, location, organisation, time (includes date), measure and language However, the types “language” for factual questions and “object” for definition questions did not occur among the generated questions.
A preliminary evaluation of the generated questions was carried out in order to determine how many usable questions could be produced by a human reader. The results of this evaluation show that the percentage of usable questions produced by the questions generator was between 47% and 58% of the total questions produced, depending on the speakers knowledge of the task guidelines. These figures show that the produced questions were more than the number of questions actually presented to participants in QAST 2009. Most unusable questions were due to the fact that human question generators forgot the guidelines many times while asking their questions. Table 3.2.1 shows the number of questions recorded, the resulting usable questions and the average of the length in words per question for each language.
French Spanish #speaker 12 7 11 #questions recorded 1096 485 403 #usable questions avg. #words 616 9.1 335 7.7 313 7.1 As in 2008, the answer files submitted by participants have been manually judged by native speaking assessors, who considered the correctness and exactness of the returned answers. They also checked that the document labeled with the returned document ID supports the given answer. One assessor evaluated the results, and another assessor manually checked each judgment of the first one. Any doubts about an answer was solved through various discussions. The assessors used the QASTLE2 evaluation tool developed in Perl (at ELDA) to evaluate the systems’ results. A simple window-based interface permits easy, simultaneous access to the question, the answer and the document associated with the answer.
After each judgment the submission files were modified by the interface, adding a new element in
the first column: the answer’s evaluation (or judgment). The four possible judgments (also used
at TREC[
The two following metrics (also used in CLEF) were used in the QAST evaluation: 1. Mean Reciprocal Rank (MRR): This measures how well the right answer is ranked in the list of 5 possible answers. 2. Accuracy: The fraction of correct answers ranked in the first position in the list of 5 possible answers. 4
A total of four groups from four different countries submitted results for one or more of the proposed QAST 2009 tasks. Due to various reasons (technical, financial, etc.), eight other groups registered but were not be able to submit any results.
• INAOE, Instituto Nacional de Astrof´ısica, Optica y Electr´ıca, Mexico; • LIMSI, Laboratoire d’Informatique et de M´ecanique des Sciences de l’Ing´enieur, France; • TOK, Tokyo Institute of Technology, Japan; • UPC, Universitat Polit`ecnica de Catalunya, Spain.
All groups participated to task T1 (EPPS EN), UPC and LIMSI participated to task T2 (EPPS ES) and only LIMSI dealt with task T3 (French BN). Each participant could submit up to 48 submissions (2 runs per task and transcription). In order to allow comparisons on the performance of the systems when using different WER levels in the transcriptions, it was mandatory for each task to submit results for all the data: the manual transcriptions and the three ASR outputs (automatic transcriptions).
The results for the three tasks in manual transcribed data are presented in Tables 5 to 7, according to the question types (factual, definitional and all questions). MRR 0.39 0.39 0.38 0.39
All
The results for the three tasks in automatically transcribed data are presented in Tables 8 to 10, according to the question types (factual, definitional and all questions). 7 systems participated in the T1 (English) task on manual transcripts and 6 on automatic transcripts.
On manual transcripts, the accuracy ranged from 28% to 5% (for written questions) and from 26% to 3% (for spoken questions).
For five of the systems, we observe a relatively small difference between written and spoken questions (from 2% to 5% loss going from written questions to spoken questions). The other two systems encountered a significant loss (13% and 16% of difference between written and spoken questions).
There were three approaches for QA on automatic speech transcripts used by the systems. The
LIMSI and UPC on all ASRs and INAOE on ASR A and ASR B took the ASR output at the only
available information. INAOE on ASR C used information extracted from all the ASR outputs,
keeping ASR C as primary. This approach could represent an application where multiple ASR
outputs from different systems are available. Combining outputs from varied systems is a standard
method in speech recognition to obtain a better word error rate [
On the ASR transcripts for the pure systems, the accuracy ranged for the best ASR (10.6%
System ASR A of WER) from 26% (written questions) to 5% (spoken questions). Accuracy goes down with increased word error rate giving a roughly 5% loss for ASR B and ASR C compared to ASR A. It is interesting to note that the differences between ASR B (WER 14%) and ASR C (WER 24%) are negligible. The INAOE multi-ASR approach paid off by giving an overall result better than what was obtained by the same system on the best ASR only.
We notice that the impact of written vs spoken questions is similar than for manual transcriptions, with two systems taking an heavy loss and the others not showing a significant difference. Four systems (2 from LIMSI and 2 from UPC) participated in the T2 (Spanish) task on manual transcripts and 3 systems (1 from LIMSI and 2 from UPC) on automatic transcripts. On manual transcripts, the accuracy ranged from 36% (written questions and spoken questions) to 14% (written questions) and 17% (spoken questions). The differences between written questions and spoken questions is very low (from 0% to 3%). The same kind of behaviour is observed on the automatic transcripts tasks, with a loss due to the speech recognition errors and no significant difference between written and spoken questions.
System 26.0% 6.0% 6.0% 8.0% 11.0% 25.0% 25.0% 8.0% 8.0% 7.0% 7.0%
Only 2 systems (both from LIMSI) participated in the T3 (French) task on manual transcripts and one (from LIMSI) on automatic transcripts.
On manual transcripts, the accuracy ranged from from 28% (both written and spoken questions) to 27% (written questions). There is no significant differences between spoken and written questions (0% to 1% loss). The results for automatic transcriptions show very little loss compared to the manual transcriptions except for the worst ASR.
The overall absolute results were worse this year compared to last year which points to a globally harder task. The question development method produces requests which qualitatively seem to be more different to what is found in the documents compared to questions built after reading the documents. In our opinion that method, while giving an harder problem, puts us closer to a real, usable application. 6
In this paper, the QAST 2009 evaluation has been described. Four groups participated in this track with a total of 86 submitted runs across 3 main tasks that included dealing with different languages (English, Spanish and French), different word error rates for automatic transcriptions (from 10.5% to 35.4%) and different question types (written and spoken questions). An original question creation method has been tried succesfully to generate spontaneous spoken questions. Qualitatively, the questions were harder and more different to the formulations found in the documents compared to those produced by the traditional method of consulting the documents first. The method used this year gives an harder problem but we think that it is a more realistic one, putting us closer to a real, usable application.
This work has been jointly funded by the Spanish Ministry of Science (TEXTMESS project TIN2006-15265-C06) and OSEO under the Quaero program. We thank to Lori Lamel, Erik Bilinski, Manuel Gonz´alez and Pere Vilarrubia their help to the organisation and data generation.