This paper describes the first participation of IR-n system at Spoken Document Retrieval, focusing on the experiments we made before participation and showing the results we obtained. IR-n system is an Information Retrieval system based on passages and the recognition of sentences to define them. So, the main goal of this experiment is to adapt IR-n system to the spoken document structure by means of the utterance splitter and the overlapping passage technique allowing to match utterances and sentences Usually, research work on natural language processing has started from written documents instead of spoken documents due to spoken document processing has a lot of disadvantages induced by its informal disposition among other reasons. As appointed by Dahlba¨ck [2]: “... spoken input is often incomplete, incorrect and contains interruptions and repairs; full sentences occur only very occasionally. Therefore new basic units for the development of dialogue models have to be proposed ...” Thus, some of the most important problems to solve in spoken document processing are [3]: ² Moreover, turns may be considered like null or empty when they do not contribute to the discourse, that is, turns having the function of pointing out the speaker is on the conversation: “ejem...”, “yes...”, “I know...”; as well as other turns without semantic content such as “good morning”, “have a good weekend”, and so on.
² The lack of punctuation marks, that impedes the well understanding of sentences because boundaries are unknown. This understanding must be induced by pause detection. This is the reason why the “sentence” concept is replaced by the “utterance” concept. Utterance is defined, from a pragmatic point of view, as a sequence of words chained by a speaker between two pauses. In the same way, the “paragraph” is replaced by the “turn” that is defined from a pragmatic point of view as the set of utterances that a speaker can express between two speaker changes (when several speakers participate in the dialogue), or the set of utterances that a speaker expresses about the same subject (in monologues or newsreels). ² Furthermore, turns can be interrupted due to overlaps, or speaker mistakes, causing repetitions and modifications of previous information.
This sort of problems is increased with problems derived from the automatic transcription process which incorporates noise, spelling mistakes, and unrecognizable words due to deficiencies in the original recording or speak recognition fails.
Due to this, the use of spoken documents in information retrieval tasks allows to test the system robustness against document mistakes. Then, the main goal of this paper is to test the robustness of IR-n System and to study some text processing techniques that could improve this robustness in spoken documents.
IR-n is an Information Retrieval system based on passages [
SDR task is based on processing non-structured documents that proceed from an automatic transcription of radio news. Our main objective is to test if IR-n system can be applied to document collections where sentence boundaries are unknown. This experiment is focused on the estimation of sentence boundaries by means of the pauses recognized along the transcription process. So, the main hypothesis is based on the following ideas: ² longest pauses mean the end of utterances ² IR-n System can accept utterances instead of sentences to define passages.
So, the experiments will focus on determining what is the average length of a pause between utterances to build an utterance splitter that will feed the IR-n system.
However, using this model, passage definitions may be faulty. The terms of a query may be dispersed among several passages, and some relevant documents may be discarded. This problem can be avoid by using passage overlapping, since this technique allows more than one passage sharing the same fragment of document. 2
Cross-Language Spoken Document Retrieval (CL-SDR) is a new track proposed for CLEF 2003.
The track is mostly based on existing resources, available by NIST, which were used at TREC-8
[
The benchmark track is an extension of evaluation data prepared by NIST for TREC 8-9 SDR tracks. It has a collection of automatic transcripts (557 hours) of American-English news recordings broadcasted by ABC, CNN, Public Radio International, and Voice of America between February and June 1998. Transcripts are provided with known story boundaries (21,754 stories); and a collection of 100 English topics, either in terse or short format. The TREC collection has been extended with translations of the short topics into five European languages: Dutch, Italian, French, German, and Spanish.
Technical specifications of the task are shown in table 1. 3
Taking advantage of using sentences in IR-n as a basic unit to the passage definition task, the sentence will be used to define the passage overlapping too.
The overlapping degree (Gsol) in IR-n system shows the sentence number from which the definition of the next passage starts. The main features of this value are the following: 1. Gsol must be lower than the passage size. Having the same value means that no overlapping is used. 2. The lower the value Gsol is, the higher the amount of text shared by two consecutive passages will be. ² Objective: the track aims at evaluating CLIR systems on noisy automatic transcripts of spoken documents with known story boundaries. ² Development data (from TREC 8 SDR): ² Evaluation data (from TREC 9 SDR): ² Primary Conditions (mandatory for all participants): 1. Monolingual IR without using any parallel collection (contrastive condition).
2. Bilingual IR from French or German. ² Secondary Condition (optional): ² Submission of runs: 1. Monolingual IR using any available parallel collections. 2. Bilingual IR from other languages.
1. Maximum 12 runs per participant, with the limit of 3 runs for each considered source language. 1. P1 = f1::f15 2. P2 = f11::f25 3. P3 = f21::f35 3. As a result, the lower the value Gsol is, the more number of passages will be defined in the document.
The use of passage overlapping means to redefine the passage concept to IR-n in the following way: - Given a document D consisting of N sentences.
D = f1::fN - Taken into account that n is the number of sentences integrating a passage. - Given an overlapping degree Gsol - The following passages will be defined from the document D
Pi = fGsol¤(i¡1)+1; :::; fmin(Gsol¤(i¡1)+n;N ); i 2 [1::N=Gsol ¡ 1]
Given that definition, and supposing a passage size of 15 sentences, an overlapping degree of 10, and a document size of 35 sentences, the passage generation will be performed in the following way: (1) (2)
The increase of the efficiency in document retrieval is an immediately advantage of passage overlapping. However, the response time increases (to a large extent when the overlapping degree is lower) because the number of passages to be evaluated is greater.
Nevertheless, the use of lower overlapping degrees improves the system results noticeably, and it has not excessive influence on the searching time.
Overlapping does not increase the searching cost so much due to two main reasons:
1. IR-n does not evaluate each one of the document passages, since the similarity measure [
For this same reason, the last passage to be evaluated is the one finishing in the last sentence of the document in which a query term appears.
These same conclusions may be extended to passages not located at the limits of the document, that is, internal passages. Given an overlapping degree Gsol, if a passage does not contain query terms during its first sentences then its evaluation can be omitted. For example, if Gsol is equal to 1, the evaluation of those passages which first sentence does not contain any query term is not needed.
Because of this, the number of passages to be evaluated is reduced, and, consequently, to use of small overlapping degrees has not the same influence as if each passage of the document is evaluated. 2. Another important aspect is related to the system implementation. IR-n implementation is based on storing all the information about word occurrences in main memory. Thus, the segmentation process is performed during the execution over data structures located at main memory.
Considering that the most influencing factors to time processing are related to disc access times, this minor increase of time when a greater number of passages is processed, it is not significant to the final time.
For this reason, IR-n uses an overlapping degree (Gsol=1) being the value that obtains the best performance. 4
According to the track specification, the test collection used in this experiment was TREC-8. During these experiments several passages sizes (from 1 to 9 sentences) and several pause recognition sizes (0.1, 0.2, and 0.3 seconds) have been valuated. Moreover, the IR-n system with and without query expansion has been tested.
Tables 2, 3 and 4 show the results without query expansion.
Tables 5, 6 and 7 show the results with query expansion.
These tables show that the best result is obtained using the model with query expansion, a passage size of 5 sentences and 0.2 seconds to recognize a pause between two utterances at the utterance splitter. 5
This system was evaluated with the TREC SDR-9 collection according to the track specification. Moreover, a bilingual test was performed using French queries that were translated into English by Power Translator, Free-translator and Babel Fish.
Both monolingual and bilingual tests were performed with and without query expansion. The best results for monolingual and bilingual queries are shown in tables 8 and 9 respectively.
Although we expected to know more information about other systems at the conference, we are pleased to see these results being above average for SDR track, taking into account that IR-n system was not designed to work on spoken documents.
Nevertheless, more experiments are expected to be done to increase the system performance. 7
This work has been partially supported by the Spanish Government (CICYT) with grant TIC20000664-C02-02 and (PROFIT) with grant FIT-150500-2002-416.