This is a summary report about the Cross-Language Spoken Document Retrieval Track held at CLEF 2004. The report gives brief details of CL-SDR task based again this year on the TREC 8-9 SDR task. This year the CL-SDR task worked with an unknown story boundaries condition. The paper reports results from the participants showing that as expected cross-language results are reduced relative to a monolingual baseline, although the amount to which they are degraded varies for topic languages.
Queries are based on a collection of 100 English topics in short format for which relevance assessments are available. For the sake of CLIR, queries were translated by native speakers into Dutch, Italian, French, German, and Spanish. Retrieval scoring software is available both for the known and unknown story boundary conditions.
Of the available 100 topics, the first 50 (topic 074 to topic 123) were intended for system development, while the latter 50 (topic 124 to topic 173) for testing. Submission format and evaluation criteria followed the same conventions as the 2000 TREC-9 SDR track1.
The following evaluation conditions were specified: • Primary Conditions (mandatory for all participants): – Monolingual IR on NIST transcripts, no parallel data.
– Bilingual IR from French/German on NIST transcripts, no parallel data. • Secondary Conditions (optional): – Bilingual IR from French/German, on NIST transcripts, with parallel data.
– Bilingual IR from any language, all transcripts,with parallel data. 3
Two sites participated in the evaluation: University of Chicago (USA) and ITC-irst (Italy). A brief description of each system is provided. 3.1
The University of Chicago participated in the CLEF 2004 spoken document retrieval task. Runs were submitted for both the baseline English monolingual task and the French-English crosslanguage task, using only the resources provided by CLEF with no external resources. 3.1.1
Query processing aimed to enhance retrieval of the potentially errorful ASR transcriptions through pseudo-relevance feedback expansion. The baseline conditions required the use of only the CLEF provided resources. This restriction limited our source of relevance feedback to the ASR transcriptions, segmented as described below. For both the monolingual English and the English translations of the original French queries, we performed the same enrichment process. We employed the INQUERY API to identify enriching terms based on the top 10 ranked retrieved segments and integrated these terms with the original query forms. Our hope was that this enrichment process would capture both additional on-topic terminology as well as ASR-specific transcriptions.
For the French-English cross-language condition, we performed dictionary-based term-by-term
translation, as described in [
This year the SDR track focused on the processing of news broadcasts with unknown story
boundaries. This formulation required that sites perform some automatic segmentation of the full
broadcasts into smaller units suitable for retrieval. Using an approach inspired by [
1See http://www.nist.gov/speech/tests/sdr/sdr2000/sdr2000.htm. 3.1.3
To produce suitable retrieval segments, we merged the fine-grained segments returned by the base retrieval process on a per-query basis. For each query, we retrieved 5000 fine-grained segment windows. We then stepped through the ranked retrieval list merging overlapping segments, assigning the rank of the higher ranked segment to the newly merged segment. We cycled through the ranked list until convergence. The top ranked 1000 documents formed the final ranked retrieval results submitted for evaluation. 3.2
The ITC-irst system is based on the following three processing steps.
First, a collection of news segments is automatically created from the continuous stream of transcripts. Text segments are produced with a shifting time-window of 30 seconds, moved with steps of 10 seconds. Moreover, segments are also truncated if a silence period longer than 5 seconds is found.
Second, the resulting overlapping texts are used as target document collection by means of a
text CLIR system [
Third, entries in the ranking list which correspond to overlapping segments are properly merged.
The implemented method works as follows. All retrieved segments of the same news show are sorted by their start time. The first retrieved segment is assumed as the beginning of a new story. If the second segment overlaps with the first, the two are merged, and the time extent of the current story is adjusted, and so on. If a following segment does not overlap with the current story, the current story is saved in a stack, and a new story begins. Finally, for all stories in the stack, only the segments with the highest retrieval score are considered. The process is repeated for all news show files, with at least one entry in the rank list. The resulting list of non overlapping segments is then sorted according to the original retrieval score. 4
Site ITC-irst U. Chicago
Source Monolingual French German Italian Spanish Monolingual French
Primary .3059 .1816 .1584
The results in Table 1 show that, particularly in the primary condition, there is a considerable
loss in retrieval effectiveness for cross-language relative to monolingual retrieval. This reduction in
average precision varies between about 40% and 60%. These figures are larger than those observed
for the known story boundary test condition in the CLEF 2003 CL-SDR task [
As we would expect the use of additional data resources produces an improvement in absolute retrieval performance figures in all cases, although the relative cross language reduction is still very large for all conditions except for Spanish topic translation.