The CLEF 2003 Cross-Language Spoken Document Retrieval Track

Abstract

The current expansion in collections of natural language based digital documents in various media and languages is creating challenging opportunities for automatically accessing the information contained in these documents. This paper describes the CLEF 2003 track investigation of Cross-Language Spoken Document Retrieval (CLSDR) combining information retrieval, cross-language translation and speech recognition. The experimental investigation is based on the TREC-8 and TREC-9 SDR evaluation tasks, augmented to form a CLSDR task. The original task of retrieving English language spoken documents using English request topics is compared with cross-language retrieval using French, German, Italian, Spanish and Dutch topic translations.

Introduction

• scoring software for the known/unknown story boundary condition

The TREC collections have been extended to a CLSDR task by manuall translating with the short topics into five European languages: Dutch, Italian, French, German, and Spanish. Track Specifications The track aimed at evaluating CLIR systems on noisy automatic transcripts of spoken documents with known story boundaries. The following specifications were defined about data and resources participants were allowed to use for development and evaluation purposes.

Development data (from TREC 8 SDR) a Document collection: the B1SK Baseline Transcripts collection with known story boundaries made available by NIST. b Topics: Short topics in English, French, German, Italian, Spanish and Dutch made available by ITC-irst. c Relevance assessments: Topics-074-123.

d Parallel document collections (optional): available through LDC.

Evaluation data (from TREC 9 SDR) a Document collection: the B1SK Baseline Transcripts collection with known boundaries made available by NIST. b Topics: Short topics in English, French, German, Italian, Spanish and Dutch. c Relevance assessments: Topics-124-173 d Parallel document collections (optional): available through LDC.

Primary Conditions (mandatory for all participants) • Monolingual IR without using any parallel collection (contrastive condition). • Bilingual IR from French or German.

Secondary Condition (optional) • Monolingual IR using any available parallel collections.

• Bilingual IR from other languages. 2

Participants

Four research groups registered to participate in this track: • University of Alicante (Spain) • Johns Hopkins University (USA) • University of Exeter (U.K.) • ITC-irst (Italy)

Official run Site resultsEnconexp UAlicante resultsEnsinexp UAlicante aplspenena JHU/APL exeengpl1.5 UExeter exeengpl3.5 UExeter Mono-brf ITC-irst resultsFRconexp UAlicante resultsFRsinexp UAlicante aplspfrena JHU/APL exefrprnsys1.5 UExeter exefrprnsys3.5 UExeter fr-en-1bst-brf-bfr ITC-irst fr-en-sys-brf-bfr ITC-irst aplspdeena JHU/APL exedeprnsys1.5 UExeter exedeprnsys3.5 UExeter de-en-dec-1bst-brf-bfr ITC-irst de-en-sys-brf-bfr ITC-irst aplspitena JHU/APL exeitprnpro1.5 UExeter exeitprnsys1.5 UExeter it-en-1bst-brf-bfr ITC-irst it-en-sys-brf-bfr ITC-irst aplspesena JHU/APL exespprnpro1.5 UExeter exespprnsys3.5 UExeter es-en-1bst-brf-bfr ITC-irst es-en-sys-brf-bfr ITC-irst aplspnlena JHU/APL

Results and Discussion

References

[1] G. J. F. Jones and M. Federico . CLEF 2002 Cross-Language Spoken Document Retrieval Pilot Track Report . In Proceedings of the CLEF 2002: Workshop on Cross-Language Information Retrieval and Evaluation , Rome, September 2002 . Springer Verlag.

[2] J. S. Garafolo , C. G. P. Auzanne , and E. M. Voorhees . The TREC Spoken Document Retrieval Track: A Success Story . In Proceedings of the RIAO 2000 Conference: Content-Based Multimedia Information Access , pages 1 - 20 , Paris, 2000 .