This year is the first year that the Information Retrieval Group at the University of Waterloo participated in CLEF. For the Cross-Language Spoken Document Retrieval track we submitted five official runs -three English automatic runs (title-only, title+desc, and title+desc+narr), a Czech automatic run (title-only) and a French automatic run (title-only). All official runs used a combination of several query formulation and expansion techniques, including phonetic n-grams and pseudo-relevance feedback expansion over a topic-specific external corpus crawled from the Web. In addition, a large number of un-official runs were generated, including German and Spanish runs. This brief report provides an overview of our experiments, which are summarized in figure 1.
All our runs were generated by the Wumpus retrieval system 1 using Okapi BM25 as the basic retrieval method.
The Wumpus implementation of Okapi BM25 is a variant of the formula given by Robertson et al. [3]. Given a term set Q, a document d is assigned the score:
where D = number of documents in the corpus D t = number of documents containing t q t = frequency that t occurs in the topic
All CLEF 2005 runs used parameter settings of k 1 = 1.2 and b = 0.75. Many of our runs incorporated pseudo-relevance feedback, following the process described in Yeung et al. [1]. For feedback purposes, we augmented the CLEF 2005 SDR corpus with a 2.5GB corpus of Web data, generated by a topic-focused crawl, seeded from 17 sites dedicated to the holocaust. Each query was first executed against this augmented corpus. Terms were extracted from the top results and added to the initial query, which was then executed against the SDR Corpus.
As an alternative to stemming, many runs were based on phoneme 4-grams. For these runs, NIST's text-to-phone tool2 was applied to translate the words in the corpus into phoneme sequences, which were then split into 4-grams and indexed. Queries were pre-processed in a similar fashion before execution.
Several runs, including our official English-language submissions, were generated by fusing word and n-gram runs. For these runs, fusion was performed using the standard CombMNZ algorithm [2].
Our non-English runs used translated queries supplied by the University of Ottawa group. The reader should consult their CLEF 2005 paper for further information.
On the training data, the fusion of feedback and phonetic n-gram runs produced a substantial performance improvement over the baseline Okapi runs. Unfortunately, this the improvement was not seen on the test data, where feedback produced only a modest improvement and the phonetic n-grams generally harmed performance.
Next year, we hope to expand our participation in CLEF, including the evaluation of additional speech-specific techniques in the context of the SDR track.