The paper presents an overview of the system build and experiments performed for the CLEF 2007 CL-SR track by the University of West Bohemia. We have concentrated on the monolingual experiments using the Czech collection only. The approach that was successfully employed by our team in the last year's campaign (simple tf.idf model with blind relevance feedback, accompanied with solid linguistic preprocessing) was used again but the set of performed experiments was broadened.
Experimentation
The Czech subtask of the CL-SR track, which was rst introduced at CLEF 2006 campaign,
is enormously challenging | let us repeat once again that the goal is to identify appropriate
replay points (that is, the moments where the discussion about the queried topics starts) in a
continuous stream of text generated by automatic transcription of spontaneous speech. Therefore,
it is neither the standard document retrieval task (as there are no true documents de ned) nor the
fully- edged speech retrieval (since the participants do not have the speech data nor the lattices,
so they can't explore alternative hypotheses and must rely on one-best transcription). However,
in order to lower the barrier of entry for teams pro cient at classic document retrieval (or, for
that matter, even total IR beginners), the last year's organisers prepared a so called Quickstart
collection with arti cially de ned \documents" that were created by sliding 3-minute window over
the stream of transcriptions with a 2-minute step (i.e., the consecutive documents have a one
minute overlap).1 The last year's Quickstart collection was further equipped with both manually
1It turned out later that the actual timing was di erent due to some faulty assumptions during the Quickstart
collection design, but since the principle of the document creation remains the same, we will still use the \intended"
time gures instead of the actual ones, just for the sake of readability.
and automatically generated keywords (see [
Our current system largely builds upon the one that was successful in the last year's campaign [3], with only minor modi cations and larger set of tested settings. 2.1
Stemming (or lemmatization) is considered to be vital for good IR performance even in the case of weakly in ected languages such as English; thus it is probably even more crucial for Czech as the representative of the richly in ectional language family. This assumption was experimentally proven by our group in the last year's CLEF CL-SR track [3]. Thus we have used the same method of linguistic preprocessing, that is, the serial combination of Czech morphological analyser and tagger [2], which provides both the lemma and stem for each input word form, together with a detailed morphological tag. This tag (namely it's rst position) is used for stop-word removal | we removed from indexing all the words that were tagged as prepositions, conjunctions, particles and interjections. 2.2
All our retrieval experiments were performed using the Lemur toolkit [1], which o ers a variety of retrieval models. We have decided to stick to the tf.idf model where both documents and queries are represented as weighted term vectors d~i = (wi;1; wi;2; ; wi;n) and ~qk = (wk;1; wk;2; ; wk;n), respectively (n denotes the total number of distinct terms in the collection). The inner-product of such weighted term vectors then determines the similarity between individual documents and queries. There are many di erent formulas for computation of the weights wi;j , we have tested two of them, varying in the tf component: Raw term frequency wi;j = tfi;j log d dfj where tfi;j denotes the number of occurrences of the term tj in the document di (term frequency), d is the total number of documents in the collection and nally dfj denotes the number of documents that contain tj .
BM25 term frequency wi;j =
k1 tfi;j tfi;j + k1(1 b + b llCd ) log d dfj where tfi;j , d and dfj have the same meaning as in (1), ld denotes the length of the document, lC the average length of a document in the collection and nally k1 and b are the parameters to be set.
The tf components for queries are de ned analogously, except for the average length of a query, which obviously cannot be determined as the system is not aware of the full query set and processes one query at a time. The Lemur documentation is however not clear about the exact way of handling the lC value for queries. (1) (2)
The values of k1 and b were set according to the suggestions made by [
We have also tested the in uence of the blind relevance feedback. The simpli ed version of the
Rocchio's relevance feedback implemented in Lemur [
We have created 3 di erent indices from the collection | using original data and their lemmatized and stemmed version. There were 29 training topics and 42 evaluation topics de ned by the organisers. We have rst run the set of experiments for the training topics (see Table 1), comparing: Results obtained for the queries constructed by concatenating the tokens (either words, lemmas or stems) from the <title> and <desc> elds of the topics (TD - upper section of the table) with results for queries made from all three topic elds, i.e. <title>, <desc> and <narr> (TDN - lower section).
Results achieved on the \original" Quickstart collection (i.e. 3-minute window with 1-minute overlap - Segments 3-1) with results computed using the collection created by using 2-minute window with 1-minute overlap (Segments 2.1).
In all cases the performance of raw term frequency (Raw TF) and BM25 term frequency (BM25
TF) is tested, both with (BRF) and without (no FB) application of the blind relevance feedback.
The mean Generalized Average Precision (mGAP) is used as the evaluation metric | the details
about this measure can be found in [
Segments 3-1
Raw TF BM25 TF no FB BRF no FB BRF words 0.0105 0.0121 0.0088 0.0121 lemmas 0.0168 0.0189 0.0126 0.0126 stems 0.0188 0.0205 0.0132 0.0161 words 0.0113 0.0142 0.0089 0.0108 lemmas 0.0205 0.0226 0.0114 0.0150 stems 0.0215 0.0215 0.0092 0.0107
Two minute \documents" seem to perform better than the three minute ones | probably the three minute segmentation is too coarse.
The simplest raw term frequency weighting scheme generally outperforms the more sophisticated BM25 | one possible explanation is that in a standard document retrieval setup the BM25 scheme pro ts mostly from its length normalization component that is completely unnecessary in our case (remember that our documents all have approximately identical length by design).
The fact that both stemming and lemmatization boost the performance by about the same margin was already observed in the last year's experiments. 4
In the CLEF 2007 CL-SR task, we have made just a little step further towards successful searching of Czech spontaneous speech. In order to make a bigger progress, we would need to really take the speech part of the task into account | that is, to use the speech recognizer lattices when searching for the desired information, or even to modify the ASR components so that it will be more likely to produce output useful for IR (for example, enrich the language model with rare named entities that are currently often being misrecognized).
This work was supported by the Grant Agency of the Czech Academy of Sciences project No. 1ET101470416 and the Ministry of Education of the Czech Republic project No. LC536. [1] Carnegie Mellon University and the University of Massachusetts. The Lemur Toolkit for Language Modeling and Information Retrieval. (http://www.lemurproject.org/), 2006. [2] Jan Hajic. Disambiguation of Rich In ection. (Computational Morphology of Czech).
Karolinum, Prague, 2004. [3] Pavel Ircing and Ludek Muller. Bene t of Proper Language Processing for Czech Speech Retrieval in the CL-SR Task at CLEF 2006. In C. Peters, P. Clough, F. Gey, J. Karlgren, B. Magnini, D. Oard, M. de Rijke, and M. Stempfhuber, editors, Evaluation of Multilingual and Multi-modal Information Retrieval - 7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006, Lecture Notes in Computer Science, Alicante, Spain, 2007.