This paper describes a system built at Charles University in Prague for participation in the CLEF 2007 Cross-Language Speech Retrieval track. We focused only on monolingual searching the Czech collection and used the LEMUR toolkit as the retrieval system. We employed own morphological tagger and lemmatized the collection before indexing to deal with the rich morphology in Czech which significantly improved our results.
2.1
The LEMUR toolkit [
An inference network (also known as a Bayesian network) consists of a document node,
smoothing parameters nodes, model nodes, representation nodes, belief nodes, and information need nodes
connected by edges representing independence assumptions over random variables. The document
node represents documents as binary vectors where each position represents presence or absence
of a certain feature of the text. The model nodes correspond to different representations of the
same document (e. g. pseudo-documents made up from all titles, bodies, etc.). The representation
concept nodes are related to the features extracted from the document representation. The belief
nodes are used to combine probabilities of different representations, other beliefs, etc. A detailed
description can be found in [
To improve retrieval results, we used Indri’s pseudo-relevance feedback which is an adaption of
Lawrenko’s relevance models [
State-of-the-art retrieval systems usually include at least some basic linguistically-motivated
preprocessing of the documents and queries such as stemming and stopword removal. Czech is a
morphologically complex language and there is no easy way how to determine stems and their
endings as it can be done in English and other languages. Stemming in Czech is not sufficient
and should be replaced by a proper lemmatization (substituting each word by its base form – the
lemma) which involves determining the part of speech of all words. In our experiments, we
employed the Czech morphological analyzer and tagger developed at Charles University [
We used two approaches to construct the stopword lists for our experiments. The first was based on frequency of word occurrences in the collection, the latter on part-of-speech of words. In the first three experiments (Prague01-03), we removed 40 most frequented words (separately from the original and lemmatized text) from the documents and the queries. In the fourth experiment (Prague04), we removed all words tagged as pronouns, prepositions, conjunctions, particles, interjections, and unknown words (mostly typos) and kept only open-class words. 2.4
Automatically created queries were constructed from the <title> and <description> fields of the topic specifications only. The text was simply concatenated and processed by the analyzer and tagger. A combination of the original and lemmatized query was used in the first experiment (Prague01). Lemmatized queries containing only nouns, adjectives, numerals, adverbs and verbs were created for the fourth experiment (Prague04).
<title>Dˇetske´ umˇen´ı v Terez´ınˇe</title> <desc>Hled´ame popis umˇeleck´ych aktivit dˇet´ı v Terez´ınˇe, jako napˇr. hudby, divadla, malov´anı´, poezie a jin´ych psan´ych dˇel.</desc> Step 2. Concatenation: Step 3. Lemmatization: Dˇetsk´e umˇen´ı v Terez´ınˇe. Hled´ame popis umˇeleck´ych aktivit dˇetı´ v Terez´ıneˇ, jako napˇr. hudby, divadla, malov´anı´, poezie a jin´ych psan´ych dˇel. dˇetsk´y umˇen´ı v Terez´ın hledat popis umˇeleck´y aktivita dˇeti v Terez´ın jako napˇrı´klad hudba divadlo malov´an´ı poezie a jin´y psan´y d´ılo Step 4. Prague01 query (original word forms plus lemmas; the suffixes .(orig) and .(lemma) reffer to the corresponding model representations): #combine(dˇetske´.(orig) umˇen´ı.(orig) v.(orig) Terez´ınˇe.(orig) hled´ame.(orig) popis.(orig) umˇelecky´ch.(orig) aktivit.(orig) dˇet´ı.(orig) v.(orig) Terez´ıneˇ.(orig) jako.(orig) napˇr.(orig) hudby.(orig) divadla.(orig) malov´an´ı.(orig) poezie.(orig) a.(orig) jin´ych.(orig) psan´ych.(orig) dˇel.(orig) dˇetsk´y.(lemma) umˇen´ı.(lemma) v.(lemma) Terezı´n.(lemma) hledat.(lemma) popis.(lemma) umˇeleck´y.(lemma) aktivita.(lemma) d´ıtˇe.(lemma) v.(lemma) Terez´ın.(lemma) jako.(lemma) napˇrı´klad.(lemma) hudba.(lemma) divadlo.(lemma) malov´an´ı.(lemma) poezie.(lemma) a.(lemma) jin´y.(lemma) psan´y.(lemma) dˇelo.(lemma)) Step 5. Prague04 query: #combine(dˇetsky´ umˇen´ı Terez´ın hledat popis umˇeleck´y aktivita d´ıtˇe Terez´ın napˇrı´klad hudba divadlo malov´anı´ poezie jin´y psan´y dˇelo) 2.5
The queries in two our experiments were created manually. In Prague02 they were constructed from lemmas (to match the lemmatized documents) and their synonyms and in Prague03 with the use of “stems“ and wildcard operators to cover all possible word forms (documents indexed in the original forms).
Step 1. The original title and description (topic 1173: Children’s art in Terezin): <title>Dˇetsk´e umˇen´ı v Terez´ınˇe</title> <desc>Hled´ame popis umˇeleck´ych aktivit dˇet´ı v Terez´ınˇe, jako napˇr. hudby, divadla, malov´anı´, poezie a jin´ych psan´ych dˇel.</desc> Step 2. The Prague02 query based on lemmas (the operator #combine() combines beliefs of the nested operators, operator #syn() represets synonymic line of equal expressions and operator #2() represents ordered window with width 2 words): #combine(#syn(d´ıtˇe dˇetsky´) umˇen´ı divadlo hudba #syn(malov´an´ı kreslen´ı) #syn(malovat kreslit) poezie b´asniˇcka) Step 3. The Prague03 query with wildcard operators (which can be used as a suffix only). #combine(dˇet* umˇen´ı divad* hud* malov* kresl* poez* b´asn*)
Prague01 Topic fields: <title>, <desc> Query construction: automatic Document fields: <title>, <heading>, <text> Word forms: original + lemmas Stop words: 40 most frequent original forms + 40 most frequent lemmas Prague02 Topic fields: <title>, <desc> Query construction: manual Document fields: <title>, <heading>, <text> Word forms: lemmas Stop words: 40 most frequent lemmas Prague03 Topic fields: <title>, <desc> Query construction: manual (with wildcard operators) Document fields: <title>, <heading>, <text> Word forms: original Stop words: 40 most frequent word forms Prague04 Topic fields: <title>, <desc> Query construction: automatic Document fields: <title>, <heading>, <text> Word forms: lemmas Stop words: pronouns, prepositions, conjunctions, particles, interjections, and unknown words 4
All our experiments were performed on the Quickstart collection provided by the track coordinators. 356 holocaust survivors testimonies in Czech were automatically transcribed by an ASR system and the output segmented into 11,373 overlapping passages used as ”documents“. Word error rate of the ASR system is approximately 35%. 29 topics and their relevance assessment were available for training and other 42 topics used for the evaluation. The following table summarizes the results (mGAP scores) for the experiments described above separately for training and evaluation topics.
Mean GAP Mean GAP
topics 42 evaluation 29 training
Prague01 0.0187 0.0266
Prague02 0.0181 0.0322
Prague03 0.0102 0.0328
Prague04 0.0190 0.0277
Interpretation of these results is quite difficult mainly because of the difference between performance on the training and evaluation topics. One possible explanation of this discrepancy is that the relevance judgments for the training topics were obtained only by search-guided assessment and not by highly ranked assessment. Thus we consider the results on the evaluation data more credible.
The best score on evaluation topics was achieved in experiment Prague04 but it is almost indistinguishable from scores of other experiments that employed lemmatization (Prague01 and Prague02). In all these experiments we achieved significantly better results than in experiment Prague03 where we indexed the original word forms (no lemmatization). This observation is also consistent with our results in Ad-Hoc tracks.
The best score on the training topics was achieved in experiment Prague03 but it is almost identical to the result in experiment Prague02. In both these experiments we used manually created queries and they significantly outperformed experiments with queries automatically generated from the topic specifications. However, we can not conclude that manually constructed queries are better because these results were not confirmed on the evaluation set of topics.
The results we achieved are quite promising and we will continue exprimenting with this valuable collection.
This work has been supported by the Ministry of Education of the Czech Republic, projects MSM 0021620838 and #1P05ME786.