In this paper we describe retrieval experiments performed at Charles University in Prague for participation in the CLEF 2007 Ad-Hoc track. We focused on the Czech monolingual task and used the LEMUR toolkit as the retrieval system. Our results demonstrate that for Czech as a highly inflectional language, lemmatization significantly improves retrieval results and manually created queries are only slightly better than queries automatically generated from topic specifications.
2.1
Being novices in the field of information retrieval we decided to use a freely available retrieval
toolkit instead of developing our own. The final choice was the LEMUR toolkit [
An inference network (also known as a Bayesian network) consists of a document node,
smoothing parameters nodes, model nodes, representation nodes, belief nodes, and information need nodes
connected by edges representing independence assumptions over random variables. The document
node represents documents as binary vectors where each position represents presence or absence
of a certain feature of the text. The model nodes correspond to different representations of the
same document (e. g. pseudo-documents made up from all titles, bodies, etc.). The representation
concept nodes are related to the features extracted from the document representation. The belief
nodes are used to combine probabilities of different representations, other beliefs, etc. A detailed
description can be found in [
To improve retrieval results, we used Indri’s pseudo-relevance feedback which is an adaption
of Lawrenko’s relevance models [
State-of-the-art retrieval systems usually include at least some basic linguistically-motivated
preprocessing of the documents and queries such as stemming and stopword removal. Czech is a
morphologically complex language and there is no easy way how to determine stems and their
endings as it can be done in English and other languages. Stemming in Czech is not sufficient
and should be replaced by a proper lemmatization (substituting each word by its base form – the
lemma) which involves determining the part of speech of all words. In our experiments, we
employed the Czech morphological analyzer and tagger developed at Charles University [
We used two approaches to construct the stopword lists for our experiments. The first was based on frequency of word occurrences in the collection, the latter on part-of-speech of words. In the first three experiments (Prague01-03), we removed 40 most frequented words (separately from the original and lemmatized text) from the documents and the queries. In the fourth experiment (Prague04), we removed all words tagged as pronouns, prepositions, conjunctions, particles, interjections, and unknown words (mostly typos) and kept only open-class words. 2.4
Automatically created queries were constructed from the <title> and <description> fields of the topic specifications only. The text was simply concatenated and processed by the analyzer and tagger. A combination of the original and lemmatized query was used in the first experiment (Prague01). Lemmatized queries containing only nouns, adjectives, numerals, adverbs and verbs were created for the fourth experiment (Prague04).
<title>Snizˇov´anı´ rizika onemocnˇen´ı cukrovkou</title> <desc>Najdˇete dokumenty zmiˇnuj´ıc´ı faktory, kter´e sniˇzujı´ riziko onemocnˇenı´ cukrovkou. </desc> Step 2. Concatenation: Step 3. Lemmatization: Sniˇzov´an´ı rizika onemocnˇen´ı cukrovkou. Najdˇete dokumenty zmiˇnuj´ıcı´ faktory, kter´e sniˇzuj´ı riziko onemocnˇen´ı cukrovkou. sniˇzov´an´ı riziko onemocnˇen´ı cukrovka naj´ıt dokument zmiˇnovat faktor kter´y sniˇzit riziko onemocneˇn´ı cukrovka Step 4. Prague01 query (original word forms plus lemmas; the suffixes .(orig) and .(lemma) reffer to the corresponding model representations): #combine(snizˇov´an´ı.(orig) rizika.(orig) onemocn´enı´.(orig) cukrovkou.(orig) najd´ete.(orig) dokumenty.(orig) zmiˇnujı´c´ı.(orig) faktory.(orig) kter´e.(orig) sniˇzuj´ı.(orig) riziko.(orig) onemocn´en´ı.(orig) cukrovkou.(orig) sniˇzov´anı´.(lemma) riziko.(lemma) onemocn´en´ı.(lemma) cukrovka.(lemma) naj´ıt.(lemma) dokument.(lemma) zmiˇnujı´cı´.(lemma) faktor.(lemma) kter.(lemma) sniˇzovat.(lemma) riziko.(lemma) onemocn´enı´.(lemma) cukrovka.(lemma)) Step 5. Prague04 query: #combine(sniˇzov´an´ı riziko onemocn´en´ı cukrovka zmiˇnuj´ıcı´ faktor sniˇzovat riziko onemocn´enı´ cukrovka) 2.5
The queries in two of our experiments were created manually. In Prague02 they were constructed from lemmas (to match the lemmatized documents) and their synonyms and in Prague03 with the use of “stems“ and wildcard operators to cover all possible word forms (documents indexed in the original forms).
<title>Sniˇzov´an´ı rizika onemocnˇen´ı cukrovkou</title> <desc>Najdˇete dokumenty zmiˇnuj´ıc´ı faktory, kter´e sniˇzujı´ riziko onemocnˇenı´ cukrovkou. </desc> Step 2. The Prague02 query based on lemmas (the operator #combine() combines beliefs of the nested operators, operator #syn() represets synonymic line of equal expressions and operator #2() represents ordered window with width 2 words): #combine(#syn(diabetes cukrovka ´uplavice) #2(sn´ıˇzenı´ riziko) prevence) Step 3. The Prague03 query with wildcard operators (which can be used as a suffix only). #combine(diabet* cukrovk* ´uplavic* sn´ıˇz* rizik* preven*)
Prague01 Topic fields: <title>, <desc> Query construction: automatic Document fields: <title>, <heading>, <text> Word forms: original + lemmas Stop words: 40 most frequent original forms + 40 most frequent lemmas Prague02 Topic fields: <title>, <desc> Query construction: manual Document fields: <title>, <heading>, <text> Word forms: lemmas Stop words: 40 most frequent lemmas Prague03 Topic fields: <title>, <desc> Query construction: manual (with wildcard operators) Document fields: <title>, <heading>, <text> Word forms: original Stop words: 40 most frequent word forms Prague04 Topic fields: <title>, <desc> Query construction: automatic Document fields: <title>, <heading>, <text> Word forms: lemmas Stop words: pronouns, prepositions, conjunctions, particles, interjections, and unknown words 4
The Czech Ad-Hoc collection consists of 81,735 documents and 50 topics. The following table summarizes the results for the experiments described above.
Mean Average Precision Mean R Precision Mean Binary Preference Precision at 10 interpolated recall level
Prague01 0.3419 0.3201 0.2977 0.5733
Prague02 0.3336 0.3349 0.3022 0.6314
Prague03 0.3202 0.3147 0.2801 0.5299
Prague04 0.2969 0.2886 0.2601 0.5367
In terms of Mean Average Precision, the best score was achieved in experiment Prague01. Indexing both original word forms and lemmas in combination with automatically generated queries seems to be a reasonable way how to build a retrieval system. In terms of other performance measures, the scores of Prague02 are slightly better but this is probably due to the use of synonyms in the manually created queries – not in the manual approach itself.
By comparing scores of Prague03 with results of Prague01 and Prague02 we can confirm that lemmatization is quite useful for searching in highly flectional languages such a Czech and can not be fully substituted by stemming.
The last lesson we learned is that using extensive stopword lists based on part-of-speech can seriously harm the performance of a retrieval system as can bee seen on the results of experiment Prague04.
We found these results quite encouraging and motivating for our future work.
This work has been supported by the Ministry of Education of the Czech Republic, projects MSM 0021620838 and #1P05ME786.