Czech Monolingual Information Retrieval Using O -The-Shelf Components - the University of West Bohemia at CLEF 2007 Ad-Hoc track Pavel Ircing 0 Ludek Muller 0 Monolingual Ad-Hoc Information Retrieval University of West Bohemia

The paper provides a brief description of the system assembled for the CLEF 2007 Ad-Hoc track by the University of West Bohemia. We have performed only monolingual experiments (Czech documents - Czech queries) using two incarnations of the tf.idf model | one with raw term frequency and the other with the BM25 term frequency weighting | as implemented in the Lemur toolkit. The e ect of the blind relevance feedback was also explored. Czech morphological analyser and tagger were used for lemmatization and stop word removal. The results achieved seem to be quite reasonable, with MAP ranging from 0.11. to 0.30.

 -

Experimentation

 Introduction

Although our group is mainly interested in the CL-SR track in the CLEF campaign, we could not resist participating in Ad-Hoc once our native language was introduced to the track. Our runs were generated essentially just by putting together o -the-shelf components available either for Czech NLP or general IR. Such seemingly unambitious approach has, however, proven to be quite successful in the past CLEF campaigns. We have performed monolingual Czech experiments only. 2.1

Linguistic preprocessing

Stemming (or lemmatization) is considered to be vital for good IR performance. This assumption was experimentally proven by our group also for the Czech language IR in the last year's CLEF CL-SR track [ 3 ]. Thus we have used the same method of linguistic preprocessing, that is, the serial combination of Czech morphological analyser and tagger [2], which provides both the lemma and stem for each input word form, together with a detailed morphological tag. This tag (namely its rst position) is used for stop-word removal | we removed from indexing all the words that were tagged as prepositions, conjunctions, particles and interjections. 2.2

Retrieval

All our retrieval experiments were performed using the Lemur toolkit [1], which o ers a variety of retrieval models. We have decided to stick to the tf.idf model where both documents and queries are represented as weighted term vectors d~i = (wi;1; wi;2; ; wi;n) and ~qk = (wk;1; wk;2; ; wk;n), respectively (n denotes the total number of distinct terms in the collection). The inner-product of such weighted term vectors then determines the similarity between individual documents and queries. There are many di erent formulas for computation of the weights wi;j , we have tested two of them, varying in the tf component: Raw term frequency wi;j = tfi;j log d dfj where tfi;j denotes the number of occurrences of the term tj in the document di (term frequency), d is the total number of documents in the collection and nally dfj denotes the number of documents that contain tj .

BM25 term frequency wi;j =

k1 tfi;j tfi;j + k1(1 b + b llCd ) log d dfj where tfi;j , d and dfj have the same meaning as in (1), ld denotes the length of the document, lC the average length of a document in the collection and nally k1 and b are the parameters to be set.

The tf components for queries are de ned analogously, except for the average length of a query, which obviously cannot be determined as the system is not aware of the full query set and processes one query at a time. The Lemur documentation is however not clear about the exact way of handling the lC value for queries.

The values of k1 and b were set according to the suggestions made by [ 5 ] and [ 4 ], that is k1 = 1:2 and b = 0:75 for computing document weights and k1 = 1 and b = 01 for query weights.

We have also tested the in uence of the blind relevance feedback. The simpli ed version of the Rocchio's relevance feedback implemented in Lemur [ 5 ] was used for this purposes. The original Rocchio's algorithm is de ned by the formula ~qnew = ~qold + ~ dR ~ dR where R and R denote the set of relevant and non-relevant documents, respectively, and d~R and d~R denote the corresponding centroid vectors of those sets. In other words, the basic idea behind this algorithm is to move the query vector closer to the relevant documents and away from the non-relevant ones. In the case of blind feedback, the top M documents from the rst-pass run are simply considered to be relevant. The Lemur modi cation of this algorithm sets the = 0 and keeps only the K top-weighted terms in d~R.

1This is actually not a choice, as the value of b is hard-set to 0 for queries in Lemur. (1) (2)

 Experimental Evaluation

There were 50 topics de ned for Ad-Hoc track, in a variety of languages. As we have already mentioned, we have used only the Czech topics for searching Czech documents. The document set consists of electronic versions of articles from two nationwide newspapers (Mlada Fronta Dnes, Lidove Noviny); following the track organisers' instructions, we have indexed only the <TITLE> and <TEXT> elds, in both the original (non-lemmatized) and the lemmatized version.

The results are summarized in Table 1. The upper section shows the MAP for queries constructed by concatenating the tokens (either words or lemmas) from the <title> and <desc> elds of the topics (TD), the lower section then the results for queries made from all three topic elds, i.e. <title>, <desc> and <narr> (TDN). Both results with (BRF) and without (no FB) application of the blind relevance feedback are shown.

Raw TF no FB BRF

BM25 TF no FB BRF TD TDN words 0.1405 lemmas 0.1765 words 0.1491 lemmas 0.1869 Our participation in the Ad-Hoc track was motivated mainly by two factors | we wanted to enrich the diversity of the pool of results and we wanted to know how our quite strong experience of dealing with Czech language processing and a rather poor experience of designing IR systems will hold up in competition. While we have hopefully succeeded in the former, we still have no idea how we have done in the latter as the organisers did not publish any cross-site comparison. Thus we look forward to seeing such ranking in the track overview paper.

 Acknowledgments References

This work was supported by the Grant Agency of the Czech Academy of Sciences project No. 1ET101470416 and the Ministry of Education of the Czech Republic project No. LC536. [1] Carnegie Mellon University and the University of Massachusetts. The Lemur Toolkit for Language Modeling and Information Retrieval. (http://www.lemurproject.org/), 2006. [2] Jan Hajic. Disambiguation of Rich In ection. (Computational Morphology of Czech).

Karolinum, Prague, 2004.

 [3] Pavel Ircing and Ludek Muller. Bene t of Proper Language Processing for Czech Speech Retrieval in the CL-SR Task at CLEF 2006 . In C. Peters, P. Clough , F. Gey , J. Karlgren , B. Magnini , D. Oard , M. de Rijke, and M. Stempfhuber, editors, Evaluation of Multilingual and Multi-modal Information Retrieval - 7th Workshop of the Cross-Language Evaluation Forum, CLEF 2006, Lecture Notes in Computer Science , Alicante, Spain, 2007 . [4] Stephen Robertson and Steve Walker . Okapi/Keenbow at TREC-8. In The Eight Text REtrieval Conference (TREC-8) , 1999 . [5] Chengxiang Zhai . Notes on the Lemur TFIDF model. Note with Lemur 1.9 documentation , School of CS, CMU, 2001 .