In this paper we will describe Berkeley's approach to the Domain Speci c (DS) track for CLEF 2006. This year we are not using the tools for thesaurus-based query expansion and de-compounding for German that were developed over the past many years and used very successfully in earlier Berkeley entries in this track. Our intent has been to incorporate those tools into the Cheshire system, but we were unable to complete the development in time for use in the o cially submitted runs. This year Berkeley submitted 12 runs, including one for each subtask of the DS track. These include 3 Monolingual runs for English, German, and Russian, 7 Bilingual runs (3 X2EN, 1 X2DE, and 3 X2RU), and 2 Multilingual runs. For many DS sub-tasks our runs were the best performing runs, but sadly they were also the only runs for a number of subtasks. In the sub-tasks where there were other entries, our relative performance was above the mean performance in 2 sub-tasks and just below the mean in another.
This paper discusses the retrieval methods and evaluation results for Berkeley's participation in the Domain Speci c track. Our submitted runs this year are intended to establish a new baseline for comparision in future Domain Speci c evaluations for the Cheshire system, and did not use the techniques of thesaurus-based query expansion or German decompounding used in previous years. This year we used only basic probabilistic retrieval methods for DS tasks. We hope, in future years, (assuming that the task will continue in future years) to be able to use those, or re nements of those techniques via the Cheshire II or Cheshire3 systems.
This year Berkeley submitted 12 runs, including one for each subtask of the DS track. These include 1 Monolingual run for each of English, German, and Russian for a total of 3 Monolingual runs, and 7 Bilingual runs (3 X2EN, 1 X2DE, and 3 X2RU), and 2 Multilingual runs.
This paper rst very brie y describes the retrieval methods used, including our blind feedback method for text, which are discussed in greater detail in our ImageCLEF paper. We then describe our submissions for the various DS sub-tasks and the results obtained. Finally we present conclusions and discussion of future approaches to this track. 2
As we have discussed in our other papers for the ImageCLEF and GeoCLEF tracks in this volume,
basic form and variables of the Logistic Regression (LR) algorithm used for all of our submissions
were originally developed by Cooper, et al. [
S log O(R j Q; D) = b0 + X bisi
i=1 P (R j Q; D) = 1 + elog O(RjQ;D) elog O(RjQ;D) where b0 is the intercept term and the bi are the coe cients obtained from the regression analysis of a sample set of queries, a collection and relevance judgements. The nal ranking is determined by the conversion of the log odds form to probabilities: 2.1
TREC2 Logistic Regression Algorithm
For all of our Domain Speci c submissions this year we used a version of the Logistic Regression
(LR) algorithm that has been used very successfully in Cross-Language IR by Berkeley researchers
for a number of years[
c0 + c1 + c2 c3
1 jXQcj log pjQcj + 1 i=1 1 jQcj
X log pjQcj + 1 i=1
tfi cl + 80 ctfi
Nt + c4 jQcj where C denotes a document component (i.e., an indexed part of a document which may be the entire document) and Q a query, R is a relevance variable, p(RjC; Q) is the probability that document component C is relevant to query Q, p(RjC; Q) the probability that document component C is not relevant to query Q, which is 1.0 p(RjC; Q) jQcj is the number of matching terms between a document component and a query, qtfi is the within-query frequency of the ith matching term, tfi is the within-document frequency of the ith matching term, ctfi is the occurrence frequency in a collection of the ith matching term, ql is query length (i.e., number of terms in a query like jQj for non-feedback situations), cl is component length (i.e., number of terms in a component), and Nt is collection length (i.e., number of terms in a test collection). ck are the k coe cients obtained though the regression analysis.
More details of this algorithm and the coe cients used with it may be found in our
ImageCLEF notebook paper where the same algorithm and coe cients were used. In addition to this
primary algorithm we used a version that performs \blind feedback" during the retrieval process.
The method used is described in detail in our ImageCLEF notebook paper. Our blind feedback
approach uses the 10 top-ranked documents from an initial retrieval using the LR algorithm above,
and selects the top 10 terms from the content of those documents, using a version of the Robertson
and Sparck Jones probabilistic term relevance weights [
In this section we describe the speci c approaches taken for our submitted runs for the Domain Speci c track. First we describe the indexing and term extraction methods used, and then the search features we used for the submitted runs. 3.1
Although the Cheshire II system uses the XML structure of documents and extracts selected portions of the record for indexing and retrieval, for the submitted runs this year we used only a single one of these indexes that contains the entire content of the document.
Name docno title topic date geoname subject
Description Document ID Article Title All Content Words Date Geographic names Controlled Vocabulary
TITLE DOC DATE GEOGR-AREA, COUNTRY-CODE CONTROLLED-TERM-x, CLASSIFICATION-TEXT-x Used no no yes no no no
Table 1 lists the indexes created for the Domain Speci c database and the document elements from which the contents of those indexes were extracted. The \Used" column in Table 1 indicates whether or not a particular index was used in the submitted Domain Speci c runs. This year we ion 0.6 ics 0.5 reP 0.4 0.3 0.2 0.1 0
DE
EN
RU
did not use the Entry Vocabulary Indexes (search term recommender) that were used by Berkeley
in previous years (see [
For all indexing we used language-speci c stoplists to exclude function words and very common words from the indexing and searching. The German language runs, however, did not use decompounding in the indexing and querying processes to generate simple word forms from compounds (actually we tried, but there was a bug that failed to match any compounds in our runs). This is another aspect of our indexing for this year's Domain Speci c task that reduced our results relative to last year. 3.2
Searching the Domain Speci c collection used Cheshire II scripts to parse the topics and submit the title and description elements from the topics to the \topic" index containing all terms from the documents. For the monolingual search tasks we used the topics in the appropriate language (English, German, or Russian), and for bilingual tasks the topics were translated from the source language to the target language using SYSTRAN (via Babel sh at Altavista.com) or PROMT via the PROMT web interface. We believe that other translation tools provide a more accurate representation of the topics for some languages (like the L&H P.C. translator used in our GeoCLEF entries) but that was not available to us for our o cial runs for this track this year. Wherever possible we used both of these MT systems and submitted the resulting translated queries as separate runs. However, some translations are only available on one system or the other (e.g., German Rightarrow Russian is only available in PROMT). All of our runs for this track used the TREC2 algorithm as described above with blind feedback using the top 10 terms from the 10 top-ranked documents in the initial retrieval. 4
The summary results (as Mean Average Precision) for the submitted bilingual and monolingual runs for both English and German are shown in Table 2, the Recall-Precision curves for these runs are also shown in Figure 1 (for monolingual and multilingual) and Figures 2 and 3 (for bilingual). In Figures 1, 2, and 3 the names are abbrevated to the letters and numbers of the full name in Table 2 describing the languages and translation system used. For example, in Figure 3 DERU+P corresponds to BERK BI DERU T2FB P in Table 2. ion 0.6 ics 0.5 reP 0.4
1 0.9 0.8 0.7 0.3 0.2 0.1 0
DEEN+B DEEN+P RUEN+B ion 0.6 ics 0.5 reP 0.4
Table 2 shows all of our submitted runs for the Domain Speci c track. Precision and recall curves for the runs are shown in Figures 1 and 2 for the Monolingual and Multilingual and Bilingual tasks respectively. Although Berkeley's results for these tasks are high, compared to the overall average MAP for all participants, not much can be claimed for this since there were very few submissions for the Domain Speci c track this year, and for many tasks (Multilingual, Monolingual Russian, Bilingual X)Russian, and Bilingual X)English) Berkeley was the only group that submitted runs. A few observations concerning translation are worth mentioning. First is that where we have comparable runs using di erent translations, all of the processing except for the topic translation was identical. Thus, it is obvious that for translations from German to English, Babel sh does a better job, and for English to Russian PROMT does a better job. We could not get PROMT (online version) to successfully translate the Russian topics to English apparently due to invalid character codes in the input topics (which were not detected by Babel sh).
It is perhaps more interesting to compare our results with last year's result, where Berkeley
was also quite successful (see [
Babel sh PROMT Babel sh PROMT Babel sh PROMT none none none
Babel sh Babel sh 0.23658 0.33013* 0.31763 0.32282* 0.10826* 0.11554 0.16482** 0.39170 0.41357 0.25422** BERK BI ENDE T2FB B BERK BI DEEN T2FB B BERK BI DEEN T2FB P BERK BI RUEN T2FB B BERK BI DERU T2FB P BERK BI ENRU T2FB B BERK BI ENRU T2FB P BERK MO DE T2FB BERK MO EN T2FB BERK MO RU T2FB Bilingual English)German Bilingual German)English Bilingual German)English Bilingual Russian)English Bilingual German)Russian Bilingual English)Russian Bilingual English)Russian
Monolingual English Monolingual Russian BERK MU DE T2FB B CMBZ
that the lack of those is what led to this relative worse performance.
The results of these comparisons with our 2005 Domain Speci c results are shown in Table 3. The only exception to the large percentage improvements seen by our best 2006 runs over the Berkeley1 2005 runs is found in Bilingual German)Russian. We suspect that there were translation problems for the German topics using PROMT, but we lack su cient language skills in each language to understand what all these problems were. But we did observe that a large number of German terms were not translated, and that many spurious additional characters appeared in the translated texts. Another di erence worth noting is that the Berkeley2 group used the L&H PowerTranslator software for it's English to X translations, while this year we used only Babel sh and PROMT. 5
Given the small number of submissions for the Domain Speci c track this year, we wonder about the viability of the track for the future. Berkeley's runs this year did not build on the successes of the Berkeley2 group from 2005, but instead worked only to establish a new baseline set of results for future for retrieval processing without query expansion using EVMs or thesaurus information. This obviously hurt our comparable overall performance in tasks with submissions from elsewhere. We did, however, see a marked improvement in performance for our speci c system (Cheshire II was also used for Berkeley1 last year) with the improvements to our probabilistic retrieval algorithms developed after last year's submissions. We suspect that with the further addition of decompounding for German, and the use of EVMs and thesaurus expansion we can match or exceed the performance of the Berkeley2 runs last year.