In this paper we will describe the Berkeley approaches to the GeoCLEF tasks for CLEF 2006. This year we used two separate systems for di erent tasks. Although of the systems both use versions of the same primary retrieval algorithm they di er in the supporting text pre-processing tools used. The Retrieval Algorithms Note that this section is virtually identical to one that appears in our ImageCLEF and Domain Speci c papers. The basic form and variables of the Logistic Regression (LR) algorithm used for all of our submissions was originally developed by Cooper, et al. [5]. As originally formulated, the
This paper describes the retrieval algorithms and evaluation results for Berkeley's o cial submissions for the GeoCLEF track. Two separate systems were used for our runs, although both used the same basic algorithm for retrieval. Instead of the automatic expansion used in last year's GeoCLEF, this year we used manual expansion for a selected subset of queries for only 2 out of the 18 runs submitted. The remainder of the runs were automatic without manual intervention in the queries (or translations). We submitted 12 Monolingual runs (2 German, 4 English, 2 Spanish, and 4 Portuguese) and 6 Bilingual runs (2 English)German, 2 English)Spanish, and 2 English)Portuguese). We did not submit any Biligual X)English runs.
This paper rst describes the retrieval algorithms used for our submissions, followed by a discussion of the processing used for the runs. We then examine the results obtained for our o cial runs, and nally present conclusions and future directions for GeoCLEF participation. LR model of probabilistic IR attempts to estimate the probability of relevance for each document based on a set of statistics about a document collection and a set of queries in combination with a set of weighting coe cients for those statistics. The statistics to be used and the values of the coe cients are obtained from regression analysis of a sample of a collection (or similar test collection) for some set of queries where relevance and non-relevance has been determined. More formally, given a particular query and a particular document in a collection P (R j Q; D) is calculated and the documents or components are presented to the user ranked in order of decreasing values of that probability. To avoid invalid probability values, the usual calculation of P (R j Q; D) uses the \log odds" of relevance given a set of S statistics, si, derived from the query and database, such that: where b0 is the intercept term and the bi are the coe cients obtained from the regression analysis of the sample collection and relevance judgements. The nal ranking is determined by the conversion of the log odds form to probabilities:
S log O(R j Q; D) = b0 + X bisi
i=1 P (R j Q; D) = 1 + elog O(RjQ;D) elog O(RjQ;D) 2.1
For GeoCLEF we used a version the Logistic Regression (LR) algorithm that has been used very
successfully in Cross-Language IR by Berkeley researchers for a number of years[
c0 + c1 + c2 c3
1 jXQcj log pjQcj + 1 i=1
1 jXQcj log pjQcj + 1 i=1
tfi cl + 80 ctfi
Nt + c4 jQcj where C denotes a document component (i.e., an indexed part of a document which may be the entire document) and Q a query, R is a relevance variable, p(RjC; Q) is the probability that document component C is relevant to query Q, p(RjC; Q) the probability that document component C is not relevant to query Q, which is 1.0 p(RjC; Q) jQcj is the number of matching terms between a document component and a query, qtfi is the within-query frequency of the ith matching term, tfi is the within-document frequency of the ith matching term, ctfi is the occurrence frequency in a collection of the ith matching term, ql is query length (i.e., number of terms in a query like jQj for non-feedback situations), cl is component length (i.e., number of terms in a component), and Nt is collection length (i.e., number of terms in a test collection). ck are the k coe cients obtained though the regression analysis.
If stopwords are removed from indexing, then ql, cl, and Nt are the query length, document length, and collection length, respectively. If the query terms are re-weighted (in feedback, for example), then qtfi is no longer the original term frequency, but the new weight, and ql is the sum of the new weight values for the query terms. Note that, unlike the document and collection lengths, query length is the \optimized" relative frequency without rst taking the log over the matching terms.
The coe cients were determined by tting the logistic regression model speci ed in log O(RjC; Q)
to TREC training data using a statistical software package. The coe cients, ck, used for our
ofcial runs are the same as those described by Chen[
In addition to the direct retrieval of documents using the TREC2 logistic regression algorithm
described above, we have implemented a form of \blind relevance feedback" as a supplement to the
basic algorithm. The algorithm used for blind feedback was originally developed and described by
Chen [
Blind relevance feedback is typically performed in two stages. First, an initial search using the original topic statement is performed, after which a number of terms are selected from some number of the top-ranked documents (which are presumed to be relevant). The selected terms are then weighted and then merged with the initial query to formulate a new query. Finally the reweighted and expanded query is submitted against the same collection to produce a nal ranked list of documents. Obviously there are important choices to be made regarding the number of top-ranked documents to consider, and the number of terms to extract from those documents. For ImageCLEF this year, having no prior data to guide us, we chose to use the top 10 terms from 10 top-ranked documents. The terms were chosen by extracting the document vectors for each of the 10 and computing the Robertson and Sparck Jones term relevance weight for each document. This weight is based on a contingency table where the counts of 4 di erent conditions for combinations of (assumed) relevance and whether or not the term is, or is not in a document. Table 1 shows this contingency table.
In doc Not in doc
Rt R Rt R
Nt Rt N Nt N R
R + Rt
Nt N N
Nt
The relevance weight is calculated using the assumption that the rst 10 documents are relevant and all others are not. For each term in these documents the following weight is calculated: wt = log
Rt R Rt
Nt Rt N Nt R+Rt (4)
The 10 terms (including those that appeared in the original query) with the highest wt are selected and added to the original query terms. For the terms not in the original query, the new \term frequency" (qtfi in main LR equation above) is set to 0.5. Terms that were in the original query, but are not in the top 10 terms are left with their original qtfi. For terms in the top 10 and in the original query the new qtfi is set to 1.5 times the original qtfi for the query. The new query is then processed using the same LR algorithm as shown in Equation 4 and the ranked results returned as the response for that topic. 3
In this section we describe the speci c approaches taken for our submitted runs for the GeoCLEF task. First we describe the indexing and term extraction methods used, and then the search features we used for the submitted runs. 3.1
The standalone version treats all text as a single \bag of words" that is extracted and indexed. For German documents it uses a custom \decompounding" algorithm to extract component terms from German compounds.
The Cheshire II system uses the XML structure and extracts selected portions of the record for indexing and retrieval.
Name docno pauthor headline topic date geotext geopoint geobox
Description Document ID Author Names Article Title Content Words Date of Publication Validated place names Validated coordinates for place names Validated bounding boxes for place names
Content Tags DOCNO BYLINE, AU HEADLINE, TITLE, LEAD, LD, TI HEADLINE, TITLE, TI, LEAD BYLINE, TEXT, LD, TX DATE, WEEK TEXT, LD, TX TEXT, LD, TX TEXT, LD, TX
Used no no no yes yes no no no no
For all indexing we used language-speci c stoplists to exclude function words and very common words from the indexing and searching. The German language runs used decompounding in the indexing and querying processes to generate simple word forms from compounds.
The Snowball stemmer was used by both systems for language-speci c stemming. 3.2
All of the runs for Monolingual English and German, and the runs for Bilingual English)German used the standalone retrieval programs developed by Aitao Chen. The Monolingual Spanish and Portuguese, and the Bilingual English)Spanish and English)Portuguese runs all used the Cheshire II system.
The English and German Monolingual runs used language-speci c decompounding of German compound words. The Bilingual English)German also used decompounding.
Searching the GeoCLEF collection using the Cheshire II system involved using TCL scripts to parse the topics and submit the title and description or the title, description, and narrative from the topics. For monolingual search tasks we used the topics in the appropriate language (Spanish and Portuguese), for bilingual tasks the topics were translated from the source language to the target language using the L&H PC-based machine translation system. In all cases the various topic elements were combined into a single probabilistic query.
We tried two main approaches for searching, the rst used only the topic text from the title and desc elements (TD), the second included the narrative elements as well (TDN). In all cases only the full-text \topic" index was used for Cheshire II searching.
Two of our English Monolingual runs used manual modi cation for topics 27, 43, and 50 by adding manually selected place names to the topics, in addition, one of these (which turned out to be our best performing English Monolingual run) also manually eliminated country names from topic 50.
Also after two initial runs for Portuguese Monolingual were submitted (BKGeoP1 aand BKGeoP2), a revised and corrected version of the topics was released, and two additional runs (BKGeoP3 and BKGeoP4) were submitted using the revised topics, retaining the original submissions for comparison. 4
The summary results (as Mean Average Precision) for the submitted bilingual and monolingual runs for both English and German are shown in Table 3, the Recall-Precision curves for these runs are also shown in Figures 1 and 2 (for monolingual) and 3 and 4 (for bilingual). In Figures 1-4 the names for the individual runs represent the language code and type of run, which can be compared with full names and descriptions in Table 3.
Table 3 indicates runs that had the highest overall MAP for the task by asterisks next to the run name. Single asterisks indicate the the highest MAP values among our own runs, while double asterisks indicate the runs where the MAP is the maximum recorded among o cial submissions.
As can be seen from the table, Berkeley's cross-language submissions using titles, descriptions, and narratives from the topics were the best performing runs for the Bilingual tasks overall. Our Monolingual submissions, on the other hand did not fare as well, but still all ranked within the top quartile of results for each language except Portuguese where we fell below the mean. This result was surprising, given the good performance for Spanish. We now suspect that errors in mapping the topic encoding to the stored document encoding, or possibly problems with the Snowball stemmer for Portuguese may be responsible for this relatively poor performance.
Last year's GeoCLEF results (see [
Although we did not do any explicit geographic processing for this year, we plan to do so in the future. The challenge for next year is to be able to obtain the kind of e ectiveness improvement seen with manual query expansion, in automatic queries using geographic processing. BKGeoED1 BKGeoED2** BKGeoES1 BKGeoES2** BKGeoEP1 BKGeoEP2** BKGeoD1* BKGeoD2 BKGeoE1 BKGeoE2 BKGeoE3 BKGeoE4* BKGeoS1* BKGeoS2 BKGeoP1 BKGeoP2 BKGeoP3 BKGeoP4* Bilingual English)German Bilingual English)German Bilingual English)Spanish Bilingual English)Spanish Bilingual English)Portuguese Bilingual English)Portuguese Monolingual German Monolingual German Monolingual English Monolingual English Monolingual English Monolingual English Monolingual Spanish Monolingual Spanish Monolingual Portuguese Monolingual Portuguese Monolingual Portuguese Monolingual Portuguese TASK