In this paper we will describe Berkeley's approach to the ImageCLEF \Photo" task for CLEF 2006. This year is the rst time that we have participated in ImageCLEF, and we chose to primarily establish a baseline for the Cheshire II system for this task, while we had originally hoped to use GeoCLEF methods for this task, in the end time constraints led us to restrict our submissions to the basic required runs for the task.
This paper discusses the retrieval methods and evaluation results for Berkeley's participation in
the ImageCLEF Photo task. Our submitted runs this year are intended to establish a simple
baseline for comparision in future ImageCLEF tasks. This year we used only text-based retrieval
methods for Imageclef, totally ignoring the images themselves (and the reference images speci ed
in the queries). We hope, in future years, to be able to use combined text and image processing
approaches similar to those used in some earlier work combining Berkeley's BlobWorld image
retrieval system with the Cheshire II system (see [
This year Berkeley submitted 7 runs, of which 4 where English Monolingual, 2 German Monolingual, and 1 Bilingual English to German.
This paper rst describes the retrieval methods used, including our blind feedback method for text, followed by a discussion of our o cial submissions and the results obtained from other methods after o cial submissions had been closed. Finally we present some discussion of the results and our conclusions.
S log O(R j Q; D) = b0 + X bisi
i=1 P (R j Q; D) = 1 + elog O(RjQ;D) elog O(RjQ;D) where b0 is the intercept term and the bi are the coe cients obtained from the regression analysis of the sample collection and relevance judgements. The nal ranking is determined by the conversion of the log odds form to probabilities: 2.1
TREC2 Logistic Regression Algorithm
For all of our ImageCLEF submissions this year we used a version of the Logistic Regression (LR)
algorithm that has been used very successfully in Cross-Language IR by Berkeley researchers for a
number of years[
The basic form and variables of the Logistic Regression (LR) algorithm used for all of our
submissions was originally developed by Cooper, et al. [
c0 + c1 + c2 c3
jQcj 1 X log pjQcj + 1 i=1 1 jQcj
X log pjQcj + 1 i=1
tfi cl + 80 ctfi
Nt + c4 jQcj where C denotes a document component (i.e., an indexed part of a document which may be the entire document) and Q a query, R is a relevance variable, p(RjC; Q) is the probability that document component C is relevant to query Q, p(RjC; Q) the probability that document component C is not relevant to query Q, which is 1.0 p(RjC; Q) jQcj is the number of matching terms between a document component and a query, qtfi is the within-query frequency of the ith matching term, tfi is the within-document frequency of the ith matching term, ctfi is the occurrence frequency in a collection of the ith matching term, ql is query length (i.e., number of terms in a query like jQj for non-feedback situations), cl is component length (i.e., number of terms in a component), and Nt is collection length (i.e., number of terms in a test collection). ck are the k coe cients obtained though the regression analysis.
If stopwords are removed from indexing, then ql, cl, and Nt are the query length, document length, and collection length, respectively. If the query terms are re-weighted (in feedback, for example), then qtfi is no longer the original term frequency, but the new weight, and ql is the sum of the new weight values for the query terms. Note that, unlike the document and collection lengths, query length is the \optimized" relative frequency without rst taking the log over the matching terms.
The coe cients were determined by tting the logistic regression model speci ed in log O(RjC; Q)
to TREC training data using a statistical software package. The coe cients, ck, used for our
ofcial runs are the same as those described by Chen[
In addition to the direct retrieval of documents using the TREC2 logistic regression algorithm
described above, we have implemented a form of \blind relevance feedback" as a supplement to the
basic algorithm. The algorithm used for blind feedback was originally developed and described by
Chen [
Blind relevance feedback is typically performed in two stages. First, an initial search using the original topic statement is performed, after which a number of terms are selected from some number of the top-ranked documents (which are presumed to be relevant). The selected terms are then weighted and then merged with the initial query to formulate a new query. Finally the reweighted and expanded query is submitted against the same collection to produce a nal ranked list of documents. Obviously there are important choices to be made regarding the number of top-ranked documents to consider, and the number of terms to extract from those documents. For ImageCLEF this year, having no prior data to guide us, we chose to use the top 10 terms from 10 top-ranked documents. The terms were chosen by extracting the document vectors for each of the 10 and computing the Robertson and Sparck Jones term relevance weight for each document. This weight is based on a contingency table where the counts of 4 di erent conditions for combinations of (assumed) relevance and whether or not the term is, or is not in a document. Table 1 shows this contingency table.
The relevance weight is calculated using the assumption that the rst 10 documents are relevant and all others are not. For each term in these documents the following weight is calculated: wt = log
Rt R Rt
Nt Rt N Nt R+Rt (3) In doc Not in doc
Relevant Rt R Rt R
Not Relevant Nt Rt N Nt N R
R + Rt
Nt N N
Nt
The 10 terms (including those that appeared in the original query) with the highest wt are selected and added to the original query terms. For the terms not in the original query, the new \term frequency" (qtfi in Equation 3 above) is set to 0.5. Terms that were in the original query, but are not in the top 10 terms are left with their original qtfi. For terms in the top 10 and in the original query the new qtfi is set to 1.5 times the original qtfi for the query. The new query is then processed using the same LR algorithm as shown in Equation 3 and the ranked results returned as the response for that topic. 3
In this section we describe the speci c approaches taken for our o cial submitted runs for the ImageCLEF photo task. First we describe the indexing and term extraction methods used, and then the search features we used for the submitted runs. 3.1
Although the Cheshire II system uses the XML structure of documents and extracts selected portions of the record for indexing and retrieval, for the submitted runs this year we used only a single one of these indexes that contains the entire content of the document.
Name
Description docno Document ID title Article Title topic All Content Words date Date of Image geoname Image Place names
Content Tags
Used DOCNO TITLE DOC DATE LOCATION no no yes no no Searching the ImageCLEF collection used Cheshire II scripts to parse the topics and submit the title or title and narrative from the topics to the \topic" index containing all terms from the documents. For the monolingual search tasks we used the topics in the appropriate language (English or German), and for bilingual tasks the topics were translated from the source language to the target language using SYSTRAN (via Babel sh at Altavista). We believe that other translation tools provide a more accurate representation of the topics (like the L&H P.C. translator used in our GeoCLEF entries. However since the runs and submission were done from a hotel room over a highly variable wireless connection we did not have access to all of the tools we would have normally used. We also did a German to English translation, but the quality of translation was so poor (partially because only short titles were available) that we decided not to submit that run.
We tried two main approaches for searching, the rst used only the topic text from the title element, the second included both titles and the narrative elements. In all cases the \topic" index mentioned above was used, and probabilistic searches were carried out. Two forms of the TREC2 logistic regression algorithm were used. One used the basic algorithm as described above, and the other used the TREC2 algorithm with blind feedback using the top 10 terms from the 10 top-ranked documents in the initial retrieval.
Our o cial runs did not make use of the example images in queries, but after our o cial submissions we realized that this information could be used in a text-only system as well. This involved retrieving the associated metadata records from the example images included in the queries and using their titles and location information to expand the basic query. We tested this in uno cial \POST" runs that are also described below following the discussion of the o cial The summary results (as Mean Average Precision) for the o cial submitted bilingual and monolingual runs for both English and German are shown in Table 3, the Recall-Precision curves for these runs are also shown in Figures 1 (for monolingual) and 2 (for bilingual). In Figures 1 and 2 the name are abbrevated to the nal letters and numbers of the full name in Table 3. Table 4 has a number of uno cial runs described in the next section, and Figures 3 and 4 show the Precision and Recall curves for those runs (with some of the runs from Figures 1 and 2 for comparison. (Note also the di erences of the Y scale in these gures).
Table 3 shows all of our submitted runs for the ImageCLEF Photo task. Precision and recall curves for the runs are shown in Figures 1 and 2. 5
The o cially submitted runs described above, when compared to participants using combined text and image methods, are not particularly distinguished. The German monolingual and bilingual
Description BERK BI ENDE TN T2FB
BERK MO DE T T2 BERK MO DE T T2FB BERK MO EN T T2 BERK MO EN T T2FB BERK MO EN TN T2 BERK MO EN TN T2FB Monolingual German Monolingual German Monolingual English Monolingual English Monolingual English Monolingual English Run Name POST BI DEEN T EXP-TL POST BI ENDE T EXP-TL POST BI ENDE TN EXP-TL POST BI FREN T EXP-TL POST MO DE T EXP-TL POST MO EN T EXP-TL POST MO EN TN EXP-TL Description
Monolingual German Monolingual English Monolingual English Title+Narr Title Title Title Title Title,Narr Title,Narr Query Title Title Title+Narr Title Title Title Title+Narr
Feedback
Y N Y N Y N Y Y Y Y Y Y Y
Y Feedback 0.1054 runs su er from the lack of decompounding in this version of the system. For the purpose of establishing a baseline performance for our system the runs are adequate. One interesting observation for is that, although the MAP for runs with blind feedback was higher than matching runs without it, the precision at low recall levels was lower than runs without feedback. We suspect that this is an e ect of the small size of the metadata records available for this task when compared to the full-text in other collections. Use of the narrative for English topics allowed us to further improve performance in the o cial runs, although the same pattern for feedback versus no feedback performance as mentioned above was observed, indicative that the pattern is likely a function of the record size and not query size.
Although our o cial runs for ImageCLEF Photo 2006 are in no way outstanding, they do provide a \foot in the door" and a baseline for tuning our future performance for this task.
Quite a di erent picture appears in our \POST" runs, shown in Table 4 and in Figures 3 and 4. All of these runs use expansion of the queries based on the \TITLE" and \LOCATION" elements of the metadata annotations associated with images included in the \image" element of the queries (e.g. rather like image processing approaches, but using only text). All of these runs, had they been o cial runs, would have ranked at or near the top of the evaluation pools. (The similarity in between our MAP results and some others would seem to indicate that others were using a similar method of query expansion, perhaps in addition to image processing).
In Table 5 we compare the best performing runs, all using blind relevance feedback, for matching tasks in our o cial and uno cial runs. For Bilingual English to German we see a dramatic 146% improvement in MAP using this query expansion method. For the comparable Monolingual tasks we see improvements ranging from almost 50% to over 80%. This is strong indication that expanding queries using the metadata of relevant images is a very good strategy for the ImageCLEF Photo task.
As an further experiment, we decided to try this expansion method for Bilingual retrieval without any translation of the original topic, that is, we used the title in the original language (in this case French) and did our expansion using appropriate metadata for the target language (English). The result from this experiment is included in Table 4 as run \POST BI FREN T EXP
Monolingual German Monolingual English Monolingual English Title+Narr Title Title Title+Narr
O cial MAP TL". If that run had been submitted as an o cial run, it would have been a 186.5% improvement in MAP over the best o cial French to English submitted for the ImageCLEF Photo task.