In this paper we will briefly describe the approaches taken by the Berkeley Cheshire Group for the Adhoc-TEL 2008 tasks (Mono and Bilingual retrieval). Since the AdhocTEL task is new for this year, we took the approach of using methods that have performed fairly well in other tasks. In particular, the approach this year used probabilistic text retrieval based on logistic regression and incorporating blind relevance feedback for all of the runs. All translation for bilingual tasks was performed using the LEC Power Translator PC-based MT system. This approach seems to be a fit good for the limited TEL records, since the overall results show Cheshire runs in the top five submitted runs for all languages and tasks except for Monolingual German.
The CLEF Adhoc-TEL collections are different from most of the data used for testing in the various CLEF tasks. The three sub-collections – British Library (BL), Biblioteque Nationale de France (BNF), and the Austrian National Library (ONB) – each represent about 1 million bibliographic records from The European Library union catalog (TEL). The records, we can assume, were originally in some version of MARC (Machine Readable Cataloging) before they were converted to a much more simplified bibliographic record based on the Dublin Core metadata schema. Each of the subcollections use somewhat differing encoding of the (assumed) original MARC data, not always including all of the fields that might be useful in retrieval.
Although each the collections were considered to be “mainly” in a particular language (English for BL, French for BNF, and German for ONB), according to the language codes of the records, only about half of each collection was in that main language, with virtually all other languages represented by one or more entries in one or another of the collections. German, French, English, and Spanish records were available in all of collections. Although this overlap of languages presents an interesting multilingual search (and evaluation) problem, it was not addressed in our experiments this year.
This paper concentrates on the retrieval algorithms and evaluation results for Berkeley’s official submissions for the Adhoc-TEL 2008 track. All of the runs were automatic without manual intervention in the queries (or translations). We submitted six Monolingual runs (two German, two English, and two French) and nine Bilingual runs (each of the three main languages to both of the other main languages (German, English and French). In addition we submitted three runs from Spanish translations of the topics to the three main languages.
This paper first describes the retrieval algorithms used for our submissions, followed by a discussion of the processing used for the runs. We then examine the results obtained for our official runs, and finally present conclusions and future directions for Adhoc-TEL participation. 2
Note that this section is virtually identical to one that appears in our papers from previous CLEF
participation[
If stopwords are removed from indexing, then ql, cl, and Nt are the query length, document length, and collection length, respectively. If the query terms are re-weighted (in feedback, for example), then qtfi is no longer the original term frequency, but the new weight, and ql is the sum of the new weight values for the query terms. Note that, unlike the document and collection lengths, query length is the “optimized” relative frequency without first taking the log over the matching terms.
The coefficients were determined by fitting the logistic regression model specified in log O(R|C, Q)
to TREC training data using a statistical software package. The coefficients, ck, used for our
official runs are the same as those described by Chen[
In addition to the direct retrieval of documents using the TREC2 logistic regression algorithm
described above, we have implemented a form of “blind relevance feedback” as a supplement to the
basic algorithm. The algorithm used for blind feedback was originally developed and described by
Chen [
Blind relevance feedback is typically performed in two stages. First, an initial search using the original topic statement is performed, after which a number of terms are selected from some number of the top-ranked documents (which are presumed to be relevant). The selected terms are then weighted and then merged with the initial query to formulate a new query. Finally the reweighted and expanded query is submitted against the same collection to produce a final ranked list of documents. Obviously there are important choices to be made regarding the number of top-ranked documents to consider, and the number of terms to extract from those documents. For ImageCLEF this year, having no prior data to guide us, we chose to use the top 10 terms from 10 top-ranked documents. The terms were chosen by extracting the document vectors for each of the 10 and computing the Robertson and Sparck Jones term relevance weight for each document. This weight is based on a contingency table where the counts of 4 different conditions for combinations of (assumed) relevance and whether or not the term is, or is not in a document. Table 1 shows this contingency table.
The relevance weight is calculated using the assumption that the first 10 documents are relevant and all others are not. For each term in these documents the following weight is calculated: wt = log
Rt R−Rt
Nt−Rt N−Nt−R+Rt (4)
The 10 terms (including those that appeared in the original query) with the highest wt are selected and added to the original query terms. For the terms not in the original query, the new “term frequency” (qtfi in main LR equation above) is set to 0.5. Terms that were in the original query, but are not in the top 10 terms are left with their original qtfi. For terms in the top 10 and in the original query the new qtfi is set to 1.5 times the original qtfi for the query. The new query is then processed using the same LR algorithm as shown in Equation 4 and the ranked results returned as the response for that topic. 3
In this section we describe the specific approaches taken for our submitted runs for the AdhocTEL task. First we describe the indexing and term extraction methods used, and then the search features we used for the submitted runs. 3.1
The Cheshire II system uses the XML structure of the documents to extract selected portions for indexing and retrieval. Any combination of tags can be used to define the index contents.
Name recid names title topic anywhere date lang subject
Table 2 lists the indexes created by the Cheshire II system for the Adhoc-TEL database and the document elements from which the contents of those indexes were extracted. The “Used” column in Table 2 indicates whether or not a particular index was used in the submitted Adhoc-TEL runs. As the table shows we used only the topic index, which contains most of the content-bearing parts of records, for all of our submitted runs. M-DE-TD-T2FB M-DE-T-T2FB M-EN-TD-T2FB M-EN-T-T2FB M-FR-TD-T2FB M-FR-T-T2FB B-ENDE-TD-T2FB B-ESDE-TD-T2FB B-FRDE-TD-T2FB B-DEEN-TD-T2FB B-ESEN-TD-T2FB B-FREN-TD-T2FB B-DEFR-TD-T2FB B-ENFR-TD-T2FB B-ESFR-TD-T2FB 0.1742 0.1980 * 0.3466 * 0.2773 0.2438 * 0.1931
For all indexing we used language-specific stoplists to exclude function words and very common words from the indexing and searching. The German language runs did not use decompounding in the indexing and querying processes to generate simple word forms from compounds. The Snowball stemmer was used by Cheshire for language-specific stemming. 3.2
Searching the Adhoc-TEL collection using the Cheshire II system involved using TCL scripts to parse the topics and submit the title and description or the title alone from the topics. For monolingual search tasks we used the topics in the appropriate language (English, German, and French), for bilingual tasks the topics were translated from the source language to the target language using the LEC Power Translator PC-based machine translation system.
The scripts for each run submitted the topic elements as they appeared in the topic to the system for TREC2 logistic regression searching with blind feedback. When both the “title” and “description” topic elements were used, they were combined into a single probabilistic query. Table 3 shows which element were used in the “Type” column, T for title only and TD for title and description. 4
The summary results (as Mean Average Precision) for the submitted bilingual and monolingual runs for English German and French are shown in Table 3, the Recall-Precision curves for these runs are also shown in Figures 1 (for monolingual) and 2 (for bilingual). In Figures 1 and 2 the names for the individual runs represent the language codes, which can easily be compared with full names and descriptions in Table 3 (since each language combination has only a single run).
Table 3 indicates runs that had the highest overall MAP for the task by asterisks next to the run name.
Obviously the “weak man” in our current implementation remains monolingual German. This may be due to decompounding issues, but the higher results for title-only monolingual seem anomalous, since for each other language, the combination of title and description performed better than title alone.
In spite of this relatively poor performance in monolingual German, we had the rather surprising results that for bilingual English to German our submitted run B-ENDE-TD-T2FB was ranked third overall among the bilingual “to German” runs submitted, and our German to French bilingual run B-DEFR-TD-T2FB was ranked first in the bilingual “to French” task well ahead of our English to French run. This would seem to indicate that the our translation system works quite well with the Adhoc-TEL topics. 5
In looking at the overall results for the various Adhoc-TEL tasks, it would appear that the basic logistic regression with blind relevance feedback approach, coupled with the LEC translation system is a fairly good combination. Since Adhoc-TEL is a new task, we took a fairly conservative approach using methods that have worked well in the past.
In our experiments for other tracks (GeoCLEF for example) we reintroduced fusion approached for retrieval that performed quite well and could be easily applied to this task as well. For future work we intend to test these approaches as well as some other approaches that would incorporate external supplementary topical indexing for the books (primarily) represented by Adhoc-TEL records.
1 0.9 0.8 0.7 ion 0.6 ics 0.5 reP 0.4 0.3 0.2 0.1 0