For the multilingual ad-hoc document retrieval track (TEL@CLEF) at at the Cross-Language Retrieval Forum (CLEF) Trinity College Dublin and Dublin City University participated in collaboration. Our retrieval experiments focus on i) investigating document expansion using an entry vocabulary module, ii) translating queries with Google translate and a statistical MT system, and iii) investigating language modeling as a retrieval method. The major results are that the document expansion approach did not increase MAP; topic translation using the statistical MT system resulted in about 70% of the mean average precision (MAP) achieved when using Google translate for topic translation, and language modeling performs equally or better in comparison with BM25. The bilingual retrieval French and German to English experiments obtained 89% and 90% of the best MAP for monolingual English.
The TEL (The European Library) task at CLEF is concerned with ad-hoc information retrieval (IR) [
Our IR experiments for the ad-hoc task at CLEF 2009 aim at investigating several aspects of retrieval:
1. employing and evaluating EVM [
The Lemur toolkit1 was employed to index and retrieve documents. Two different retrieval models were
employed: BM25 [
For most runs, pseudo-relevance feedback was applied for query expansion (QE): the top ten ranked
documents and 30 terms were used for BM25 and the top five documents and 20 added terms for LM. A
variant of query expansion using information an external resource was also explored (QE2) for bilingual
retrieval. The top 10 results for the query in the source language were extracted and translated with Google
translate. Highly co-occurring terms were extracted for query expansion [
For the bilingual retrieval experiments, topics were translated using either Google translate (GT) 3 or a
statistical machine translation system (MT) [
The TEL collection contains documents of ranging from very short documents, because the presence or absence of a field with bibliographic information leads to documents with varying length. Furthermore, some fields contain information not in natural language (i.e. alphanumeric codes or classifications). The 1http://www.lemurproject.org/ 2http://snowball.tartarus.org/ 3http://translate.google.com/ 0.4900 0.2400 0.2500 main idea behind some of our experiments was to apply a document expansion method to obtain longer documents.
In the English documents, there are more than 585,000 fields containing a valid Dewey Decimal Code (DDC), and about 50% of all documents contain a corresponding field. The percentage of LCC in the English collection is considerably lower and DDC and LCC for the German and French collection are not present or occur only sparsely, so the experiments were focused on the DDC classification and the English document collection.
Before conducting the official experiments, we performed a test experiment with the English TEL documents. We divided the document collection into short (less than 80 characters) and long documents and into document with a DDC and without. Results for the test run based on CLEF 2008 data for the different sets of documents are shown in Table 1. For short and long documents, retrieval performance is very similar, but less short documents were assessed as relevant. In contrast, documents with DDC make up a large portion of relevant documents (72%), while about half of all English documents are associated with a DDC. Possible explanations might be that in previous experiments, the DDC has been treated as a separate index term which could be used in relevance feedback or that longer documents provide more context for relevance assessment. The relative and absolute performance for documents without DDC classification is lower. As a result of this analysis, we tried to expand documents via an automatic DDC classification to create documents with a more evenly distributed length.
The DDC is a hierarchical library classification. The classification system defines ten main classes, 100 divisions, and 1000 sections, each denoted by digits. For example, the DDC 627 represents the main class “technology”, division “engineering and applied operations”, section “hydraulic engineering”.
The main idea for document expansion was to train a classifier on documents containing a DDC and apply it to obtain classification codes for all other documents. All classification codes are then replaced with their natural language description, which is preprocessed and added to the index. The natural language descriptions are available in English only and originate from the OCLC web site4. The natural language description for these codes was compiled into a machine-readable format using the sources from OCLC. The resulting description contained all 1110 entries for the DDC of which 933 were actually used in the document collection. The documents were modified as follows: documents with a DDC are expanded by appending the natural language description of the DDC to their content; documents without a DDC are first classified using an EVM and then processed as described above.
Entry Vocabulary Modules (EVM, [
3
EN EN EN EN FR FR FR FR DE DE DE DE DE DE FR FR
FR Results for the ad-hoc IR experiments are shown in Table 2. Some experiments achieved a performance among the top five participants at the TEL track at CLEF 2009, i.e. run TCDDCU DEEN1 was 4th in bilingual English (0.3333 MAP), run TCDDCU DE3 was 4th in monolingual German (0.2686 MAP), and run TCDDCU EN3 was 5th in monolingual English (0.3696 MAP).
In all cases, runs with blind relevance feedback to expand queries yield a higher MAP compared to the corresponding runs without blind feedback. The query expansion variant based on external information from web pages found by Google web search did not show the expected results as it degraded the performance (TCDDCU DEEN3 vs. TCDDCU DEEN1).
Obviously using only a subset of the document fields yields a slightly higher precision (e.g. TCDDCU DE3 vs. TCDDCU DE4).
BM25 and language modeling perform similar for the retrieval experiments in all languages. Because of small differences in the experimental setup (e.g. the fields indexed), some additional experiments will have to be conducted before testing for significant differences.
For the bilingual runs with target language English, 89.9% and 90.1% of the MAP for the best monolingual English runs was achieved for French and German, respectively. Using the MaTrEx system for topic translation achieves a MAP of 70.1% in comparison to topic translation by Google translate (TCDDCU FREN2 vs. TCDDCU FREN1). 4
Future work will include an analysis of the accuracy of the DDC classification based on a manually extracted and annotated sample of the English document collection. Also, blind relevance feedback using external resources will be further investigated.
Thanks to Andy Way’s group at DCU for providing topic translations with the MaTrEx statistical MT system.
This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (CNGL) project.