The paper describes the participation of the CACAO project consortium to the TEL@CLEF 2008 task targeted at retrieving relevant items from collections of library catalogues. CACAO proposes the development of an infrastructure for multilingual access to digital content, including an information retrieval system able to search for books and texts in all the available languages. For each monolingual and bilingual subtask two different experiments have been conducted, one involving additional query expansion and one not. Results evidenced a poor initial performance of the system, however since the project started few months ago they can constitute a valuable baseline in order to measure the future advancement of the system. The TEL@CLEF is a new task proposed this year to the CLEF campaign participants with the aim of searching and retrieving relevant items from collections of library catalogues. The data is different from the corpora previously used in the CLEF ad hoc tracks and consists of bibliographic metadata divided into 3 collections, extracted from British, French and Austrian national libraries. TEL@CLEF task offers a set of subtasks reflecting the multilinguality of the data, respectively focusing on monolingual and bilingual information retrieval; 50 topics has been prepared for each of the 3 main collection languages and each topic has 2 fields: a title with 2-4 key terms and a description field, containing a sentence that specifies in more detail the information needs of the user. In our participation to the TEL@CLEF task we focused both on the monolingual and on the bilingual retrieval subtasks since CACAO project aims at offering a cross-language acces to the contents of a federation of digital libraries and the tasks constitute a perfect opportunity to test the baseline version of the CACAO system prototype and obtain feedbacks for its enhancement. This paper is organized as follows. We present the architecture of our system in Section 2, in Section 3 we describe our experiments, the evaluation measures and the evaluation results, and finally conclude in Section 4.
as in the MACS approach, https://macs.vub.ac.be/pub/). The system will then yield results in all desired languages.
Validation is another important aspect in the project: all CACAO core technologies are indeed sound, but they have never been massively deployed in the field of digital libraries. CACAO aims at crossing the chasm between sound innovation and adoption by library institutions for real life purposes. 2.1
• • • • 3 CACAO proposes the development of an infrastructure for multilingual access to digital content, including an information retrieval system able to search for books and texts in all the available languages. The core of the search engine takes advantage of information contained in existing catalogues and texts of the digital libraries that is enriched by means of NLP techniques such as word sense disambiguation and named entities recognition. The goal of such integration is to avoid confusing the user by providing irrelevant results due to bad translations and thus enabling a better access to the digital content.
The general architecture of the Cacao system could be summarized as the result of the interactions of few functional subsystems, coordinated by a central manager and reacting to external stimuli represented by end users queries:
Harvesting subsystem is in charge of collecting data from digital libraries, abstracting from the multiplicity of standards and protocols, and storing them into a repository.
Corpus Analysis subsystem performs specific analysis on the data collected from libraries and infers new information used to support query processing and resource retrieval (e.g. query expansion, terms disambiguation,..).
Web Services subsystem represents third party software providing specific services (e.g. linguistic analysis, translations,..).
Query Processing subsystem: a set of components is devoted to process the original monolingual user query, transforming and enriching it by means of translations and expansions.
For each monolingual and bilingual subtask two different experiments have been conducted, one involving additional query expansion and one not; in the following subsections the experiments set-up, topic processing and experimental results are described. 3.1
In order to acquire the metadata of TEL@CLEF collections into the CACAO system, GoNetwork s.r.l. (one of
the CACAO partners) deployed a specific harvester module for importing the XML corpus documents. The
textual information contained in the dc:subject, dc:title and dc:dexcription has been lemmatised using the XIP
incremental parser from XEROX (see [
By means of lexical semantics technologies (we exploited Random Indexing approach, see [
The approach adopted by CACAO system for dealing with user queries is based on the free keywords search; therefore while the title field of TEL topics already fitted this model, the description field has been processed in order to extract a set of relevant keywords from the sentence. For this purpose a simple keyword extractor module has been used for each of the main languages present in the corpus (English, French and German). Each description sentence has been analysed in order to extract two different kinds of information, one representing the content type of the items to be retrieved (as novels, poetry or photo collections) and the other conveying additional detail on user interests.
The keywords retrieved in this process have been lemmatised and the system assigned different weight to them according to their frequency in both of the topic fields (title and description). In the lemmatisation process named entities were also identified, as they have been treated differently from common keywords with respect to translation to target languages. According to the subtasks (monolingual or bilingual) the keywords were translated to the target language or directly submitted to the Lucene search engine.
The translation process exploited internal resources (inter-lingual indexes or bilingual dictionaries) and online
dictionaries as Ergane (see [
As a first step the system automatically groups keywords in sets of semantically related terms by comparing their
similarity, defined as the cosine of the angle between the vector representations of the terms. This process allows
the system to group together all the keywords bearing a common meaning; then the translation candidates of
each keywords group are analysed in order to prune away all the elements with a low similarity with the center
of the translation group, computed as the sum of the vector representation of terms (a variation of the algorithm
proposed by [
Experiments involving query expansion enriched the keywords groups (either in the original or in the target language) exploiting the corpus based semantic vectors by adding the N nearest neighbours of each group center (not already present in the keyword set), where the actual value of N depended on the cardinality of the keyword group. We submitted two runs for each monolingual and bilingual subtask for each target language for a total of 18 runs. The results of these experiments are provided in the following tables where results are separated for target language.
CACAO project has started only few months ago and despite the early stage of development of the system prototype the consortium decided to participate to the TEL task of the CLEF campaign in order to asses a baseline of the system and hence measure the advancement of the system during the project timeline. Compared to other participants the experimental results evidenced a poor initial performance of the CACAO system; however such poor performance is partly due to a misinterpretation of the task: we in fact assumed that the task was completely partitioned according to languages. Therefore when performing a bilingual subtask (i.e. the task DE->FR on the TEL data of BNF, the French National Library) we assumed that only the elements in the target language should be retrieved, while the golden standard over which evaluation was performed actually contains a not negligible percentage of results in other languages. All these records were not even searched, dramatically lowering recall and other measures. 5
This work has been supported and founded by CACAO EU project (ECP 2006 DILI 510035). 6