=Paper=
{{Paper
|id=Vol-1173/CLEF2007wn-DomainSpecific-PetrasEt2007
|storemode=property
|title=The Domain -Specific Track at CLEF 2007
|pdfUrl=https://ceur-ws.org/Vol-1173/CLEF2007wn-DomainSpecific-PetrasEt2007.pdf
|volume=Vol-1173
|dblpUrl=https://dblp.org/rec/conf/clef/PetrasBS07a
}}
==The Domain -Specific Track at CLEF 2007==
https://ceur-ws.org/Vol-1173/CLEF2007wn-DomainSpecific-PetrasEt2007.pdf
The Domain-Specific Track at CLEF 2007
Vivien Petras, Stefan Baerisch, Maximillian Stempfhuber
GESIS Social Science Information Centre, Lennéstr. 30, 53113 Bonn, Germany
{vivien.petras | stefan.baerisch | max.stempfhuber@gesis.org}
Abstract
The domain-specific track uses test collections from the social science domain to
test monolingual and cross-language retrieval in structured bibliographic databases.
Special attention is given to the existence of controlled vocabularies for content
description and their potential usefulness in retrieval. Test collections and topics are
provided in German, English and Russian. This year, a new English test collection
(from the CSA Sociological Abstracts database) was added. We present an overview
of the CLEF domain-specific track including a description of the tasks, collections,
topic preparation, and relevance assessments as well as contributions to the track. A
summary of results is given. The track participants experimented with different
retrieval models ranging from classic vector-space to probabilistic to language
models. The controlled vocabularies were used for query expansion or as bilingual
dictionaries for query translation.
Categories and Subject Descriptors
H.3 [Information Storage and Retrieval]: H.3.1 Content Analysis and Indexing; H.3.3
Information Search and Retrieval
General Terms
Measurement, Performance, Experimentation
Keywords
Information Retrieval, Evaluation, Controlled Vocabularies
1 Introduction
The CLEF domain-specific track evaluates mono- and cross-language information retrieval on
structured scientific data. A point of emphasis in this track is research on leveraging the structure
of data in collections (i.e. controlled vocabularies and other metadata) to improve search. In recent
years, the focus of the domain-specific data collections was on bibliographic databases in the
social science domain.
The domain-specific track was established at the inception of CLEF in 2000 and was funded by
the European Union from 2001-2004 (Kluck & Gey, 2001; Kluck, 2004). It is now continued at
the GESIS German Social Science Information Centre (Bonn) in cooperation with the DELOS
Network of Excellence on Digital Libraries.
The GIRT databases (now in version 4) are extracts from the German Social Science Information
Centre’s SOLIS (Social Science Literature) and SOFIS (Social Science Research Projects)
databases from 1990-2000. In 2005, the Russian Social Science Corpus (RSSC) was added as a
Russian-language test collection (94,581 documents), which was changed in 2006 to the INION
ISISS corpus covering social sciences and economics in Russian. This year, another English-
language social science collection was added. The second English collection is an extract from
CSA’s Sociological abstracts providing more documents and another thesaurus to the test bed.
In addition to the four test collections, various controlled vocabularies and mappings between
vocabularies were made available. As is standard for the domain-specific track, 25 topics were
prepared in German and then translated into English and Russian.
2 The Domain-Specific Task
The domain-specific track includes three subtasks:
• Monolingual retrieval against the German GIRT collection, the English GIRT and CSA
Sociological Abstract collections, or the Russian INION ISISS collection;
• Bilingual retrieval from any of the source languages to any of the target languages;
• Multilingual retrieval from any source language to all collections / languages.
2.1 The Test Collections
In recent years, pseudo-parallel collections in German and English (GIRT) and one or two Russian
test collections were provided (Kluck & Stempfhuber, 2005; Stempfhuber & Baerisch, 2006). This
year, only one Russian but two English collections were provided.
Every test collection is in the format of a bibliographic database (records include title, author,
abstract and source information) with the addition of subject metadata from controlled
vocabularies.
German
The German GIRT collection (the social science German Indexing and Retrieval Testdatabase) is
now used in its forth version (Girt-description, 2007) with 151,319 documents covering the years
1990-2000 using the German version of the Thesaurus for the Social Sciences. Almost all
documents contain an abstract (145,941).
English
The English GIRT collection is a pseudo-parallel corpus to the German GIRT collection,
providing translated versions of the German documents. It also contains 151,319 documents using
the English version of the Thesaurus for the Social Sciences but only 17% (26,058) documents
contain an abstract.
New additions this year were the documents from the social science database Sociological
Abstracts from Cambridge Scientific Abstracts (CSA) with 20,000 documents, 94% of which
contain an abstract. The documents were taken from the SA database covering the years 1994,
1995, and 1996. Additional to title and abstract, each document contains subject-describing
keywords from the CSA Thesaurus of Sociological Indexing Terms and classification codes from
the Sociological Abstracts classification.
Russian
For Russian retrieval, the INION corpus ISISS with bibliographic data from the social sciences
and economics with 145,802 documents was once again used. ISISS documents contain authors,
titles, abstracts (for 27% of the test collection or 39,404 documents) and keywords from the Inion
Thesaurus.
2.2 Controlled Vocabularies
The GIRT collections have assigned descriptors from the GESIS IZ Thesaurus for the Social
Sciences in German and English depending on the collection language. The CSA Sociological
Abstracts documents contain descriptors from the CSA Thesaurus of Sociological Indexing Terms
and the Russian ISISS documents are provided with Russian INION Thesaurus terms. GIRT
documents also contain classification codes from the GESIS IZ classification and CSA SA
documents from the Sociological Abstracts classification. Table 1 shows the distribution of
subject-describing terms per document in each collection.
Collection GIRT-4 CSA INION ISISS
(German or Sociological
English) Abstracts
Thesaurus descriptors
10 6.4 3.9
/ document
Classification codes /
2 1.3 n/a
document
Table 1. Distribution of subject-describing terms per collection
Vocabulary mappings
Additional to the “mapping table” for the German and English terms from the GESIS IZ
Thesaurus for the Social Sciences, which is really a translation, a bidirectional mapping between
the GIRT and CSA Thesauri was provided.
Vocabulary mappings are one-directional, intellectually created term transformations between two
controlled vocabularies. They can be used to switch from the subject metadata terms of one
knowledge system to the other, enabling a retrieval system to treat the subject descriptions of two
or more different collections as one and the same. This year’s mappings were equivalence
transformations, showing only term mappings that were found to be equivalent between two
different controlled vocabularies.
We provided mappings between the German Thesaurus for the Social Sciences and the English
CSA Thesaurus of Sociological Indexing Terms. Since the German Thesaurus for the Social
Sciences exists in an English version as well, we also provided the mapping from the English
Thesaurus for the Social Sciences to the English CSA Thesaurus of Sociological Indexing Terms
for monolingual retrieval.
An example for a mapping from the English Thesaurus for the Social Sciences to the English CSA
Thesaurus of Sociological Indexing Terms would be:
agricultural area
Rural areas
This example shows that a mapping can overcome differences in technical language and the
treatment of singular and plural in different controlled vocabularies.
2.3 Topic Preparation
As is standard for the CLEF domain-specific track, 25 topics were prepared.
For topic preparation we were supported by our colleagues from the GESIS Social Science
Information Centre. As a special service to the social science community in Germany, the
Information Centre biannually publishes updates on new entries in the SOLIS and SOFIS
databases (from which the GIRT collections were generated). The specialized updates are prepared
in 28 subject categories by subject specialists working at the Centre. Topics range from general
sociology, family research, women’s and gender studies, international relations, research on
Eastern Europe to social psychology and environmental research. An overview of the service
including the 28 topics can be found at the following URL:
http://www.gesis.org/en/information/soFid/index.htm.
We asked our colleagues to think of between 2-5 topics related to their subject area and potentially
relevant in the years 1990-2000 (the coverage of our test collections). The suggestions from 15
different colleagues were then checked according to breadth, variance from previous years and
coverage in the test collections. 25 topics were selected and edited into the CLEF topic XML
format. Figure 1 is an example for a topic.
All topics were created in German and then consequently translated into English and Russian.
192
System change and family planning in East Germany
Find documents describing birth trends and family
planning since reunification in East Germany.
Of interest are documents on demographic changes which
have taken place after 1989 in the territory of the former GDR as well
as the slump in birth numbers, decline in marriages and
divorces.
Figure 1. Example topic in English
Table 2 lists all 25 topic titles in English to give a perspective on the variance in topics.
Sibling relations Class-specific leisure behaviour
Unemployed youths without vocational Mortality rate
training Economic elites in Eastern Europe and Russia
German-French relations after 1945 System change and family planning in East
Multinational corporations Germany
Partnership and desire for children Gender and career chances
Torture in the constitutional state Ecological standards in emerging or
Family policy and national economy developing countries
Women and income level Integration policy
Lifestyle and environmental behaviour Tourism industry in Germany
Unstable employment situations Promoting health in the workplace
Value change in Eastern Europe Economic situations of families
Migration pressure European climate policy
Quality of life of elderly persons Economic support in the East
Table 2. Topic titles for domain-specific CLEF track 2007
To date, 200 topics have been created for the domain-specific track.
3 Overview of the 2007 Domain-Specific Track
More details of the individual runs and methods employed can be found in the corresponding
articles by the participating groups.
3.1 Participants
Although 10 groups had registered for the domain-specific task, only 5 groups submitted runs.
Four groups have submitted descriptions to the working notes so far (Clinchant and Renders,
2007; Fautsch et al., 2007; Kürsten and Eibl, 2007; Larson, 2007). Table 3 lists the participants.
Abbreviation Group Institution Country
Media Informatics, Chemnitz University
Chemnitz Germany
of Technology
Cheshire School of Information, UC Berkeley USA
Xerox Research Centre Europe - Data
Xerox France
Mining Group
Moscow Moscow State University Russia
Computer Science Department,
Unine Switzerland
University of Neuchatel
Table 3. Domain-specific track 2007 - participants
3.2 Submitted Runs
Experiments for all tasks (monolingual, bilingual and multilingual retrieval) were submitted to the
track. Monolingual and bilingual experiments were equally attempted, whereas multilingual
retrieval runs were only submitted by 2 groups. Russian remains slightly less popular than the
other two languages. Table 4 provides the number of submitted runs per task, table 5 provides an
overview over submitted runs per task per participant.
Task Runs
Monolingual
- against German 13
- against English 15
- against Russian 11
Bilingual
- against German 14
- against English 15
- against Russian 9
Multilingual 9
Table 4. Submitted runs per task in the domain-specific track
Task Participants (Runs)
Monolingual
- against German Chemnitz (3), Cheshire (2), Unine (4), Xerox (4)
- against English Chemnitz (3), Cheshire (2), Moscow (2), Unine (4), Xerox (4)
- against Russian Chemnitz (3), Cheshire (2), Moscow (2), Unine (4)
Bilingual
- against German Chemnitz (4), Cheshire (4), Xerox (6)
- against English Chemnitz (3), Cheshire (4), Moscow (2), Xerox (6)
- against Russian Chemnitz (3), Cheshire (4), Moscow (2)
Multilingual Chemnitz (3), Cheshire (6)
Table 5. Submitted runs per task and participant
3.3 Relevance Assessments
In previous years, the domain-specific relevance assessments were administrated and overseen at
least partly in-house at the Social Science Information Centre (using a self-developed Java-Swing
program). This year all relevance assessments were administered and processed in the DIRECT
system (Distributed Information Retrieval Evaluation Campaign Tool) provided by Giorgio M. Di
Nunzio and Nicola Ferro from the Information Management Systems (IMS) Research Group at the
University of Padova, Italy.
This provided tremendous assistance for the CLEF group at the Information Centre and was
positively accepted by the five assessors. Some problems occurred because of bandwidth and
execution problems, but overall the assessment stage went smoothly.
Documents were pooled using the top 100 ranked documents from each submission. Table 6
shows the pool sizes for each language.
German 16,288
English 17,867
Russian 14,473
Table 6. Pool sizes in the domain-specific track
For the German assessments, 652 documents per topic were judged on average and about 22%
were found relevant. However, assessments vary from topic to topic. Figure 2 shows the German
assessments per topic.
German Relevance Assessments
1200
Documents
1000
Relevant
800
600
400
200
0
176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200
Figure 2. German assessments per topic
For the English assessments, 715 documents per topic were judged on average and about 25%
were found relevant.
For the Russian assessments, 3 topics were found to have no relevant documents in the ISISS
collection: 178, 181 and 191. For the assessments, 579 documents per topic were judged and only
10% were found relevant.
Figures 3 and 4 show the English and Russian relevance assessments numbers.
English Relevance Assessments
1200
1000 Documents
Relevant
800
600
400
200
0
176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200
Figure 3. English assessments per topic
Russian Relevance Assessments
1000
900 Documents
800 Relevant
700
600
500
400
300
200
100
0
176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200
Figure 4. Russian assessments per topic
At first glance, several topics seem to yield particularly many relevant documents over all 3
languages despite different collections (e.g. 188, 190, 195) whereas others seem to yield
particularly few (e.g. 181, 191). One explanation might be the timeliness and specificity of topics.
The topics yielding many relevant topics (Quality of life of elderly persons, Mortality rate,
Integration policy) seem to be rather broad and ongoing themes in the social science literature. The
other two topics (Torture in the constitutional state, Economic elites in Eastern Europe and Russia)
could be considered more specific and geared towards more recent time frames than others.
4 Domain-Specific Experiments
Every group used the controlled vocabularies and structured data in some facility or other. One
point of emphasis was query expansion with the help of the subject description provided by the
thesauri. However, the translation and mapping tables were also used as bilingual dictionaries for
the cross-language experiments.
4.1 Retrieval models
The Chemnitz group (Kürsten and Eibl, 2007) used a redesigned version of their retrieval system
based on the Lucene API and utilized two indices in retrieval: a structured index (taking the
structure of the documents into account) and a plain index without considering the structure of the
documents. To combine the two indices, a data fusion approach using the z-score introduced by
the Unine group (Savoy, 2004) was employed. They found that the unstructured indexed
outperformed the structured one.
The Berkeley group (Larson, 2007) used a probabilistic model employing a logistic regression
algorithm successfully used for cross-language retrieval since TREC-2 and implemented it with
the Cheshire retrieval system.
Unine (Fautsch et al., 2007) used several retrieval models for comparison purposes: the classical tf
idf vector space model, probabilistic retrieval with the Okapi algorithm and four variants of the
DFR (Divergence from Randomness) approach as well as a language modelling approach. Data
fusion was applied using the z-score to combine these different models. They also compared word-
based and n-gram indexing for retrieval with the Russian language corpus.
The Xerox group (Clinchant and Renders, 2007) used a language modelling approach for their
retrieval experiments.
4.2 Language Processing for Documents and Queries
Standard language processing for documents and queries in the form of stopword-removal and
stemming or normalization was employed by all groups. The Unine group successfully developed
a new light-weight stemmer for the Russian language.
For the German language, Unine and Xerox used a decompounding module to split German
compounds whereas Berkeley and Chemnitz did not.
4.3 Query Expansion
Three of the groups focused on query expansion in some way or another. Berkeley used a version
of Entry Vocabulary Indexes (Gey et al., 2001) based on the same logistic regression algorithm as
their retrieval system to associate title and description terms from topics with controlled
vocabulary terms from documents. Another approach was a thesaurus-lookup where title and
description words were looked up in a thesaurus that combined all subject-describing keywords
from the different collections. The terms from the controlled vocabularies were added to the query.
As part of its standard retrieval process, the Cheshire system also implemented a blind feedback
algorithm based on the Robertson and Sparck Jones term weights. Whereas the Entry Vocabulary
Index approach worked better for the English target language, the thesaurus look-up worked better
for German and Russian.
Unine used the Thesaurus for the Social Sciences to enhance queries with terms from the
thesaurus. Thesaurus entries were indexed as documents and retrieved in response to query terms,
then simply added to the query. They also used blind query feedback with Rocchio’s formula as
well as an idf-based approach described in Abdou and Savoy (2007). The blind feedback approach
improved the average precision of results, whereas the thesaurus expansion did not.
Xerox used lexical entailment to provide query expansion whereby a language modelling approach
is employed to find similar terms from corpus documents in relation to query terms. They found
that this approach outperformed simple blind feedback but a combined approach worked best.
4.4 Translation
Another focus of research was query translation, where the provided mapping tables were utilized
as bilingual dictionaries.
Berkeley used the commercially available LEC Power Translator program for translation in all
languages.
Chemnitz implemented a translation-plug-in to their Lucene retrieval system utilizing well-known
freely-available translation services like Babel Fish, Google Translate, PROMT and Reverso. They
also used the bilingual mapping table from the thesauri for translation.
Finally, Xerox compared their Statistical Machine Translation System MATRAX with a
sophisticated language-model-based approach of dictionary adaptation. Dictionary adaptation
attempts to select one out of several translation possibilities for a term using a bilingual dictionary
and calculating the probability of a target term given the language context of the source query
term. They found that this approach worked well compared to the statistical machine translation
system tested.
5 Results
In the Appendix of this volume, mean average precision numbers (MAP) for each run per task and
recall-precision graphs for the top-performing runs for each task are listed.
6 Outlook
This year’s experiments have shown that leveraging a controlled vocabulary for query expansion
or translation can improve results in structured test collections. A new collection and new
vocabulary (CSA Sociological Abstracts) was added and a mapping table between the CSA
Thesaurus and the GIRT Thesaurus provided for experiments. As new collections are added and
distributed search across several collections becomes more common, the seamless switching
between controlled vocabularies becomes crucial to utilize expansion and translation techniques
developed for individual collections.
For this purpose, several resources for terminology mapping have been developed at the German
Social Science Information Centre (KoMoHe Project Website, 2007). Among them are over 40
bidirectional mappings between various controlled vocabularies. A web service to retrieve mapped
terms is being developed. Besides the expansion of test collections, these vocabulary mapping
services could be a future branch of research for the domain-specific track within CLEF.
Acknowledgements
We would like to thank Cambridge Scientific Abstracts for providing the documents for the new
Sociological Abstracts test collection.
We greatly acknowledge the support of Natalia Loukachevitch and her colleagues from the
Research Computing Center of M.V. Lomonosov Moscow State University in translating the
topics into Russian as well as providing parts of the Russian relevance assessments.
Very special thanks also to Giorgio Di Nunzio and Nicola Ferro from the Information
Management Systems (IMS) Research Group at the University of Padova for providing the
DIRECT system and all their help in the assessments process and for providing the graphs and
numbers for the results analysis.
Claudia Henning and Jeof Spiro did the German and English assessments; Monika Gonser and
Oksana Schäfer provided the rest of the Russian assessments.
References
Abdou, S. and Savoy J. (2007). Searching in Medline: Stemming, query expansion, and manual
indexing evaluation. Information Processing & Management, to appear.
Stephane Clinchant and Jean-Michel Renders (2007). XRCE’s Participation to CLEF 2007
Domain-specific Track. This volume.
Claire Fautsch, Ljiljana Dolamic, Samir Abdou and Jacques Savoy (2007). Domain-Specific IR
for German, English and Russian Languages. This volume.
Fredric Gey, Michael Buckland, Aitao Chen, and Ray Larson (2001). Entry vocabulary – a
technology to enhance digital search. In Proceedings of HLT2001, First International Conference
on Human Language Technology, San Diego, pages 91–95, March 2001.
Girt Description (2007). GIRT - Mono- and Cross-language Domain-Specific Information
Retrieval (GIRT4). http://www.gesis.org/en/research/information_technology/girt4.htm
Michael Kluck and Frederik C. Gey (2001). The Domain-Specific Task of CLEF - Specific
Evaluation Strategies in Cross-Language Information Retrieval . In: Carol Peters (ed.): Cross-
Language Information Retrieval and Evaluation. Workshop of the Cross-Language Information
Evaluation Forum, CLEF 2000, Lisbon, Portugal, September 21-22, 2000, Revised Papers.
Berlin/Heidelberg/New York: Springer 48-56 (Lecture Notes in Computer Science, 2069)
Michael Kluck (2004). The GIRT Data in the Evaluation of CLIR Systems – from 1997 until
2003. In: Peters, C., Gonzalo, J., Braschler, M., Kluck, M. (Eds..) Comparative Evaluation of
Multilingual Information Access Systems. 4th Workshop of the Cross-Language Evaluation
Forum, CLEF 2003, Trondheim, Norway, August 21-22, 2003, Revised Selected Papers.
Berlin/Heidelberg/New York: Springer 2004, 379-393 (Lecture Notes in Computer Science, 3237)
Michael Kluck and Maximilian Stempfhuber (2005). Domain-Specific Track CLEF 2005:
Overview of Results and Approaches, Remarks on the Assessment Analysis. Working Notes for
the CLEF 2005 Workshop, 21-23 September, Vienna, Austria. http://www.clef-
campaign.org/2005/working_notes/workingnotes2005/kluck05.pdf
KoMoHe Project Website (2007). Competence Center Modeling and Treatment of Semantic
Heterogeneity. http://www.gesis.org/en/research/information_technology/komohe.htm
Jens Kürsten and Maximilian Eibl (2007). Domain-Specific Cross Language Retrieval: Comparing
and Merging Structured and Unstructured Indices. This volume.
Ray Larson (2007). Experiments in Classification Clustering and Thesaurus Expansion for
Domain Specific Cross-Language Retrieval. This volume.
Savoy, Jacques (2004). Data Fusion for Effective European Monolingual Information Retrieval.
Working Notes for the CLEF 2004 Workshop, 15-17 September, Bath, UK. http://www.clef-
campaign.org/2004/working_notes/WorkingNotes2004/22.pdf
Maximilian Stempfhuber and Stefan Baerisch (2006). Domain-Specific Track CLEF 2005:
Overview of Results and Approaches, Remarks on the Assessment Analysis. Working Notes for
the CLEF 2006 Workshop, 20-22 September, Alicante, Spain. http://www.clef-
campaign.org/2006/working_notes/workingnotes2006/stempfhuberOCLEF2006.pdf