This paper reports the outcomes of a longitudinal study on the CLEF Ad Hoc track in order to assess its impact in the last fteen years on the e ectiveness of monolingual, bilingual and multilingual information access and retrieval systems.
Experimental evaluation has been a key driver for research and innovation in the Information Retrieval (IR) eld since its inception. Large-scale evaluation campaigns such as Conference and Labs of Evaluation Forum (CLEF)1, are known to act as catalysts for research by o ering carefully designed evaluation tasks for di erent domains and use cases and, over the years, to have provided both qualitative and quantitative evidence about which algorithms, techniques and approaches are most e ective.
As a consequence, some attempts have been made to determine their
impact [
Therefore, we carry out a longitudinal study on the Ad-Hoc track of CLEF
in order to understand its impact on monolingual, bilingual, and multilingual
? The extended version of this abstract has been published in [
For this study we apply standardization to Average Precision (AP) calculated for all the runs submitted to the ad-hoc tracks of CLEF (i.e. monolingual, bilingual and multilingual tasks from 2000 to 2007) and to The European Library (TEL) tracks (i.e. monolingual and bilingual tasks from 2008 to 2009).
All the CLEF results that we analysed in this paper are available through
the Distributed Information Retrieval Evaluation Campaign Tool (DIRECT)
system2 [
In the following we report the main research questions we tackled and we
provide a short summary of the main ndings for each of them. More detailed
experimental results concerning those research questions could be found in [
The longitudinal study we carried out was aimed at tackling four research questions for which we report a brief insight of our ndings.
RQ1. Do performances of monolingual systems increase over the years? Are more recent systems better than older ones?
From the analysis of mean standardized AP (sMAP) across monolingual tasks we can see an improvement of performances, even though it is not always steady from year to year, see Table 1. The best systems are rarely the most recent ones; this may be due to a tendency towards tuning well performing systems relying on established techniques in the early years of a task while focusing on understanding and experimenting new techniques and methodologies in later years. In general, the assumption for which the life of a task is summarized by increase in system performances, plateau and termination oversimpli es reality: researchers and developers do not just incrementally adding new pieces on existing algorithms, rather they often explore completely new ways or add new components to the systems, causing a temporary drop in performances. Thus, we do not have a steady increase but rather a general positive trend.
RQ2. Do performances of bilingual systems increase over the years and what is the impact of source languages?
System performances in bilingual tasks show a growing trend across the years although it is not always steady and it depends on the number of submitted runs as well as on the number of newcomers. The best systems for bilingual tasks are often the more recent ones showing the importance of advanced linguistic resources that become available and improved over the years. Source languages 2 http://direct.dei.unipd.it/
Task AH Mono ES AH Mono DE TEL Mono DE AH Mono FR TEL Mono FR AH Mono IT AH Mono NL have a high impact on the performances of a given target language, showing that some combinations are better performing than others { e.g. Spanish to Portuguese has a higher median sMAP than German to Portuguese.
RQ3. Do performances of multilingual systems increase over the years?
Multilingual systems show a steady growing trend of performances over the years despite the variations in target and source languages from task to task. We can identify a growing trend of performances especially for top systems. For instance the multilingual task with four languages reports a major improvement of median sMAP from 2002 to 2003 even though the top system of 2003 has lower sMAP than the one of 2002; the multilingual task with 8 languages reports the lowest median sMAP and, at the same time, the best performing system of all multilingual tasks.
RQ4. Do monolingual systems have better performances than bilingual and multilingual systems?
Systems which operate on monolingual tasks prove to be more performing than bilingual ones in most cases, even though the di erence between top monolingual and top bilingual systems reduces year after year and sometimes the ratio is even inverted. In some cases, multilingual systems turn out to have higher performances than bilingual ones and the top multilingual system, as shown in Table 2, has the highest sMAP of all the systems which participated in CLEF tasks from 2000 to 2009: the work done for dealing with the complexity of multilingual tasks pays o in terms of overall performances of the multilingual systems. 3
This study opens up diverse analysis possibilities and as future works we plan to investigate several further aspects regarding the cross-lingual evaluation activities carried out by CLEF; we will: (i) apply standardization to other largelyadopted IR measures { e.g. Precision at 10, RPrec, Rank-Biased Precision, bpref { with the aim of analysing system performances from di erent perspectives; (ii) aggregate and analyse the systems on the basis of adopted retrieval techniques to better understand their impact on overall performances across the years; and (iii) extend the analysis of bilingual and multilingual systems grouping them on a source and target language basis thus getting more insights into the role of language morphology and linguistic resources in cross-lingual IR.