This paper examines the current way of keeping the data produced during the evaluation campaigns and highlights some shortenings of it. As a consequence, we propose a new approach for improving the management evaluation campaigns' data. In this approach, the data are considered as scientific data to be cured and enriched in order to give full support to longitudinal statistical studies and long-term preservation.
The experimental evaluation of an Information Retrieval System (IRS) is a
scientific activity whose outcomes are different kinds of scientific data:
– experiments: the primary scientific data produced by the participants of an
evaluation campaign;
– performance measurements: the metrics, such as precision and recall [
When we deal with scientific data, “the lineage (provenance) of the data
must be tracked, since a scientist needs to know where the data came from [. . . ]
and what cleaning, rescaling, or modelling was done to arrive at the data to
be interpreted” [
Furthermore, when scientific data are maintained for further and future use, they should be enriched and, besides information about provenance, also changes at sources occurred over time need to be tracked. Sometimes the enrichment of a portion of scientific data can make use of a citation for explicitly mentioning and making references to useful information.
In this paper we examine whether the current methodology properly deals
with the data produced during an evaluation campaign by recognizing that they
are in effect valuable scientific data. Furthermore, we describe the data curation
approach [
The paper is organized as follows: Section 2 introduces the motivations and the objectives of our research work; Section 3 describes the work carried out in developing the CLEF infrastructure; finally, Section 4 draws some conclusions. 2 2.1
Nowadays, the experimental evaluation of an IRS is carried out in important international evaluation forums which bring research groups together, provide them with the means for measuring the performances of their systems, discuss and compare their work. The Text REtrieval Conference (TREC)1 has been the first initiative in this field and has laid the groundwork for the other subsequent initiatives; TREC developed a common evaluation procedure in order to compare IRSs by measuring the effectiveness of different techniques, and to discuss how differences between systems affected performances. After TREC, other international important initiatives have been launched, in particular Cross-Language Evaluation Forum (CLEF), NII-NACSIS Test Collection for IR Systems (NTCIR) and INitiative for the Evaluation of XML Retrieval (INEX). The CLEF2 aims at evaluating Cross Language Information Retrieval (CLIR) systems that operate on European languages in both monolingual and cross-lingual contexts. The NTCIR3 is the Asian counterpart of CLEF where 1 http://trec.nist.gov/ 2 http://clef.isti.cnr.it/ 3 http://research.nii.ac.jp/ntcir/index-en.html the traditional Chinese, Korean, Japanese, and English languages are the basis of the evaluation of cross-lingual tasks. The INEX4 provides participants with evaluation procedures for content-oriented eXtensible Markup Language (XML) retrieval in order to measure the effectiveness of IRSs that manage XML documents. These evaluation forums are usually further organized into tracks, which investigate different facets of the evaluation of the information access components of a Digital Library Management System (DLMS).
All of the previously mentioned initiatives are generally carried out according
to the Cranfield methodology, which makes use of experimental collections [
Nevertheless, the Cranfield methodology is mainly focused on creating comparable experiments and evaluating the performances of an IRS rather than modeling and managing the scientific data produced during an evaluation campaign.
As an example, note that the exchange of information between organizers and participants is mainly performed by means of textual files formatted according to the TREC data format, which is the de-facto standard in this field. Note that this information represents a first kind of scientific data produced during the evaluation process. The following is a fragment of the results of an experiment submitted by a participant to the organizers, where the gray header is not really present in the exchanged data but serves here as an explanation of the fields.
141 Q0 AGZ.950609.0067 0 141 Q0 AGZ.950613.0165 1 ...
0.440873414278 IMSMIPO 0.305291658641 IMSMIPO As a further example, the following is a fragment of relevance judgements sent back by organizers to participants: 4 http://inex.is.informatik.uni-duisburg.de/ specifies the unique identifier of the topic (e.g. 141), the field for the unique identifier of the document (e.g. AGZ.950609.0067), the field which identifies the experiment (e.g. IMSMFPO), the field which specifies whether a document is relevant for a topic (e.g. 1) or not (e.g. 0) and so on, as specified by the gray headers.
As it can be noted from the above examples, this format is suitable for a simple data exchange between participants and organizers. Nevertheless, neither this format provides any metadata explaining its content nor a scheme exists in order to define the structure of each file, the data type of each field, and various constraints on the data, such as numeric floating point precision. Moreover, this format is not very suitable for modelling the information space involved by an evaluation forum because the relationships among the different entities (documents, topics, experiments, participants) are not modeled and each entity is treated separately from the others.
Furthermore, present collections keeping over time does not permit
systematic studies on reached improvements by participants over the years, for example
in a specific multilingual setting [
We argue that the information space implied by an evaluation forum needs
an appropriate conceptual model which takes into consideration and describes all
the entities involved by the evaluation forum. In fact, an appropriate conceptual
model is the necessary basis to make the scientific data produced during the
evaluation an active part of all those information enrichments, as data provenance
and citation, we have described in the previous section. This conceptual model
can be also translated into an appropriate logical model in order to manage the
information of an evaluation forum by using the database technology, as an
example. Finally, from this conceptual model we can derive also appropriate data
formats for exchanging information among organizers and participants, such as
an XML format that complies with an XML Schema [
The Cranfield methodology is mainly focused on how to evaluate the
performances of two systems and how to provide a common ground which makes the
experimental results comparable. [
To address this issue, evaluation forums have traditionally carried out
statistical analyses, which provide participants with an overview analysis of the
submitted experiments, as in the case of the overview papers of the different tracks
at TREC and CLEF; some recent examples of this kind of papers are [
Thus, we can observe that, in general, there is a limited support to the systematical employment of statistical analysis by participants. For this reason, we suggest that evaluation forums should support and guide participants in adopting a more uniform way of performing statistical analyses on their own experiments. In this way, participants can not only benefit from standard experimental collections which make their experiments comparable, but they can also exploit standard tools for the analysis of the experimental results, which make the analysis and assessment of their experiments comparable too.
As introduced in Section 1, scientific data, their enrichment and interpretation are essential components of scientific research. The Cranfield methodology traces out how these scientific data have to be produced, while the statistical analysis of experiments provide the means for further elaborating and interpreting the experimental results. Nevertheless, the current methodologies does not require any particular coordination or synchronization between the basic scientific data and the analyses on them, which are treated as almost separated items. On the contrary, researchers would greatly benefit from an integrated vision of them, where the access to a scientific data item could also offer the possibility of retrieving all the analyses and interpretations on it. Furthermore, it should be possible to enrich the basic scientific data in an incremental way, progressively adding further analyses and interpretations on them.
Let us consider what is currently done in an evaluation forum: – Experimental collections: • there are few or no metadata about document collections, the context they refer to, how they have been created, and so on; • there are few or no metadata about topics, how they have been created, the problems encountered by their creators, what documents creators found relevant for a given topic, and so on; • there are few or no metadata about how pools have been created and about the relevance assessments, the problems which have been faced by the assessors when dealing with difficult topics; 5 http://users.cs.dal.ca/~jamie/pubs/IRSP-overview.html 6 http://www.r-project.org/ 7 http://www.spss.com/ 8 http://www.mathworks.com/ – Experiments: • there are few or no metadata about them, such as what techniques have been adopted or what tunings have been carried out; • they can be not publicly accessible, making it difficult for other researchers to make a direct comparison with their own experiments; • their citation can be an issue; – Performance measurements: • there are no metadata about how a measure has been created, which software has been used to compute it, and so on; • often only summaries are publicly available while all the detailed measurements may be not accessible; • their format can be not suitable for further computer processing; • their modelling and management needs to be dealt with; – Descriptive statistics and hypothesis tests: • they are mainly limited to task overviews produced by organizers; • participants may not have all the skills needed to perform a statistical analysis; • participants can carry out statistical analyses only on their own experiments without the possibility of comparing them with the experiments of other participants; • the comparability among the statistical analyses conducted by the participants is not fully granted due to possible differences in the design of the statistical experiments.
These issues are better faced and framed in the wider context of the curation
of scientific data, which plays an important role on the systematic definition of
a proper methodology to manage and promote the use of data. The e–Science
Data Curation Report gives the following definition of data curation [
This definition implies that we have to take into consideration the possibility of information enrichment of scientific data, meant as archiving and preserving scientific data so that the experiments, records, and observations will be available for future research, as well as provenance, curation, and citation of scientific data items. The benefits of this approach include the growing involvement of scientists in international research projects and forums and increased interest in comparative research activities. Furthermore, the definition introduced above reflects the importance of some of the many possible reasons for which keeping data is important, for example: re-use of data for new research, including collection based research to generate new science; retention of unique observational data which is impossible to re-create; retention of expensively generated data which is cheaper to maintain than to re-generate; enhancing existing data available for research projects; validating published research results.
As a concrete example in the field of information retrieval, please consider
the data fusion problem [
Thus, we consider all these points as requirements that should be taken into
account when we are going to produce and manage scientific data that come
out from the experimental evaluation of an IRS. In addition, to achieve the full
information enrichment discussed in Section 1, both the experimental datasets
and their further elaboration, such as their statistical analysis, should be first
class objects that can be directly referenced and cited. Indeed, as recognized
by [
As discussed in the previous section, we need to design and develop a proper conceptual model of the information space involved by an evaluation forum. Indeed, this conceptual model provide us with the basis needed to offer all the information enrichment and interpretation features described above.
Figure 1 shows the Entity–Relationship (ER) schema which represents the
conceptual model we have developed. The conceptual model is built around five
main areas of modelling:
– evaluation forum: deals with the different aspects of an evaluation forum,
such as the conducted evaluation campaigns and the different editions of each
campaign, the tracks along which the campaign is organized, the subscription
of the participants to the tracks, the topics of each track;
– collection: concerns the different collections made available by an evaluation
forum; each collection can be organized into various files and each file may
contain one or more multimedia documents; the same collection can be used
by different tracks and by different editions of the evaluation campaign;
– experiments: regards the experiments submitted by the participants and
the evaluation metrics computed on those experiments, such as precision and
recall;
– pool/relevance assessment: is about the pooling method [
ASSESSOR (1,N) ASSESS (1,N) ID Threshold
Email Pwd
Relevant Prefix
FILE Position
Sufix (1,N) Modified
CONTAIN DocID (1,1) Content
DOCUMENT (0,N)
POOLED ID (0,N) Name (1,N)
(1,N) (1,N)
TRACK (0,N) SUBSCRIBE (1,N) PARTICIPANT (1,N)
QUERY (0,N)
CloseTop ID (1,N)OpenTop
COCOlpolpeosensenDeDTTeeiitstsllcece TOPICGROUP EXPERIMENT (1,N) OpenNarCrloseNarr (1,N) (0,N) CONTAIN (0,N)
ID (1,1) Title
Prc1A1vg AvgPrcRetr (1,N)
TOPIC (0,N) METRIC Description(0,N) NarrativeRelRet P9rc ExtPrc Rel
ITEM Rank
Score
ContactMail Priority (1,N)SubmitedFile
SUBMIT Date ID (1,1)TrecEvalFile Experiments
Modelling (1,N) (0,N) CONTAIN
Notes
Type (0,N)
TEST
ID alpha p-value (0,N) STATANALYSIS GROUPTEST (0,N) ID (0,N)
AddInfo (0,N) TestedFigure
TYPIFY RUNGROUP Name (1,1) Description
Hypothesis STATTEST
Statistical Analysis Modelling Collection Modelling
DtdFile Name Language Description SourceTarget PRatehaTdoMZeipFFililee COLLECTION (1,N) USE
Copyright Creators(1,N) Modified Created
CONSISTOF Name (1,1) Created
EDITION (1,1) StartDate(1,N)EndDate (1,N)
STRUCTURE
ID (1,1) Description Evaluation Forum Modelling Workshop Year ShortName Ful Name WebSite
ContactName
ContactMail ORGANIZE (1,N) CAMPAIGN
ID Type Content DISSEMINATE
PUBLISH Date((11U,,11s))erNamePPAwPEdRName ConFtoarcmtNaatme
Pool / Releva(n1,Nc)e Assessment Modelling – statistical analysis: models the different aspects concerning the statistical analysis of the experimental results, such as the type of statistical test employed, its parameters, the observed test statistic, and so forth.
Data Logic The data logic layer deals with the persistence of the different information objects coming from the upper layers. There is a set of “storing managers” dedicated to storing the submitted experiments, the relevance assessments and so on. We adopt the Data Access Object (DAO)9 and the Transfer Object (TO)9 design patterns. The DAO implements the access mechanism required to work with the underlying data source, acting as an adapter between 9 http://java.sun.com/blueprints/corej2eepatterns/Patterns/ Stand-alone Applications
Matlab Statistics Toolbox
Java-Matlab Bridge
Java-Treceval
Engine
Information Access Evaluation Service for a Scientific meta -DLMS
UsPearrtIinctieprafancte UsAerssInetsesrofarce UAsdemriInnitsetrrfaatcoer Statistical Analysis Management Tool
Run Management Tool
Pool-Assessment Management Tool
Evaluation Forum Management Tool
User Management Tool
Log Management Tool Statistical Analisys Storing Manager
Run Storing Manager
Pool-Assessment Storing Manager
Evaluation Forum Storing Manager
User Storing Manager
Log
Storing Manager Service Integration Layer Storing Abstraction Layer
Databases the upper layers and the data source. If the underlying data source implementation changes, this pattern allows the DAO to adapt to different storage schemes without affecting the upper layers.
In addition to the other storing managers, there is the log storing manager which fine traces both system and user events. It captures information such as the user name, the Internet Protocol (IP) address of the connecting host, the action that has been invoked by the user, the messages exchanged among the components of the system in order to carry out the requested action, any error condition, and so on. Thus, besides offering us a log of the system and user activities, the log storing manager allows us to fine trace the provenance of each piece of data from its entrance in the system to every further processing on it.
Finally, on top of the various “storing managers” there is the Storing Abstraction Layer (SAL) which hides the details about the storage management to the upper layers. In this way, the addition of a new “storing manager” is totally transparent for the upper layers.
Application Logic The application logic layer deals with the flow of operations within Distributed Information Retrieval Evaluation Campaign Tool (DIRECT). It provides a set of tools capable of managing high-level tasks, such as experiment submission, pool assessment, statistical analysis of an experiment.
For example, the Statistical Analysis Management Tool (SAMT) offers the functionalities needed to conduct a statistical analysis on a set of experiments. In order to ensure comparability and reliability, the SAMT makes uses of wellknown and widely used tools to implement the statistical tests, so that everyone can replicate the same test, even if he has no access to the service. In the architecture, the MATLAB Statistics Toolbox10 has been adopted, since MATLAB is a leader application in the field of numerical analysis which employs stateof-the-art algorithms, but other software could have been used as well. In the case of MATLAB, an additional library is needed to allow our service to access MATLAB in a programmatic way; other softwares could require different solutions. As an additional example aimed at wide comparability and acceptance of the tools, a further library provides an interface for our service towards the trec eval package11. trec eval has been firstly developed and adopted by TREC and represents the standard tool for computing the basic performance figures, such as precision and recall.
Finally, the Service Integration Layer (SIL) provides the interface logic layer with a uniform and integrated access to the various tools. As we noticed in the case of the SAL, thanks to the SIL also the addition of new tools is transparent for the interface logic layer.
Interface Logic It is the highest level of the architecture, and it is the access point for the user to interact with the system. It provides specialised User Interfaces (UIs) for different types of users, that are the participants, the assessors, and the administrators. Note that, thanks to the abstraction provided by the application logic layer, different kind of UIs can be provided, either standalone applications or Web-based applications. 3.3
The proposed service has been implemented in a first prototype, called
Distributed Information Retrieval Evaluation Campaign Tool (DIRECT) [
DIRECT was successfully adopted during the CLEF 2005 campaign. It was
used by nearly 30 participants spread over 15 different nations, who submitted
more than 530 experiments; then 15 assessors assessed more than 160,000
documents in seven different languages, including Russian and Bulgarian which do
not have a latin alphabet. During the CLEF 2006 campaign, it has been used
by nearly 75 participants spread over 25 different nations, who have submitted
around 570 experiments; 40 assessors assessed more than 198,500 documents
in nine different languages. DIRECT was then used for producing reports and
overview graphs about the submitted experiments [
DIRECT has been developed by using the Java12 programming language, which ensures great portability of the system across different platforms. We used the PostgreSQL13 DataBase Management System (DBMS) for performing the actual storage of the data. Finally, all kinds of UI in DIRECT are Web-based interfaces, which make the service easily accessible to end-users without the need of installing any kind of software. These interfaces have been developed by using the Apache STRUTS14 framework, an open-source framework for developing Web applications. 4
The discussed data curation approach that can help to face the test-collection challenge for the evaluation and future development of information access and extraction components of interactive information management systems. On the basis of the experience gained keeping and managing the data of interest of the CLEF evaluation campaign, we are testing the considered requirements to revise the proposed approach.
A prototype of the carrying out the proposed approach, called DIRECT, has been implemented and widely tested during the CLEF 2005 and 2006 evaluation campaigns. On the basis of the experience gained, we are enhancing the proposed conceptual model and architecture, in order to implement a second version of the prototype to be tested and validated during the next CLEF campaigns.
The work reported in this paper has been partially supported by the DELOS Network of Excellence on Digital Libraries, as part of the Information Society Technologies (IST) Program of the European Commission (Contract G038507618).
The authors would like to warmly thank Carol Peters, coordinator of CLEF, for her continuous support and advice. 12 http://java.sun.com/ 13 http://www.postgresql.org/ 14 http://struts.apache.org/