This paper proposes a methodology based on discounted cumulated gain measures and visual analytics techniques in order to improve the analysis and understanding of IR experimental evaluation results. The proposed methodology is geared to favour a natural and e ective interaction of the researchers and developers with the experimental data and it is demonstrated by developing an innovative application based on Apple iPad.
H.3.3 [Information Search and Retrieval]: [Search process]; H.3.4 [Systems and Software]: [Performance evaluation (e ciency and e ectiveness)]
The Information Retrieval (IR) eld has a strong and longlived tradition, that dates back to late 50s/early 60s of the last century, as far as the assessment of the performances of an IR system is concerned. In particular, in the last 20 years, large-scale evaluation campaigns, such as the Text REtrieval Conference (TREC)1 in the United States and the CrossLanguage Evaluation Forum (CLEF)2 in Europe, have conducted cooperative evaluation e orts involving hundreds of
Copyright c 2011 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by the editors of euroHCIR2011. research groups and industries, producing a huge amount of valuable data to be analysed, mined, and understood.
The aim of this work is to explore how we can improve the comprehension of and the interaction with the experimental results by IR researchers and IR system developers. We imagine the following scenarios: (i) a researcher or a developer is attending the workshop of one of the large-scale evaluation campaigns and s/he wants to explore and understand the experimental results as s/he is listening at the presentation discussing them; (ii) a team of researchers or developers is working on tuning and improving an IR system and they need tools and applications that allow them to investigate and discuss the performances of the system under examination in a handy and e ective way.
These scenarios call for: (a) proper metrics that allow us to understand the behaviour of a system; (b) e ective analysis and visualization techniques that allow us to get an overall idea of the main factors and critical areas which have in uenced performances in order to be able to dig into details; (c) for tools and applications that allow us to interact with the experimental result in a both e ective and natural way.
To this end, we propose a methodology which allows us to
quickly get an idea of the distance of an IR system with
respect to both its own optimal performances and the best
performances possible. We rely on the (normalized) discounted
cumulated gain (n)DCG family of measures [
The contribution of this paper is to improve on the
previous work [
The proposed model provides the basis for the
development of Visual Analytics (VA) techniques that give us the
possibility to get a quick and intuitive idea of what
happened in a result list and what determined its perceived
performances. Visual Analytics [
The idea of exploring and applying VA techniques to the
experimental evaluation in the IR eld is quite innovative
since it has never been attempted before and, due to the
complexity of the evaluation measures and the amount of
data produced by large-scale evaluation campaigns, there is
a strong need for better and more e ective representation
techniques. Moreover, visualizing and assessing ranked list
of items, to the best of the authors' knowledge, has not been
addressed by the VA community. The few related
proposals, see, e.g., [
On top of the proposed model, we have built a running prototype where the experimental results and data are stored in a dedicated system accessible via standard Web services. This allows for the design and development of various client applications and tools for exploiting the managed data. In particular, in this paper, we have started to explore the possibility of adopting the Apple iPad3 as an appropriate device to allow users to easily and naturally interact with the experimental data and we have developed an initial prototype that allows us for interactively inspecting the actual experimental data in order to get insights about the behaviour of a IR system.
Overall, the proposed model, the proposed visualization techniques, and the implemented prototype meet all the (ac) requirements for the two scenarios introduced above.
The paper is organized as follows. Section 2 introduces the model underlying the system together with its visualization techniques; Section 3 describes the interaction strategies of the system, Section 4 describes the overall architecture of the system, and Section 5 concludes the paper, pointing out ongoing research activities.
According to [
vector of n documents V , i.e., V [
GT (V [i]); if i x GT (V [i])= logx(i); if i > x (1) The quality of a result can be assessed using the discounted cumulative gain function DCG(V; i) = Pij=1 DF (V [j]) that estimates the information gained by a user that examines the rst i documents of V .
The DCG function allows for comparing the performances of di erent search engines, e.g., plotting the DCG(i) values of each engine and comparing the curve behavior.
However, if the user's task is to improve the ranking performance of a single search engine, looking at the misplaced documents (i.e., ranked too high or too low with respect to the other documents) the DCG function does not help: the same value DCG(i) could be generated by di erent permutations of V and it does not point out the loss in cumulative gain caused by misplaced elements. To this aim, we introduce the following de nitions and novel metrics.
We denote with OptP erm(V ) the set of optimal permutations of V such as that 8OV 2 OptP erm(V ) it holds that GT (OV [i]) GT (OV [j])8i; j <= n V i < j, that is, OV maximizes the values of DCG(OV; i)8i. In other words, OptP erm(V ) represents the set of the optimal rankings for a given search result. It is worth noting that each vector in OptP erm(V ) is composed by k + 1 intervals of documents sharing the same GT values. As an example, assuming a result vector composed by 12 elements and k = 3, a possible sequence of GT values of an optimal vector OV is <3,3,3,3,2,2,2,2,1,1,0,0>; according to this we de ne the max index(V; r) and min index(V; r) functions, with 0 r k, that return the greatest and the lowest indexes of elements in a vector belonging to OptP erm(V ) that share the same GT value r. As an example, considering the above 12 GT values, min index(V; 2) = 5 and max index(V; 2) = 8.
Using the above de nitions we can de ne the relative position R P os(V [i]) function for each document in V as follows: ( 0; if min index(V; GT (V [i]) i max index(V; GT (V [i]) min index(V; GT (V [i]) i; if i < min index(V; GT (V [i]) max index(V; GT (V [i]) i; if i > max index(V; GT (V [i]) R P os(V [i]) allows for pointing out misplaced elements and understanding how much they are misplaced: 0 values denote documents that are within the optimal interval, negative and positive values denote elements that are respectively below and above the optimal interval. The absolute value of R P os(V [i]) gives the minimum distance of a misplaced element from its optimal interval.
According to the actual relevance and rank position, the same value of R P os(V [i]) can produce di erent variations of the DCG function. We measure the contributions of misplaced elements with the function Gain(V; i) that compares 8i the actual values of DF (V [i]) with the corresponding values in OV , DF (OV [i]): Gain(V; i) = DF (V [i]) DF (OV [i]).
A multi-touch prototype interface based on the model presented in section 2 has been designed for the iPad device. It has been developed and tested on the iOS 4.24 with the integration of the Core Plot5 plotting framework for the graphical visualization of data. The interface allows the end user for comparing the curve of the ranked results, for a given experiment/topic, with the optimal one and with the ideal one. This facilitates the activities of failure analysis, easily locating misplaced elements, blue or red items, that pop up from the visualization together with the extent of their displacement and the impact they have on DCG.
Figure 1 shows a screenshot of the current interface: the main list on the left represents the top n = 200 ranked result for a given experiment/topic and it can be easily scrolled by the user. Each row corresponds to a document ID, a short snippet of the content is included in the subtitle of each cell and more information on a speci c result (i.e. relevance score, DCG, R P os, Gain) can be viewed by touching the row. On the right side there are two coloured vectors which show the R P os and Gain functions. The R P os vector presents the results using di erent color shadings: light green, light red and light blue respectively for documents that are within, below and above the optimal interval. It allows for locating misplaced documents and, thanks to the shading, understanding how they are far from the optimal 4http://developer.apple.com/ 5http://code.google.com/p/core-plot/ position. Similarly, the Gain vector codes the function using colors: light blue refers to positive values, light red codes negative values, and green 0 values. Moreover, if the user touches a speci c area of the R P os vector (that is simulated by the gray round in Figure 1), the main results list automatically scrolls back, providing the end user with a detailed view on the corresponding documents. The rightmost part of the screen shows the DCG graphs of the ideal, the optimal and the experiment vector, i.e. the ranking curves. The navigation bar displays a back button on the right which let the user visualize the results for a di erent topic. 4.
The design of the architecture of the system bene ts from
what has been learned in ten years of work for the CLEF and
in the design and implementation of Distributed Information
Retrieval Evaluation Campaign Tool (DIRECT), the system
developed in CLEF since 2005 to manage all the aspects of
an evaluation campaign [
The approach to the architecture is the implementation of a modular design, as sketched in Figure 2, with the aim to clearly separate the logic entailed by the application into three levels of abstraction { data, application, and interface logic { able to reciprocally communicate, easily extensible and implementable using modular and reusable components. The Data Logic layer, depicted at the bottom of Figure 2, deals with the persistence of the information coming from the other layers. From the implementation point of view, data stored into databases and indexes are mapped to resources and communicate with the upper levels through the mechanism granted by the Data Access Object (DAO) pattern6 | see point (1) in Figure 2. The Application Logic
layer is in charge of the high-level tasks made by the
system, such as the enrichment of raw data, the calculation
of metrics and the carrying out of statistical analyses on
experiments. These resources (2) are therefore accessible
via HTTP through a RESTful Web service [
We have presented a model and a prototype which allow users to easily interact with the experimental results and to work together in a cooperative way while actually accessing the data. This rst step uncovers new and interesting possibilities for the experimental evaluation and for the way in which researchers and developers usually carry out such activities. For example, the proposed techniques may alleviate the burden of certain tasks, such as failure analysis, which are often overlooked due to their demanding nature, thus making easier and more common to perform them and, as a consequence, improving the overall comprehension of system behaviour. This will be explored in the future work. Patterns/DataAccessObject.html
The work reported in this paper has been partially supported by the PROMISE network of excellence (contract n. 258191), as a part of the 7th Framework Program of the European commission (FP7/2007-2013). 6.