Evaluation has a crucial role in Information Retrieval (IR) and developing tools to support researchers and analysts when analyzing results and investigating strategies to improve IR system performance can help make the analysis easier and more e ective. To this purpose we present a Visual Analytics-based approach to support the analyst in performing failure and what-if analysis.
Designing, developing, and testing an IR system is a challenging task, especially when it comes to understanding and analysing the behaviour of the system under di erent conditions in order to tune or to improve it as to achieve the level of e ectiveness needed to meet the user expectations.
Failure analysis is especially resource demanding in terms of time and human e ort, since it requires inspecting, for several queries, system logs, intermediate output of system components, and, mostly, long lists of retrieved documents which need to be read one by one in order to try to gure out why they have been ranked in that way with respect to the query at hand.
Considering this, it is important to de ne new ways to help IR researchers, analysts and developers to understand the limits and strengths of the IR system under investigation. Visual analytics techniques can give assistance to this process by providing graphic tools which interacting with IR techniques may ease the work of the users.
The goal of this paper is to exploit a visual analytics approach to design
a methodology and develop an interactive visual system which support IR
researchers and developers in conducting experimental evaluation and improving
their systems by: (i) reducing the e ort needed to conduct failure analysis; (ii)
allowing them to anticipate what the impact of a modi cation to their system
could be before needing to actually implement it.
? The extended version of this abstract has been published in [
Starting from the Discounted Cumulative Gain (DCG) measures, we introduce two functions: the relative position, which quanti es how much a document has been misplaced with respect to its ideal (optimal) position, and the delta gain, which quanti es how much each document has gained/lost with respect to its ideal (optimal) DCG. On top of this ranking model, we propose a visualization, see Figure 1, where the DCG curves for the experiment ranking, the ideal ranking, and the optimal ranking are displayed together with two bars, on the left, representing the relative position and the delta gain. Please note that an equivalent graph can be obtained by using nDCG in the place of DCG.
The proposed ranking model and the related visualization are quite
innovative because, usually, information visualization and visual analytics are exploited
to improve the presentation of the results of a system to the end user, rather
than applying them to the exploration and understanding of the performances
and behaviour of an IR system. Secondly, comparisons are usually made with
respect to ideal ranking only while our method allows user to compare a system
also which respect to the optimal ranking produced with the system results, thus
giving the possibility of better interpreting the obtained results [
Therefore, the overall goal is to have an initial raw estimate of the e ect of a planned modi cation before actually implementing it in terms of e ect both for the topic under examination and for the other topics. This gives researchers and developers the possibility of exploring several alternatives before having to implement them and of determining a reasonable trade-o between the e ort and costs for given modi cations and the expected improvements.
Figure 2 shows the block diagram describing the pipeline of the data exchanged in the whole process. We consider the general-purpose IR scenario composed by a set of topics T , a collection of documents D, and a ranking model RM; an IR system for a given topic tk 2 T retrieves a set of documents Dj D.
The ranking model RM generates for each topic tk 2 T a ranked document list RLj . The whole set of ranked lists constitute the input for building the Clustering via Learning to Rank Model that is in charge of generating, for each document, a similarity cluster. The Visualization deals with one topic t at time: it takes as input the ranked document list for the topic t and the ideal ranked list, obtained choosing the most relevant documents in the collection D for the topic t and ordering them in the best way. While visually inspecting the ranked list, it is possible to simulate the e ect of interactively reordering the list, moving a target document d and observing the e ect on the ranking while this shift is propagated to all the documents of the cluster containing the documents similar to d. This cluster of documents simulates the \domino e ect" within the given topic t.
When the analyst is satis ed with the results, i.e. when he has produced a new ranking of the documents that corresponds to the e ect that is expected by modi cations that are planned for the system, he can feed the Clustering via Learning to Rank Model with the newly produced ranked list, obtain a new model which takes into account the just introduced modi cations, and inspecting the e ects of this new model for other topics. This re-learning phase simulates the \domino e ect" on the other topics di erent from t caused by a possible modi cation in the system. 4
This paper presented a fully- edged analytical and visualization model to support interactive exploration of IR experimental results. The overall goal of the paper has been to provide users with tools and methods to investigate the performances of a system and explore di erent alternatives for improving it avoiding a continuous iteration of trials-and-errors to see if the proposed modi cations actually provide the expected improvements.
Acknowledgements The work reported in this paper has been supported by the PROMISE network of excellence (contract n. 258191) project as a part of the 7th Framework Program of the European commission (FP7/2007-2013).