In this paper, we describe Combinatorial visuaL Analytics system for Information Retrieval Evaluation (CLAIRE), a Visual Analytics (VA) system for exploring and making sense of the performances of a large amount of Information Retrieval (IR) systems, in order to quickly and intuitively grasp which system con gurations are preferred, what are the contributions of the di erent components and how these components interact together.
Information Retrieval (IR) systems are constituted of \pipelines" of components such as stop lists, stemmers and IR models, which are stacked together in order to process both documents and user queries and to match them returning a ranked result list of documents in decreasing order of estimated relevance. Currently, the only viable means to determine the contribution to the system e ectiveness of single components is to measure their impact on the overall performances by testing all the di erent combinations of such components. This leads to a very high number of cases to be considered, making the space of system combinations large and complex to explore.
Besides requiring a great deal of e ort and resources to be produced, these
combinatorial compositions constitute a challenge when it comes to explore,
analyze, and make sense of the experimental results with the goal of understanding
how di erent components contribute to the overall performances and interact
together. Indeed, it is typically needed to resort to rather complex statistical tools
(e.g. multi-way ANalysis Of VAriance (ANOVA) models) requiring a careful
experimental design and producing results which call for a considerable extent of
expertise to be interpreted [
In this paper we present a Visual Analytics (VA) system, called
Combinatorial visuaL Analytics system for Information Retrieval Evaluation (CLAIRE),
which allows for exploring and making sense of the performances of a large
amount of IR systems generated from all the possible combinations of
components under examination, in order to quickly and intuitively grasp which system
con gurations are preferred, what are the contributions of the di erent
components and how these components interact together. To showcase and evaluate
CLAIRE, we developed an extensive set of 612 6 = 3; 672 systems { i.e. the
Grid of Points (GoP) [
CLAIRE addresses the necessity to analyse large combinations of system
components due to the proliferation of open source IR systems [
The CLAIRE system is available at http://awareserver.dis.uniroma1.it: 11768/claire/. We considered three main components of an IR system: stop list, stemmer, and IR model. We selected a set of alternative implementations of each component and, by using the Terrier v.4.0 open source system, we created a run for each system de ned by combining the available components in all possible ways. The selected components are: { Stop list : nostop, indri, lucene, snowball, smart, terrier; { Stemmer : nolug, weakPorter, porter, snowballPorter, krovetz, lovins; { Model : bb2, bm25, dfiz, dfree, dirichletlm, dlh, dph, hiemstralm, ifb2, inb2, inl2, inexpb2, jskls, lemurtfidf, lgd, pl2, tfidf.
We considered the following standard and shared collections, each track using 50 di erent topics, TREC Adhoc tracks T07 and T08, TREC Web tracks T09 and T10, TREC Terabyte tracks T14 and T15. Overall, these components de ne a 6 6 17 factorial design with a GoP consisting of 612 system runs. They represent nearly all the state-of-the-art components which constitute the common denominator almost always present in any IR system for English retrieval and thus they are a good account of what can be found in many di erent operational settings.
The design of the system followed agreed solutions in the eld of Infovis: the CLAIRE main structure relies on multiple coordinated views. CLAIRE comprises the three main areas shown in Figure 1. (1) Parameters Selection area, dealing with the exploration coordinates, i.e., collections, stop lists, stemmers, IR models, and measures. (2) the System Con gurations Analysis area, enabling http://www.terrier.org/ the performance analysis of the system con gurations using the actual evaluation measure where each box represents a speci c component (stop list in the gure) and the tiles within the box contain all the combinations of the other two components (IR models and stemmers in the gure) with the component in the box. Each tile represents a system con guration, with its color, ranging from white to deep blue, represents a low mean value or a high mean value respectively, averaged over all the topics of a track. The size of the tile represents the con dence interval (small sizes tied to low values), again averaged over all the topics of a track. (3) the Overall Evaluation area, where the system congurations performances are evaluated on the complete set of given evaluation measures. Moreover, to automate the selection of relevant subsets of systems, CLAIRE uses the available measures to select clusters of similar systems.
In order to present an example of obtainable evidences from the use of CLAIRE, Figure 2 reports three model tiles corresponding to di erent visual archetypes: (a) the bb2 model needs a stop list to function well, (b) the tfidf model works better with a stemmer, and (c) the bm25 model su ers the absence of the stop list and also, even though the e ect is less marked, the absence of a stemmer; It is also possible to see that the three models need a stop list and/or a stemmer, but they do not discriminate between di erent stop lists and stemmers. 3
We presented a relevant case implying the exploration of almost 1.5M data points corresponding to di erent performance measures of hundreds of IR systems. To this end, we developed a novel VA system, CLAIRE, that supports the analysis of a large set of IR systems. As future work, we will extend the CLAIRE system by allowing users to upload their proprietary systems and components and compare them against the standard open-source baselines present in the CLAIRE GoP.