Knowledgeworkers need special information retrieval (IR) tools because their IR tasks and practices differ from the general public and from each other [1]. Several special IR tools for academic knowledge workers have been proposed, some of which visualize the search space of literature [2,3]. These tools are sometimes called macroscopes, that is, tools for visualizing big or complex data [4]. Macroscopes facilitate document search through browsing because the visual content and context provides additional information. This information is particularly helpful when Boolean queries are inadequate for an IR task, e.g. if the user does not know the relevant terms to search for. It is difficult to perform offline evaluations of IR macroscopes because there are no standards for the simulation of the stopping point in a browsing task. An analysis of this difficulty of stopping point simulation for document retrieval can be found in the work of Maxwell et al. [5].
To assist knowledge workers, we have prototyped an IR macroscope for academic search literature based on citation clusters. We refer to this tool as SciMacro (Science Macroscope). SciMacro clusters papers based on their citations and summarizes the content of each cluster. This is different from previous works that cluster papers based on their textual content, like Iris [6], or use citations to find related papers, like PaperPoles [7]. The user can obtain smaller clusters from the papers of a given cluster for a more detailed visualization. As an example, let us consider a fictive user that has a set of documents from a multidisciplinary journal and that wants to know which ones are related to the visualization of big data. The user provides his initial set of documents to SciMacro, which are then clustered into 3 clusters. The descriptors of the clusters are Mathematics, Statistics and Physics. Then, the user selects the Statistics cluster and gets 3 smaller clusters from the next clustering level labelled Visual, Modeling and Bayesian. Because these new clusters were created for documents from the Statistics cluster, the user knows that they are also related to statistics. Finally, the user selects the Visual cluster, and obtains the smaller clusters Analytic, GIS, and Big. Now the user sees the Big cluster related to visualization and statistics, enabling him to screen the documents in that cluster.
In this paper, we evaluate the performance of SciMacro at retrieving relevant scientific literature for systematic reviews. Our work is of particular relevance because the potential of IR macroscopes that allow for cluster-based browsing of the complete document set of all scientific literature has not been studied before.
The task goals in our study were to find the relevant literature for 17 systematic reviews (SRs). SRs are review articles that report the relevant literature found by the authors [8,9]. We used a public test set of these reports to simulate the IR tasks [10]. We obtained the results of the SciMacro IR tasks from a simulation of five user models with different emphasis on precision and recall.
We address the following research questions:
1. What is the potential of SciMacro for finding the relevant literature for SRs? 2. How do the user's preferences affect the performance of SciMacro? The main contribution of this paper is that it presents the first evaluation of SR search through citation clusters.
The idea of using citations for academic IR browsing is not new: A number of prior works [11,3] have proposed IR tools that visualize papers based on their citation networks, while others [6,7] have proposed IR tools that visualize clusters of papers. The tool Citation Gecko [2] visualizes a citation network that expands from given papers, while the work by Haunschild and Marx [12] uses a citation network to find the seminal paper on a topic. On the other hand, text processing can also be used for IR browsing: A number of prior works [13,14] have proposed IR tools that cluster the semantics of the papers, while others [15,16] have proposed IR tools that suggest terms for complex Boolean queries.
Our tool, SciMacro, belongs to the academic IR browsing tools that visualize clusters of papers. One possibility is to cluster papers based on textual similarity (see for instance the tool Iris [6]). SciMacro does not use textual similarity, but instead uses citations. Citations represent primarily the intellectual relation between papers, while text may represent (broader) topical clustering.
Following citations is a common strategy for authors of SRs [17]. Some IR tools have been proposed for SRs that classify the relevance of papers combining their citations and text [18]. Other tools reduce the workload of the authors by ranking their search results [19,20]. These tools are different from SciMacro because they are not based on browsing. For a more complete overview of IR techniques for SRs, we refer to [21].
For non-academic IR browsing, a prominent model is the Scatter/Gather browsing model [22,23], which has inspired the development of SciMacro. In this browsing model, a cluster of documents is split into smaller clusters, each with their own label. The clusters can also be combined to give the user control over the clustering solution. Scatter/Gather has been previously used for clustering web services [24] and web search results [25]. SciMacro is the first to use the Scatter/Gather model for academic literature search.
During the SIGIR 2010 Workshop on the Simulation of Interaction it was argued that the simulation of different search types (browsing, directed and drifting) requires different user models [26]. We follow up on this work, but instead of simulating query search we simulate the browsing behaviour in citation clusters.
Beyond simulations, Mahdi et al. [27] proposed a framework for evaluating browsing tasks with real users. We also want to highlight the work of Leuski [28], who worked with real users to evaluate a web search tool in which clusters of search results are presented. His work emphasized that clustering the search results gives the user a sense of control over the feedback process, which is a highly valued feature in professional search [1]. In line with this, we designed SciMacro in such a way that the user also has control over the feedback process.
We model the search process of the authors of SRs as an IR task. We start from the following idea: When an author of a SR decides to read the full text of a document, based on the abstract and/or title, we consider this document to be relevant. Therefore, we argue that an IR tool should find all documents that the user considers relevant enough to read for a SR. With this consideration in mind, we decided to use the SRs published by the Cochrane Library database [29], which requires authors to report the documents of which they read the full text, regardless of whether they included these documents in the SR or not. We will refer to this set of documents as the relevant documents of a particular SR.
This search model retrieves the documents of a given cluster (see Section 3.4 for an explanation of the clustering). To select a cluster, we defined the following simulation protocol, similar to a greedy algorithm (see Figure 1 for an example):
1. Select the cluster from level 1 with the highest relevance score (the score is explained below).
2. If the selected cluster has subclusters, obtain the highest score of the subclusters. Otherwise, retrieve the selected cluster. 3. If the highest score of the subclusters is higher than the score of the selected cluster, select the subcluster with the highest score and go back to step 2. Otherwise, retrieve the selected cluster.
The goal of this protocol is to simulate the behaviour of a real user. In our simulation the user has perfect knowledge of the relevance scores of the clusters; this is a common simplification in simulation for evaluation [10]. In a real situation the user has to deduce this knowledge from the cluster labels. 1To evaluate this IR task, we need to know the number of relevant retrieved documents, non-relevant retrieved documents and relevant non-retrieved documents from the simulation:
• The total number of retrieved documents equals the number of documents in the retrieved cluster. • The number of relevant retrieved documents equals the number of relevant documents that are in the retrieved cluster. • The number of non-relevant retrieved documents equals the total number of retrieved documents minus the number of relevant retrieved documents. • The relevant non-retrieved documents are the documents of the SR that are not in the retrieved cluster.
With these values we can also calculate the weighted F-score of each cluster:
(1)
We created five user models that differ in which F-score they prefer: F0.25, F0.5, F1, F2 or F4. The different F-scores reflect the different needs of real users: for example, a real user that wants a short overview of a topic will emphasize precision over recall. Lower subscript F-scores emphasize precision, and higher subscript F-scores emphasize recall. F1 gives an equal weight to precision and recall. For each user model, the relevance score in the cluster selection protocol is given by the weighted F-score. The goal is to maximize this F-score.
We obtained the SRs and their relevant documents from the dataset published by Scells et al. [10]. It contains the PubMed ids of 177 randomly selected SRs published by the Cochrane Library plus their relevant documents, excluding relevant documents that lack PubMed ids. We selected the SRs from this collection that had 10 or more relevant documents.
The citation network was created based on the in-house Web of Science database at the Centre for Science and Technology Studies (CWTS) at Leiden University, the Netherlands, which includes papers published since 1980 (as well as a small number of papers published in earlier years). We excluded the documents in this database without PubMed id. We transformed citation links into undirected links (to comply with the requirements of our clustering algorithm). The SRs in the Cochrane Library database include the relevant documents in their reference list. Therefore our database also contains these citation links. These links are advantageous for the clustering algorithm, but they would not exist in a real IR task, so we excluded from the citation network all documents published in the same year as the SR and in later years. We selected as our focal year the year with the largest number of SRs. This was the year 2014 with 17 SRs. Therefore our citation network only included documents published before 2014. In the end, the citation network contained over 13.2 million documents and 280.4 million citation links. The SR set contained the 17 SRs published in 2014 and for each SR it contained the relevant documents present in the citation network.
We clustered the citation network with the Leiden algorithm [30] based on the methodology developed by Waltman and van Eck [31]. However, they built the clustering hierarchy in a bottom-up manner while we took a top-down approach. Also, they merged small clusters, which we did not do. The use of citation links to cluster documents is common practice in the field of bibliometrics. Textual information, for instance from the titles and abstracts of documents, is also often used to cluster documents. In this paper, we choose to use citation links. The use of textbased cooccurrence links could be explored in future research. We refer to Waltman et al. [32] for a comparison of different approaches for clustering documents. The clustering algorithm maximizes the following quality function:
In this quality function, i and j are documents, xi is the cluster of document i, and r is the resolution parameter. aij equals 1 if there is a citation link between documents i and j, otherwise aij equals 0. δ equals 1 if i and j are in the same cluster, otherwise δ equals 0. The value of r is given for each level of the clustering hierarchy (see below). The quality function ensures that related documents tend to be assigned to the same cluster. The higher the value of the resolution parameter, the larger the number of clusters and the smaller the number of documents per cluster.
We created a hierarchical clustering consisting of 7 levels. Starting from the highest level, at each level we applied our clustering algorithm to the citation network of the documents of each cluster obtained at the prior level, except at the highest level, where we applied the clustering algorithm to all documents in our citation network. At each lower level we multiplied the value of the resolution parameter by 10 to obtain smaller clusters.
At the highest level, we used the value 10 -7 for the resolution parameter. This is similar to the value of 8*10 -8 used by Waltman and van Eck [31] in their clustering solution with the lowest granularity. Using the value 10 -7 , we obtained a clustering solution in which 40% of the documents belonged to the largest cluster and 88% of the documents belonged to the 10 largest clusters. The size of the 15 largest clusters is shown in Figure 2.
In the end, we obtained a nested system of clusters and subclusters that was used for the evaluation. We removed clusters with fewer than 5 documents to avoid precision artefacts.
Figures 3 and 4 show the results of the tasks. Figure 3 shows the F-scores of each SR for the different user models. We observe that the F-score of most SRs increases monotonically from F0.25 to F4. Figure 4 shows the recall and precision values of each SR in the user models F0.25 and F4. Most SRs have higher recall and lower precision in F0.25 than in F4, 5 SRs have the same recall and precision, and 1 has both lower recall and lower precision.
The monotonic increase of the F-score from the F0.25 to the F4 user model observed in Figure 3 suggests that SciMacro is better at tasks that require high recall than tasks that require high precision. This inference is supported by the results observed in Figure 4. Here we observe that most tasks had higher recall than precision and that the user model F4 greatly increased recall at the expense of relatively little precision, especially when the recall of F0.25 was low.
The SR in Figure 4 for which the user model F0.25 has both lower recall and lower precision than the user model F4 is an instance where optimizing for precision was objectively worse than optimizing for recall. We will explore this phenomenon in more detail in the future. The fact that many SRs had the same precision and recall for both user models suggests that the selected cluster was unambiguously the best for these SRs.
In order to draw conclusions from the obtained F-scores, we need a comparison to other (query-based) search methods on the same testset. This is part of our currently ongoing work. We are also extending our evaluation set to include more SR tasks. However, we can make an informed guess of the effectiveness of SciMacro by comparing our results with the results reported by Scells and Zuccon [33]. They replicated the self-reported Boolean query searches of 51 SRs from the same test set as we used, and reported the average F-scores (Table 1), average precision and recall (Table 2) and average number of retrieved and relevant documents (Table 3). The Fscores and precision are 10 to 20 times higher for SciMacro than for the Boolean query searches, while the recall is 20% to 60% lower, depending on the user model. Therefore, we expect that SciMacro will perform well in precision-focussed tasks when compared with query-based search methods.
The answers to our research questions can be summarized as follows:
What is the potential of SciMacro for finding the relevant literature for SRs? The preliminary results presented in this paper do not allow us to draw strong conclusions. We will need further simulations on a larger benchmark set, a more direct comparison to query search, and ultimately a follow-up user study to answer this question. However, our informed guess, based on the results reported in prior work, is that SciMacro will perform well in precision-focussed tasks when compared with querybased search methods.
How do the user's preferences affect the performance of SciMacro? Our simulation has shown that SciMacro performs better at tasks focused on recall than precision.
At the moment, we are working on evaluating SciMacro using our current simulation setup and comparing it to other academic IR methods. Also, we are currently expanding our evaluation set with more SR tasks to present more rigorous results on the potential of SciMacro.