Introduction

CLEF 2017 introduced a new eHealth retrieval problem -that of providing technological assistance to reviewers of systematic reviews -where the goals of the task were to explore how Information Retrieval techniques could be used to: (i) identify relevant material more quickly in the ranking challenge and (ii) identify when reviewers could stop processing documents in the thresholding challenge [3,2]. During the review process, reviewers will routinely examine hundreds to thousands of abstracts to decided if the document (and evidence it contains) could be included in the systematic review that they are conducting [5]. Once they have identified a subset of abstracts, which are potentially relevant, they examine the document's contents to decide whether the document should be included or excluded. The track focused on the first part, identifying potential relevant documents during the, so called, screening phase.

In this work we considered two different approaches -one which uses topic modelling and the other which uses relevance feedback. In selecting these approaches we thought that such techniques could be used in the following way. For topic modelling, we envisaged that the download abstracts could be semantically clustered -and the different clusters could be presented to the reviewerthe reviewer could then start the review process by selecting a cluster that they felt was most likely to contain the relevant documents. Since we did not have recourse to reviewers, we explored a number of different ways to automatically select the best cluster. For relevance feedback, we envisaged that as the reviewer starts to examine documents, the query could be updated to bring back the next most relevant documents, so that they would quickly find all the relevant material as soon as possible. Obviously, if the aim is to reduce the workload of the reviewers, then we need to be able to select a point where the reviewer can stop assessing documents -however, this runs the risk of losing relevant documents. To this end, we explore various heuristics to select the threshold such that we minimize effort and maximise recall (but ideally obtain total recall).

Experimental Set-up

Data: Given the list of topic descriptions the PubMed IDs were extracted, and a scripted fetched the Abstract and associated Metadata from the PubMed API. From the topics, we extracted the title for each topic and use that as the query. Indexing and Retrieval System: We used Lucene 6.2 to create a separate index for each of the topic (where stop words were removed, no stemming was applied). A Lucene Document was created where the following fields were index: Title, Abstract, Author, and Publication Name. The baseline retrieval algorithm we employed was fielded BM25 with standard parameters settings i.e. b = 0.75, and equal weights between fields (denoted as BM25). Relevance Feedback: We implemented Rocchio's Algorithm [4] in Lucenewhere feedback was used to provide relevance information. In each round of feedback, 30 documents were examined, and the query model updated, to provide a re-ranking of the subsequent documents. This was performed on the first 10%, 20%, etc of documents associated with the topic. Here, we only report the 30% runs (AL30) as these generally performed the best and little change in performance was observed on the training set using more feedback. Topic Modelling: We used MALLET toolkit, and thus Latent Dirichlet Allocation [1] to semantically cluster the documents within each topic. We set the number of latent topics to 5, and α =. To rank the documents we selected one of the latent topics z i , and ordered the documents by the probability p(z i |d|). In an attempt to select the cluster that provides the best ranking, we took the BM25 ranking from above, and use the top 100 ranked documents as pseudo-relevance feedback. Then we ranked each latent topic (given the ordering by p(z i |d|)) and select the one with the highest overlap with BM25 (TMBM). Combined: Since our topic modelling approaches do not use feedback, we decided to see whether we could start the process of relevance feedback using the topic modelling run, and refine the query model accordingly. Thus, we selected the best performing method TMAL, given the training data, and used this as starting point for the active learning. Again we considered obtained feedback for the first 10%,20% and 30% of the documents per topic. Thresholded Runs To create thresholded runs, we took the BM25 run and applied a simple score based threshold. Using Lucene the scores for a query given BM25 are from 1.0 or greater, so we used thresholds 1.0, 1.5, 2.0 and 2.5. This led to a reasonable reduction in the number of documents without sacrificing much recall.

Tables 1 and 3 report the performance on the Ranking Task, while Tables 2 and 4 report the performance on the Threshold Task. Our best performing run on the Ranking Task, in terms of normalized area under the gain curve, was AL30, which used 30% of the documents as feedback. From our results, it is clear that the topic modelling approach, as we have employed it, has not lead to significantly better improvements over the BM25 baseline. However, when inspecting the individual topic based ranking, the most probably topic, was not always the best performing topic. So we will direct more research into topic selection. This is because, when formulating queries, reviewers could use topic modelling to understand the space of documents retrieved, and then refine their query further -and thus make savings a priori rather than have to trawl through hundreds and thousands of documents. Another factor that could significantly improve our results is that for these initial runs we used the title as the query, as opposed to the boolean query provided within the topics. It is quite possible that the more complex and verbose boolean queries could lead to a better initial ranking -and so when used in conjunction with relevance feedback the relevant items could be found sooner. We leave these directions for further work.