In this paper we describe our participation to the CLEF eHealth 2017 Technology Assisted Reviews track (TAR). This track aims to evaluate and advance search technologies aimed at supporting the creation of biomedical systematic reviews. In this context, the track explores the task of screening prioritisation: the ranking of studies to be screened for inclusion in a systematic review. Our solution addresses this challenge by developing ranking strategies based on learning to rank techniques and exploiting features derived by the use of the PICO framework. PICO (Population, Intervention, Control or comparison and Outcome) is a technique used in evidence based practice to frame and answer clinical questions and is used extensively in the compilation of systematic reviews. Our experiments show that the use of the PICO-based feature within learning to rank provides improvements over the use of baseline features alone.
A systematic review is a type of literature review that appraises and synthesises the work of primary research studies to answer one or more research questions. Most authors follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) method for conducting and reporting these reviews. This includes the de nition of a formal search strategy to retrieve studies which are to be considered for inclusion in the review.
Given a research question and a set of inclusion/exclusion criteria, researchers undertaking a systematic review de ne a search strategy (the query) to be issued to one or more search engines that index published literature (e.g. PubMed). In medical and biomedical research, search strategies are commonly expressed as (large) boolean queries. After the search strategy has been executed, the title, and then abstract, of studies retrieved by it are reviewed in a process known as screening. Where the study appears relevant the full-text is then retrieved for more detailed examination.
The compilation of systematic reviews can take signi cant time and resources,
hampering their e ectiveness. Tsafnat et al. report that it can take several years
to complete and publish a systematic review [
The challenge of compiling systematic reviews can be fertile ground for
information retrieval (IR) research, as this can provide techniques to improve current
screening and screening prioritisation processes. The CLEF eHealth 2017
Technology Assisted Reviews track (TAR) [
We trained a learning to rank model using domain speci c features to provide an e cient ordering of studies retrieved by a systematic review. Speci cally, we aim to observe what e ect PICO features have with respect to learning to rank algorithms. PICO (Population, Intervention, Control or comparison and Outcome) is a technique used in evidence based practice to frame and answer clinical questions and is used extensively in the compilation of systematic reviews. We investigated several learning to rank algorithms and observed the e ect queries annotated with PICO elements had on the reordering of results compared to the original Boolean queries.
We trained two learning to rank models using both the original queries
provided by the task organiser, and another modi ed set of queries which contains
annotations from the PICO framework. In total, we used seven features to train
our learning to rank model. Table 1 summarises the features used. The rst four
features (IDFSum, IDFStd, IDFMax, and IDFAvg) calculate the inverse
document frequency (idf) for each of the terms in the document that also appear
Id Feature
1 IDFSum
2 IDFStd
3 IDFMax
4 IDFAvg
5 PopulationCount
6 InterventionCount
7 OutcomeCount
in the query. IDFSum sum of all idf scores, IDFStd is the standard deviation
of the idf scores, IDFMax is the maximum idf score and IDFAvg is the mean
idf score. The other three features (PopulationCount, InterventionCount, and
OutcomeCount) are the number of terms in the document and in the query
that also appear in the respective PICO annotation. PICO annotations for
documents were automatically extracted using RobotReviewer [
Initial testing on a recent collection we developed [
We then empirically evaluated the selected ve learning to rank algorithms listed in Table 2 and found that Coordinate Ascent provided us with the best MAP score on validation data for CLEF eHealth 2017 over the other models. Each time we trained a model, we used the default values for that model1 and we set aside the same 30% of queries for validation. Table 2 summarises the NDCG@10 and average precision (AP) scores for both the original Boolean queries and the annotated PICO queries. We found that Coordinate Ascent was the best algorithm for learning to rank these types of studies. Additionally, we found that Random Forests and MART methods both had similar levels of NCG@10 and AP.
Additionally, we also used Elasticsearch (version 5.3) to produce a re-ranking. We did this by issuing the Boolean and PICO query to Elasticsearch and limited the results to only the PubMed identi ers contained in the topic le for each query. We then let Elasticsearch rank these documents using BM25 with the default settings2. We considered the Elasticsearch runs as our baseline. 1 The default values for each model can be found at the following URL https:// sourceforge.net/p/lemur/wiki/RankLib%20How%20to%20use/ 2 k1 = 1:2, b = 0:75.
AdaRank Coordinate Ascent* LambdaMART Random Forests* Boolean 0.397 We found that a learning to rank approach to re-ranking studies for systematic reviews shows promising results. Table 2 illustrates our submitted runs compared to the baseline Elasticsearch ranking and additional runs performed post-submission. The models trained using the search strategies annotated using PICO achieved slightly better results than the provided Boolean search strategies. None of our models were able to score higher than the baseline in NCG@10, however the Coordinate Ascent model trained using PICO annotations outperformed the baseline in AP.
Additionally, we report AP, NCG@10, WSS@100 and the position of the last relevant document (last rel) in Figure 1. These visualisations show that the Coordinate Ascent model provides the most e ective ranking of documents (in terms of AP and WSS@100) and scores the highest amongst the learning to rank models for recall based measurements (NCG@10). Figure 1c shows that learning to rank models trained on the Boolean search strategies positioned the last relevant document in the re-ranked list the highest; and that the baseline Elasticsearch runs do not do this as well.
Figure 2 examines the e ect PICO had on re-ranking. The e ect appears negligible on the baseline, however, we notice an increase in precision when PICO annotations are used as training data for learning to rank models. This suggests that the use of PICO provides a trade o between precision and recall. Our results illustrate this clearly when precision-based measures are compared against recall-based measures. 4
We plan to further increase the precision of our experiments by tuning the hyper parameters of the best performing learning to rank models. Our learning to rank models were trained using only a small number of features. We will investigate the e ects of other features that are commonly used for learning to rank, and explore more domain speci c features in addition to PICO. pico_es bool_es pico_CA pico_LambdaMART pico_MART bool_CA pico_RF bool_LambdaMART
bool_RF bool_MART pico_AdaRank pico_RankNet bool_RankNet bool_AdaRank 0.0 pico_CA bool_es pico_es pico_LambdaMART
pico_RF pico_MART
bool_CA bool_LambdaMART
bool_MART pico_AdaRank
bool_RF pico_RankNet bool_RankNet bool_AdaRank 0.00 (c) Position in the re-ranked list of the last (d) Position in the re-ranked list of the last study retrieved (including baselines: bool es study retrieved (including baselines: bool es and pico es). and pico es).
Fig. 1: Comparison of the e ects each algorithm had on di erent measures. 0.25 0.0 0.2 0.4
0.6
Recall (a) Elasticsearch
0.25 n o is0.20 i c e r dP0.15 e t a l rop0.10 e t n I0.05 0.8 1.0 0.0 0.4
0.6
Recall (b) Coordinate Ascent 0.8
1.0
PICO 0.0 0.2 0.4 0.6 Recall 0.8 1.0 0.0 0.4 0.6 Recall