This paper describes the participation of the LIMSI-MIROR team at CLEF eHealth 2018, task 2. The task addresses the automatic ranking of articles in order to assist with the screening process of Diagnostic Test Accuracy (DTA) Systematic Reviews. We ranked articles by stacking two models, one linear regressor trained on untargeted training data, and one model using active learning. The workload reduction to retrieve 95% of the relevant articles was estimated at 82.4%, and we observe a workload reduction less than 70% in only two topics. The results suggest that automatic assistance is promising for ranking the DTA literature.
Systematic reviews seek to gather all available published evidence for a given
topic and provide an informed analysis of the results. This work constitutes some
of the strongest forms of scienti c evidence. Systematic reviews are an integral
part of evidence based medicine in particular, and serve a key role in informing
and guiding public and institutional decision-making. Systematic reviews for
Diagnostic Test Accuracy (DTA) studies have been shown particularly challenging
compared to other types of reviews because of the di culty in de ning search
strategies o ering acceptable recall [
Methods for automating the screening process in systematic reviews have
been actively researched over the years [
This paper describes the work underlying our participation in the CLEF 2018
eHealth Task 2 [
The remainder of this paper is organized as follows; Section 2 presents the dataset used for system development. Section 3 provides an overview of our system and describes each component. Finally, section 4 reports our results and section 5 provides an analysis of our methods and participation in the task. 2
In this work we have used the Clef dataset [
For each of the datasets we know the inclusion decisions based on the abstracts, as well as the inclusion decisions based on the full text. We thus have two de nitions of positive examples, depending on whether we use the abstract decisions or full text decisions as the gold standard.
We use a tripartite labeling to re ect this: { No (N) is the set of articles that were excluded based on the abstract { Maybe (M) is the set of articles that were preliminarily included based on
the abstract, but later excluded based on the full text { Yes (Y) is the set of articles that were included based on both the abstract and the full text, and later used in the meta-analysis cnrs static Our static ranker uses logistic regression trained on a large number (> 500,000) of features. This model is trained once on train split 1 (Table 1), and can then be used to rank candidate articles in any unseen DTA systematic review, without a provided search query or topic description. This model is intended to capture diagnostic test accuracy studies without considering whether the articles are topically relevant.
Absolute number Y M N 4 7 15065 24 54 14844 1 1 12699 1 47 274 3 36 3211 17 106 1398 5 2 3946 5 108 12591 47 48 2450 9 113 3757 14 18 1472 60 559 7582 64 53 1064 41 103 7847 49 166 1738 12 150 1422 41 35 43287 20 53 1243 28 426 7738 41 233 2940 11 19 10875 19 58 714 23 69 1523 14 38 929 0 45 7957 17 39 2729 48 154 10670 9 3 52 4 7 230 55 405 6071 10 15 2223 3 3 163 8 12 328 99 0 5121 46 58 5867 1 1 623 34 11 10462 79 59 6317 11 36 269 16 30 1865 10 13 5472 9 105 12689 3 1 1569 18 5 91 1 2 10314 15 9 2050 24 30 5441 6 4 2055 10 16 944 4 3 87 1 7 6149 1 126 7117 3 41 2461 5 6 311 0 26 8379 26 271 885 1 10 206 9 14 9853 2 53 9388 7 58 5579 33 566 14599 35 104 1720 19 556 7473 5 37 209 4 33 499 10 114 78679 29 11 4010 57 215 1639 35 44 9073 18 18 1370 9 7 145 38 15 4549 47 261 9914 42 18 872 117 187 9528 71 158 2756 30 39 1430 8 282 6540 7 5 2223 9 4 5987
Y
0.0%
0.2%
0.0%
0.3%
0.1%
1.1%
0.1%
0.0%
1.8%
0.2%
0.9%
0.7%
5.4%
0.5%
2.5%
0.8%
0.1%
1.5%
0.3%
1.3%
0.1%
2.4%
1.4%
1.4%
0.0%
0.6%
0.4%
14.1%
1.7%
0.8%
0.4%
1.8%
2.3%
1.9%
0.8%
0.2%
0.3%
1.2%
3.5%
0.8%
0.2%
0.1%
0.2%
15.8%
0.0%
0.7%
0.4%
0.3%
1.0%
4.3%
0.0%
0.0%
0.1%
1.6%
0.0%
2.2%
0.5%
0.1%
0.0%
0.1%
0.2%
1.9%
0.2%
2.0%
0.7%
0.0%
0.7%
3.0%
0.4%
1.3%
5.6%
0.8%
0.5%
4.5%
1.2%
2.4%
2.0%
0.1%
0.3%
0.1%
cnrs RF (uni-/bigram) We construct two relevance feedback (active learning) models uses
logistic regression on a smaller number ( 2,000) of features. These models are
trained using relevance feedback on the target topic, starting with the topic
description as an arti cial seed document. The unigram model is a
reimplementation of the cal model by Cormack and Grossman [
We describe each system in detail in the remainder of this section. 3.2
We here use a machine learning approach and train a classi er on the training
split, largely identical to the implementation of our static model submitted in
2017 [
We use logistic regression trained using stochastic gradient descent (sklearn) on a sparse feature matrix consisting of a large number (> 500,000) of features. We have tried using other classi ers, including svms, random forests, feed-forward neural networks, convolution networks and lstms, but logistic regression yields consistently better performance in our experiments with a fraction of the training time.
We handle class imbalance by class reweighting. We have implemented
undersampling mechanisms, but these tend to decrease performance. We set the
weight for the positive class to 80 for the initial intertopic classi er. We have
determined this to be a reasonable weight experimentally in previous experiments
on another dataset [
This model was trained on the 2017 training split. 3.3
We here use an active learning approach, where we at each timestep train a classi er (ranker) on the relevant articles screened so far. We start the process using the topic description as an arti cial seed document. The model is intended to capture topical relevance, and to use the data collected through the screening process, which is generally more targeted than the data we have available in the training split.
The model largely follows the continuous active learning approach of
Cormack and Grossman [
At each timestep we rank the candidate articles and show the top B articles to the oracle, and the oracle labels these as Y, M, or N. The number of articles B is initially set to 1 and is incremented by bBc at each timestep.
We use the following process to construct positive training data: { if Y have been encountered:
Then we use all encountered Y as positive training data. The synthetic seed document and any encountered M are discarded. { else if M have been encountered, but no Y:
Then we use all encountered M as positive training data. The synthetic seed document is discarded. { else (no Y or M have been encountered):
We use the synthetic seed document as positive training data.
To construct negative training data we sample 100 articles (or as many as remains) from the unseen candidates and temporarily label these N, irrespective of their true labels. Any articles already shown to the oracle are not considered for use as negative data.
We train our model on using the above positive and negative data to re-rank the candidate articles and repeat the process until all articles have been shown to the oracle.
This model only uses the candidate articles and the topic description as training data, and thus do not depend on other training data, such as the topics in the training split. 3.4
We use a three-layer dense neural network as a function approximator to estimate the joint score for a candidate document given the scores from our static and active models. We use 16 nodes in each layer, apply 30% dropout after each layer and use softmax activation on the nal layer to simulate two-class logistic regression.
The model is trained by sampling training data uniformly from recorded active learning output. We have tried using uncertainty sampling, but this has yielded inferior results.
As input to the model we use the score values we get from the static and active learning models, along with meta-level features. The full set of features is as follows: 1. Static model document score (static) 2. Active model document score (RF bigram) 3. Number of Y found 4. Amount of relevance feedback (absolute number) 5. Amount of relevance feedback (percentage) 6. Relevance feedback stage (whether using seed, M or Y as positive training data) Features 3 and 4 are normalized using the following log transform sgn(x) log2(1 + jxj) 8 to keep numbers in mainly in the range [0; 1]. We do not truncate large numbers. Feature 6 take discrete values in f 1; 0; 1g
However, we observe that features 5 and 6 decrease model performance and we therefore excluded these in the model used in our o cially submitted runs.
This model is trained on data generated from training split 2 (Table 1) to avoid over tting. We generate the training data for the stacked model by letting the active model run on the training data, and at each step in the process we record the score generated by the active learning model, as well as the above features. We do this 100 times for each topic. One data point thus consists of the score from the static model (feature 1), and features 2{6 from this pre-generated data.
We train the stacked model on data sampled randomly from this pool of data points, by sampling 50 runs in each iteration, and sampling an equal number of positive and negative training examples from each run (with a minimum of 20 total). The model is trained on a batch of size 32. The training data is resampled every training iteration. 4
We present our results for average precision in table 2, WSS@95 in table 3, WSS@100 in table 4, Last Rel in table 5, as well as the aggregate scores in table 6. For comparison, we also calculate a baseline by evaluating each metric on the data ordered randomly. The baseline values are calculated using the average and the standard deviation of 1000 repetitions.
The RF unigram, and RF bigram, and the combined model were submitted as our o cial runs.
The results omit one topic with no Y (CD010680). 5 5.1
One of the topics in the CLEF dataset, CD010653, has no Y. While we can still calculate performance scores relative to M, this topic might arguably have been omitted from the test data. One of the topics, CD008803, similarly has no M.
This also happens to be the topic with the second largest number of Y.
As a general tendency, we can observe that the relative number of Y / M / N in the CLEF dataset varies dramatically across topics. At the one end we have one topic consisting of 14.06% Y (CD008760), and one topic consisting of 15.79% Y (CD010705). At the other end we have ve topics with less than 0.1% Y (CD011548, CD011549, CD012019, CD011515, and CD009263). The number of N also varies wildly, from 52 up to 78,679.
RF Table 2: Average precision score for each topic, evaluated using either inclusion decisions based on full text (YjjMN), or based on abstract and title (YMjjN). The combined model uses the static and RF bigram as subcomponents. Table 3: WSS@95 score for all topics in the CLEF dataset, evaluated using either inclusion decisions based on full text (YjjMN), or based on abstract and title (YMjjN).
The combined model uses the static and RF bigram as subcomponents. Table 4: WSS@100 score for all topics in the CLEF dataset, evaluated using either inclusion decisions based on full text (YjjMN), or based on abstract and title (YMjjN).
The combined model uses the static and RF bigram as subcomponents.
RF Metric static unigram bigram combined baseline static unigram bigram combined baseline
AP 0.169 0.176 0.124 0.203 0.014 0.000 0.313 0.314 0.218 0.337 0.053 0.000 WSS@95 0.741 0.815 0.668 0.824 0.104 0.024 0.513 0.617 0.519 0.657 0.028 0.009 WSS@100 0.640 0.762 0.633 0.779 0.130 0.024 0.349 0.460 0.339 0.510 0.027 0.007 Last Rel 3349.448 1305.034 3798.000 1224.655 6405.696 272.238 5708.400 5173.467 5500.600 4378.900 7131.769 36.629 No single model performs best on all topics. Generally however, RF unigram consistently outperforms the static model, and the combined model (static + RF bigram) outperforms the other three models.
Surprisingly, the RF unigram model consistently outperforms the RF bigram model, despite using a subset of the features of the RF bigram model.
For this reason it seems likely that a stacked model consisting of the static model and the RF unigram model would have achieved better performance than the stacked model submitted as our o cial run.
The RF unigram model is particularly adept at nding all relevant articles, resulting in better last rel score than the static model for 19 topics out of 29, and a better last rel score than the RF bigram model for 24 out of 29. This also results in a WSS@100 score of 76.2% for the RF unigram, versus 64.0% for the static model, and 63.3% for RF bigram.
Note however that last rel generates scores of wildly varying scale, and the large last rel scores for static and RF bigram are therefore almost entirely due to a few large outliers. In particular, 59% of the information contained in the last rel score for RF bigram is due to a single topic with a large number of candidate articles (CD009263). The metric may thus be useful when interpreted on individual topics, but not when averaged. The WSS@100 metric, which is equivalent to last rel on individual topics, produces scores on the same scale and therefore makes sense also when averaged. 6
Our best system combines a static model and a relevance feedback model using stacking. The workload reduction to retrieve 95% of relevant articles is estimated at 82.4% on average, with a minimum workload reduction of 47.6%, and a maximum workload reduction of 94.7%. The workload reduction is consistent across topics, and we note a workload reduction less than 70% in only two topics. Due to the highly variable number of candidate articles in di erent topics, however, we may still need to screen several thousands of articles to nd all relevant articles in any given systematic review.
Our remarks on the implementation of the shared task model and task
organization from last year [
This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 676207.