In this paper, we describe the participation of the Information and Language Processing System (ILPS) group at CLEF eHealth 2019 Task 2.2: Technologically Assisted Reviews in Empirical Medicine. This task is targeted to produce an e cient ordering of the documents and to identify a subset of the documents which contains as many of the relevant abstracts for the least e ort. Participants are provided with systematic review topics with each including a review title, a boolean query constructed by Cochrane experts, and a set of PubMed Document Identi ers (PID's) returned by running the boolean query in MEDLINE. We handle the problem under the Continuous Active Learning framework by jointly training a ranking model to rank documents, and conducting a \greedy" sampling to estimate the real number of relevant documents in the collection. We nally submitted four runs.
Systematic reviews are a type of literature review that uses systematic methods
to reliably bring together the ndings from multiple studies that address a
question and are often used to inform policy and practice, e.g. the development of
medical guideline in evidence-based medicine [
The CLEF eHealth Task 2 \Technology Assisted Reviews in Empirical Medicine
Introduction" aims to automate this process [
We participated Task 2.2 and submitted 4 runs: abs-th-ratio-ilps@uva, abshh-ratio-ilps@uva, doc-th-ratio-ilps@uva, doc-hh-ratio-ilps@uva. 2
Task 2.2 is Abstract and Title Screening. The participants are given the document collection extracted through the Boolean Search in Task 2.1, and are asked to produce an the e cient ordering of the documents, such that all of the relevant abstracts are retrieved as early as possible, and at the same time to identify a subset of A which contains all or as many of the relevant abstracts for the least e ort (i.e. total number of abstracts to be assessed). 3
3.1
The model
In this paper, we propose a novel model for the TAR process inspired by [
Let S denote the set of sampled documents and n denote the number of documents in S, Lt denote the labelled documents (the training set) at batch t and Ut denote the unlabelled documents at t. Initially, Lt is empty and we ll it with a pseudo document d0 which is made of the description of the topic provided. In line 3, k documents are uniformly sampled from Ut, and temporarily labeled non-relevant and added to Lt. In line 4, a ranking model is trained on Lt. In line 5-7, a sampling distribution Pt is constructed based on the ranked list of documents produced by the ranking model and a xed number of b documents are independently and with replacement sampled from Pt. In line 8, reviewers assess the relevance of the sampled documents. Note that the sampled b documents may contain duplicates, therefore reviewers only need to assess the unique documents. In line 10, Rct and vdar(Rct) are calculated. In lines 11-15, the stopping module uses Rct and var(Rct) to decide whether to stop or not.
d In line 17, produce the ordering of documents by sampled relevant, sampled non-relevant, un-sampled, with the stopping threshold at the rst un-sampled documents.
Algorithm 1: Automatic thresholding algorithm
Input: Target topic; document collection C, target recall rtarget.
Output: A list of retrieved documents with stopping threshold. 1 Lt = fpseudo document d0 labelled relevantg 2 while not stop do 3 We use the TF-IDF vector of a document as its features. Considering e ectiveness and e ciency we employ Logistic Regression as the ranking model. We use its implementation in scikit-learn1. At each batch t, a new model is trained from scratch using the current training data Lt. Sampling distribution Note that in Algorithm 1 we need to sample documents from a distribution Pt = npito (for notation simplicity we use P in this section). Ideally, the selection probability pi should be positively correlated with the relevance labels, which allows an estimator Rb with low variance (see Section 3.4). However, the relevance labels are not known until documents are assessed by the reviewers. What we have instead is a ranking model that can predict the 1 https://scikit-learn.org/ pi =
The second-order inclusion probability i;j { the probability of any two di erent document di and dj being included { is given by i;j = i + j h1
T
Y 1
t=1
pit
ptj nt i
Inclusion probability We derive the rst-order and second-order inclusion
probabilities, which is indispensable to calculate Rb. We adopt sampling with
replacement as our sampling method. At each batch t and for each draw, a
document is sampled independently from one of the aforementioned distributions.
Let selection probability denote the probability that a document is sampled for
a draw, and inclusion probability the probability that a document is included
in the sample set considering all the draws. Under sampling-with-replacement
design, the rst-order inclusion probability i is given by
probability of relevance and output a list of ranked documents, which we can
use to construct P. We use Power Law distribution which assumes the selection
probability of a document is a power function of its position in the ranked list,
de ned as
We employ Horvitz-Thompson estimator and Hansen-Hurwitz estimator to
estimate R and var(R). Both of them are designed for sampling with unequal
probabilities, Hansen-Hurwitz estimator is only restricted for with-replacement
sampling, while Horvitz-Thompson estimator is for any design. For more details
of the derivation the reader can refer to Chapter 6 in [
Horvitz-Thompson estimator The Horvitz-Thompson estimator provides an
unbiased estimator of population total under a general sampling theory [
In our case, the population total is the total number of relevant documents for a target topic, denoted as R = PN
i=1 yi. The Horvitz-Thompson estimator of R is
RHT = bt
X yi i2Set0 i where Set = [k=1Sk denote the accumulated sample set till batch t, yit is relet vance of document dit. We use 0 to denotes the operation to remove duplicated documents, andeto denote the operation to cumulate documents in all previous batches.
Hansen-Hurwitz estimator Hansen-Hurwitz estimator provides an unbiased
estimator of population total under sampling with replacement from the same
distribution [
In our case, the sampling distribution changes at each batch and converges to the ultimate distribution produced by the ranking model trained on the whole documents. The Hansen-Hurwitz estimator of R on St is (4) (5) RHH = 1 bt We propose a stopping strategy based on Rb. With sampling continuing, the strategy repeatedly examines whether rt0 > Rb rtarget, and if so stop TAR process.
e The intuition is straight forward, if we have collected more relevant than the target number we estimated, we should feel con dent to stop. 4
The dataset consists of 72 topics for training and 31 topics for testing. For each topic, participants will be provided with 1. Topic-ID 2. The title of the review, written by Cochrane experts; 3. The Boolean query manually constructed by Cochrane experts; 4. The set of PubMed Document Identi ers (PID's) returned by running the query in MEDLINE. 5
The proposed method is topic-wise in the sense that it repeatedly trains a new ranker based on the current assessed documents. It doesn't need extra training topics. Our runs are directly run on test data.
We submitted four runs: abs-th-ratio-ilps@uva, abs-hh-ratio-ilps@uva, docth-ratio-ilps@uva, doc-hh-ratio-ilps@uva. abs and doc denote whether qrels at abstract level or at content level is used for the relevance feedback in assessment module. th and hh denote whether Horvitz-Thompson estimator or HansenHurwitz estimator is used to estimate R. A description of each run is presented below. 1. abs-th-ratio-ilps@uva abs qrels and Horvitz-Thompson estimator 2. abs-hh-ratio-ilps@uva abs qrels and Hansen-Hurwitz estimator 3. doc-th-ratio-ilps@uva doc qrels and Horvitz-Thompson estimator 4. doc-hh-ratio-ilps@uva doc qrels and Hansen-Hurwitz estimator For all the four runs, we set
= 0:8, b = 100, k = 100, target recall rtarget = 0:8. 6
Our method targets on nding a stopping threshold given a target recall. We re-rank all the sampled relevant documents on the top, followed by all the nonrelevant documents and all the un-sampled documents. The stopping threshold is set at the position of the rst un-sampled documents. As all the sampled documents are before the stopping threshold, the stopping threshold also indicates the cost. As a consequence it is not valid to apply ranking metrics such as Average Precision, we report thresholding based metrics instead.
Table 1 shows the thresholding result on the test set. First, on both abs and content level, the Horvitz-Thompson estimator has recall threshold closer to the target recall 0.8 than the Hansen-Hurwitz estimator, which indicates a more accurate estimation of R. Second, both estimators stop at early stage when sampled documents are less than 50% of the complete documents.
Figure 1 and 2 shows the topic-wise recall threshold v.s. norm threshold. Horvitz-Thompson estimator stops at various recall for di erent topics, while Hansen-Hurwitz estimator stops between 0.8 - 1.0 for most topics. It indicates the estimation of R can help to stop viewing documents, but the variance of the estimated R is large for di erent topics. RUN norm threshold recall threshold abs-th-ratio-ilps@uva 0.423 0.838 abs-hh-ratio-ilps@uva 0.47 0.89 doc-th-ratio-ilps@uva 0.392 0.894 doc-hh-ratio-ilps@uva 0.426 0.95
In this paper, we presented the runs we submitted to the CLEF 2019 eHealth Task 2.2. We handle the problem under the Continuous Active Learning framework by jointly training a ranking model to rank documents, and conducting a \greedy" sampling to estimate the real number of relevant documents in the collection. We nally submitted four runs.
The result indicates the method can retrieve most relevant documents (around 80% to 90%) with the cost viewing less than 50% of the complete documents. The estimation of R can help to stop viewing documents, but the variance of the estimated R is large for di erent topics. Further work needs to be done to reduce the variance of the estimated R. 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 1.0 0.9 0.8 0.7 0.6
Topicwise recall_threshold v.s. norm_threshold (abs-th-ratio-ilps@uva).
0.1 0.2 0.3 0.4 0.5 0.6 0.7
0.8 Topicwise recall_threshold v.s. norm_threshold (abs-hh-ratio-ilps@uva).
0.2 0.4 0.6
0.8 (b) abs-hh-ratio-ilps@uva
Fig. 1: Topicwise recall threshold v.s. norm threshold at abs level. 1.0 0.8 0.6 0.4 0.2 0.0 1.0 0.8 0.6 0.4 0.2 0.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
(a) doc-th-ratio-ilps@uva Topicwise recall_threshold v.s. norm_threshold (doc-hh-ratio-ilps@uva).
0.2 0.4 0.6 0.8
(b) doc-hh-ratio-ilps@uva Fig. 2: Topicwise recall threshold v.s. norm threshold at content level.