Systematic reviews require researchers to identify the entire body of relevant literature. Algorithms that lter the list for manual scanning with nearly perfect recall can signi cantly decrease the workload. This paper presents a novel stopping criterion that estimates the score-distribution of relevant articles from relevance feedback of random articles (S-D Minimal Sampling). Using 20 training and 30 test topics, we achieve a mean recall of 93.3%, ltering out 59.1% of the articles. This approach achieves higher F2-Scores at signi cantly reduced manual reviewing work loads. The method is especially suited for scenarios with su ciently many relevant articles (>5) that can be sampled and employed for relevance feedback.
Systematic reviews give a comprehensive overview of all published evidence on
a given topic. It has been estimated that every year, more than 4000
systematic reviews are conducted and published with each review requiring at least
6-12 months of preparation time [
Systematic reviews of diagnostic test accuracy (DTA) compare the e ectiveness of index tests for a target condition. Filtering relevant studies for DTA reviews has been identi ed to be exceptionally challenging due to an increased class-imbalance, a broader-than usual target class de nition, and a lack of metadata quality e.g., missing abstracts [12]. However, signi cant advances in this domain are expected to be applicable to other areas as well. Due to unreliable performance, the Cochrane Organization, a leading authority in systematic DTA reviews does not, currently, recommend to use any search lters in the review process [13].
This paper describes ETH Zurich's participation in \Task 2: Technologically
Assisted Reviews in Empirical Medicine" at the CLEF eHealth Evaluation lab
2017 [
We propose a learning to rank pipeline that extracts information for each article from PubMed, creates numerical features from this information, ranks each article and nally determines a cut-o point on the ranked list based on a novel score distribution approach.
The remainder of this document is structured as follows. Section 2 describes our learning-to-rank system alongside a statistical stopping criterion for manual result list inspection. Section 3 empirically compares the proposed method with a wide range of state-of-the-art baselines. Section 4 discusses a number of qualitative observations and Section 5 concludes with an outlook on future research directions. 2
In order to judge the relevance of an article for a given query, we propose a learning-to-rank pipeline that extracts information for each candidate article and represents them as dense numerical feature vectors. These vectors can be used to train ranking systems that create ordered lists of articles. In a nal step the algorithm decides where to cut o the ranked list for manual inspection.
For each article, we extract the title, abstract, MeSH headings, a list of publication types and the publication language via the NCBI EUtilities1. MeSH headings are a list of tags from a comprehensive controlled vocabulary for indexing journal articles in the life sciences. While a title is available for all papers, the abstract is missing in many cases (12.2% on the training set). 2.1
From this data we extract a number of features and group them into two categories: dynamic and static features. While static features only depend on the article, dynamic features depend on the article in relation to each query. Static features will capture the aptness of an article to be included in any systematic review, while dynamic features express its relevance for the query at hand. Static features include a similarity score between each article's 128-dimensional Doc2Vec embedding [11] and an average embedding of relevant documents, a number of statistical features denoting how likely publication type and language are relevant, the publication year and the number of words in abstract, title and MeSH headings. Dynamic features include a tf-idf similarity measure of query 1 https://www.ncbi.nlm.nih.gov/books/NBK25500/ text and query title to document title, abstract and MeSH headings. The tf-idf score is calculated using Lucene and is optimized using a stop word lter. Also included in the dynamic features is the cosine similarity of the document embeddings between the query and various document elds. For this purpose also the query and query title are mapped to a vector using Doc2Vec. The method includes a total of 51 features the respective e ectiveness of which will be discussed in Section 3.
occurrence of words that frequently appear in relevant documents for arbitrary DTA reviews was used to lter the articles [14]. We propose a deep learning approach that models the similarity to frequent words by creating a document embedding of a common relevant document.
We create the vectors
Doc2V ecq+ = Doc2V ecq =
X d2v(a) a2Aq+ jAq+j X d2v(a) a2Aq jAq j where d2v denotes the Doc2Vec representation of an article. Now we create a vector that averages document representations of relevant articles from training topics Q
Doc2V ec+ = X Doc2V ecq+ q2Q jQj This vector contains the word embedding of the average relevant article in our training set. We expect this vector to produce words that frequently appear in relevant articles and that are similar to the ones proposed by Vincent et al. [14]. The 20 words most likely contained in our document embedding Doc2V ec+ closely resemble the list proposed by Vincent et al.: chronic, abnormal, clinically, diagnosis, complication, diagnose, patients, treatment, diagnosing, non-invasive, patient, dysfunction, tissue, symptomatic, abnormalities, minimally, treatments, diagnostic, complications, insu ciency
Similarly to modelling this centroid of relevancy, we can subtract the average irrelevant article across training queries, giving:
Doc2V ec+ = X Doc2V ecq+
Doc2V ecq+ q2Q jQj If we generate related words we get: diagnosis, non-invasive, diagnose, diagnosing, diagnostic, imaging, clinically, patients, patient, chronic, abnormal, minimally, treatment, complication, assess, noninvasive, evaluating, helpful, assessing, scans The similarity of title, abstract and MeSH headings to vectors created by both methods become ranking features. We call the rst approach the classic method, the second one the di erence method. This score is calculated for title, abstract and MeSH headings, respectively.
MetaMap MetaMap2 is a tool that maps biomedical text to the UMLS
Metathesaurus using symbolic, natural-language processing (NLP) and
computationallinguistic techniques. It is a state-of-the-art library that has shown to be highly
e ective [
Treating missing values After creating features for each article and its topic combination, we process the feature le to account for missing data. For missing abstracts, we set all abstract related features to the average on the training set and do the same for MeSH headings. We tried setting these features to an unused value such as -100, which produced worse results. 2.2
On the basis of the previously described features, we use Ranklib3 to train a
number of rankers and validate their performance. After evaluating a broad
range of models including coordinate ascent, MART [
2.3
While generating a model to score the relevancy of articles is essential to nding a set of relevant articles, it is equally important to nd the right point to stop retrieving more documents. In order to threshold the ranking, we propose a naive baseline technique that cuts o at a xed rank and an extended baseline that will decide based solely on previous distributions of relevant articles according to their retrieval model score.
In order to nd a suitable cuto method we need to de ne a metric to optimize for. In a systematic review we emphasize recall over precision and reliability of optimality. Additionally, we would like to use as little manual relevance feedback as possible for our ranking. The optimal cost Copt for some optimal rank r trading o between recall, precision and relevance feedback can be found according to:
Copt =
min 8r2(1;jAj)
f (r; jA+<rj; jA+>rj; jRj) for any weighting formula f , a list of articles A, a list of positive articles that are retrieved A+<r, a list of positive articles that are not retrieved A+<r and a set of documents the method got relevance feedback for R. A weighting formula that weights precision with and recall with looks the following: f (r; jA+<rj; jA+>rj; jRj) = (1 jA+<rj ) + r (1 jA+<rj ) jA+>rj Static cuto models We use two very simple baseline models that cut o the list at a xed rank r or a xed score s for every topic. The parameters r and s can be t on the training topics. We use an improved static method that normalizes the scores for each ranking linearly, which yields better results than the original version.
BMI Method We take as another baseline method the default TREC rule for stopping to read a ranked list using relevance feedback for each document. We stop when the number of documents reviewed exceeds 2R+1000, where R+ is the number of relevant documents retrieved so far.
Knee Method This method locates a so-called \knee" or negative in ection point in the gain curve of relevant documents. The gain curve indicates for each rank x how many relevant documents were found up to that rank. The method stops when the slope following the knee is less than 1 of the slope before the knee and the index is higher than some .
Classic S-D Method Score distributions (S-Ds) of relevant and irrelevant documents have been studied since the early days of IR. By modelling the scoredistribution of relevant documents P+ for a topic we can estimate the number of relevant documents jA+<rj that are retrieved until rank r using jA+<rj =
Z r 1
P+(score(x)) dx
where score(x) denotes the score at rank x. We can approximate the best cuto
according to some metric that uses jA+<r j for all r and r. Using the distribution
we can estimate a cost function for all possible cuto points and select the
best position. We follow Arampatzis et al. [
In the simplest case we give each document the same probability j A1j to be sampled and t a Gaussian distribution P+feedback on X+. However, notice that we do not use the irrelevant articles at all. Sampling irrelevant documents will create additional work and may, in some cost functions, be penalized for the use of relevance feedback as well (e.g. CostU niRF ). It is therefore desirable to ask for feedback on as few irrelevant articles as possible. However, since the classes are very imbalanced, if we sample each document with uniform probability we will get much more irrelevant documents than relevant ones. We correct for this observation by boosting the sampling-probability of articles with a high score which will increase the probability to get feedback on a relevant article. We use a sampling distribution Psample(s) that assigns a sampling probability according to the score s = score(x) of an article x now. During sampling a random score s with probability Psample(s) is drawn and the document with the closest score is sampled.
If we try to t a Gaussian distribution to the data X+ after the optimized sampling step we notice that the mean of the resulting distribution is biased towards higher scores. A document with score s will be in X with the sampling probability Psample(s) instead of the true probability of a score in the entire data set Poverall(s). Thus, a document with score s will be in the sampled data X with Psample(s)/Poverall(s) times the true probability. A distribution that is t on the data X will therefore show approximately the same bias. In order to remove this bias from the sampled data X we discretize the scores into N chunks Chunk = [1; ::; N ] associating a score scorec(n) = s0 + nnsn from the lowest score s0 to the highest score sn to the pieces. For each chunk we calculate the sampling bias
B(n) =
Poverall(scorec(n)) Psample(scorec(n) ; n 2 Chunk , where Poverall is obtained for each score by tting a Gaussian to the score distribution of all articles A. The bias B(n) indicates how much more often a data point in the chunk n was sampled relative to how often it would be sampled if every document had the same probability. Now we obtain an unbiased P^+ N (m^u; ^) by tting a Gaussian to the n data points PsampleB(c(nh)unk(n)) 8n 2 Chunk.
The standard deviation ^ was found by using the data points of the n chunks instead of the raw sampled data X. We nd a better standard deviation by iteratively testing the t of P+ N (m^u; ) on the sampled data X and selecting the best . We test the method with two sampling distributions: the uniform distribution and a triangular distribution with Ptriangular(s) = (s+ ) with such that Ptriangular(s) is a probability distribution and the o set such that P (s) > 0 for all scores.
Some topics will have very few relevant documents. In the most extreme case just a single one, making it impossible to make a robust estimate. A very low number of relevant documents from sampling will result in a high variance in the score distribution of the relevant samples which will decreases the similarity of our predicted score distribution to the true one. For a very low number of relevant sampled articles jAs+ampledj < the method will perform worse than some other method M 0 for some parameter (e.g. 4). Fitting a Gaussian on the relevant articles requires at least two relevant articles in the sampling phase. Thus, in both cases the method should use the alternative technique M to handle this run. The method M can be any of the methods described before. We introduce a parameter that de nes the sampling rate = jAsajmApjledj of sampled articles and use the xed score method if jAsampledj < for some optimal on the training set. 3 3.1
The experiments are evaluated on the data set provided by the CLEF 2017 Task
2 eHealth Challenge [
Work Saved over Sampling (WSS) WSS is an intuitive measure that indicates how much less work w(r) has to be done to achieve a recall r on the set of articles A if an optimal cuto is applied to the ranking, compared to an unordered list. Since in an unordered list we need to examine a fraction of r documents to achieve recall r we get:
W SS(r) = r w(r)
A j j share of relevant documents missed jA+>rj weighted by (jAj
jA+j CLEF 2017 Task 2 Uniform Cost CLEF eHealth Challenge Task 2 2017 provides , , and -parametrized cost based metric that measures the performance of a ranking of documents j j with relevant documents A+ that is cut
A o at rank r. The cost depends on the amount of relevance feedback used jRj weighted by , the e ort r jRj to review the documents weighted by and the r).
C = (r jRj) + (jAj r)( jAA+>+rjj ) + jRj j The weights proposed by CLEF are = 1, = 2, = 2, we denote the measure as CostU niRF . In a more realistic setting we would not get punished for using relevance feedback ( = ), which we will simply call CostU ni. Note that the o cial evaluation script assigns = 3, likely because a document using relevance feedback has cost for being shown plus cost for the feedback. We use = 2 as indicated on the o cial evaluation measure description. 3.3
Table 1 reports WSS@95, precision, recall and last relevant document scores
obtained for each of the 30 test topics. Unless stated otherwise we use the minimal
feedback based S-D method as a stopping criterion. The mean WSS@95 score is
0.544 at precision 0.091 and recall 0.933. Precision and recall compare favorably
to the search lters that are reviewed in the Cochrane handbook [13], albeit on
di erent datasets. Howard et al. [
Features We assess the descriptive power of features from three di erent content elds: Abstracts, Title and MeSH headings (see Table 2) by training our model only using features that are derived from one content eld at a time, excluding all other sources of evidence. We nd MeSH headings to only insigni cantly improve the performance of our ranking. Abstracts are the best source for ranking the documents even though they might not be present in some cases. We also compare dynamic features, that show the similarity of query and document with static features, that re ect the general aptness of a document for a systematic DTA review in Table 2. Using only dynamic features yields a MAP score of 0.239, while using only static features yields a MAP score of 0.148. Adding both feature groups together results in a MAP of 0.2866.
We further asses the e ectiveness of six di erent feature groups: Gensim Similarities, tf-idf, Metamap, Gensim Relevancies, language and publication types in Table 3. For these features we estimate the importance of a feature group by observing the e ect of systematically removing one group at a time and comparing the results to the original results obtained using all features. We nd features that are not text related such as language and publication type will only weakly a ect the ranking. Also tf-idf achieves the highest gain among the three methods to estimate similarity of a query to a document, even though it does not natively consider synonyms and word combinations.
Stopping Criteria Table 4 examines the e ectiveness of eight cuto methods according to the evaluation measures above. We nd the sampling S-D methods to be more e ective than all other methods on F2-Score, CostU niRf and CostU ni, while CostU niRf was optimized by it. Minimal S-D sampling proves 11.2% more e ective on the average F2-Score than the best non-S-D sampling method on average. The average cost CostU ni is 17:1% lower than on the best non S-D sampling method. Using CostU niRf and thus penalizing relevance feedback the method is 9:2% stronger than the second best contestant. The cost of Minimal S-D sampling is 4:3% lower than using uniform S-D sampling.
The parameters of our stopping criteria were optimized on the 20 available training topics. This yielded an optimal average normalized score of 1.45 and Dynamic Features Static Features Overall Random tf-idf, Gensim Similarity, Gensim Relevancy, Number of words tf-idf, Gensim Similarity Gensim Relevancy, Number of words, 0.148 Language, Publication type an optimal cuto rank of 1410. For the knee method we found a slope after the knee parameter of = 9 and the minimum cuto rank = 500 to be optimal. For the relevance-based S-D methods we found the sampling = 5% and the minimum number of feedback = 4.
Fit to normal distribution The S-D based stopping methods assume the
score distribution of relevant documents P+ to be Gaussian distributed.
Figure 1 visualizes the t of a standard normal distribution to the summed up and
normalized score distributions of each topic. Aside from the intuitively
appealing t, we conduct a Kolmogorov-Smirnov test of normality [
0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0-3 -2 -1 0 1 2 3
Fig. 1. Shape of positive score distribution compared to normal gaussian
relevance-based S-D methods rely on removing the sampling bias from the data. Figure 2 compares the distribution before and after being corrected on some examples. We observe the goodness-of- t to be much higher and very close to the actual distribution after the correction. By increasing the amount of relevance feedback we can increase the goodness-of- t of our distribution to that of the best normal distribution.
4 3
Stopping Criterion Figure 3 shows the correlation of CostU nif orm of our algorithm against the number of relevant articles sampled in each run. We evaluate the same ten topics multiple times, each time sampling until we reach N relevant documents. As expected, when the number of sampled relevant documents increases the cost of the run decreases. Increasing the sample size will decrease the variance and thus improve the predictive quality of the method. However, if we continue increasing this number the cost of sampling will grow as it will be harder to nd more relevant documents. 4 4.1
Static features that do not depend on the topic query but only on the article can be observed to perform much worse in predicting the relevance of an article. Dynamic features however rely heavily on the initial query which is created by human experts using the system and will thus be written in a very di erent way for each topic. Also, researchers have a focus on di erent properties of a 3
N4umber of5relevant6documen7ts samp8led 9
10 6000 5000 5
10 15 20 25 Number of relevant documents sampled 30
35 Fig. 4. Correlation of CostUniform and number of relevant documents sampled by the minimal sampling method on the 30 test topics document. Figure 5 shows relevant articles from three topics. They are arranged on the plane such that we can see the sum of dynamic features on one axis and the sum of static features on the other. As predicted by the above hypothesis, we see that the preference for inclusion for these two feature groups is very di erent. In one topic A three articles are included that have low static features, while in topic B there are only articles with high static features. Also probably due to a query that was created very di erently, we observe low dynamic features in topic B, but much higher ones in topic A.
An e ective system will need to make use of relevance feedback for the articles in order to be able to capture the great di erences between each topic and rank the documents more accurately. Human experts will also judge di erently which articles need a full content scan and which can be refused immediately. This is re ected in the wide score range of relevant documents that we observe on the data in absolute numbers (2 to 460) as well as relative numbers (0.029% to 20%). In addition, the quality of a ranking will vary depending on medical focus and the initial query. Figure 6 shows the scores of relevant documents in the 30 test topics. As expected each topic shows a unique score distribution with very di erent means. We conclude that the cuto point will be hard to predict using the scores or ranking position without using any relevance feedback. Systems using relevance feedback rely on sampling some relevant documents which is hard due to the high class imbalance in the data of systematic DTA reviews [12]. When sampling randomly one might need to include up to 3000 irrelevant documents to sample a single relevant document. Therefore, a system using relevance feedback needs a high initial ranking quality and should use feedback mostly at the top of this ranking. 3.5 m u trS 3 e u a e F itc2.5 a t S 2 Petersen et al. [12] hypothesize that missing metadata is one of the main reasons that systematic DTA reviews are di cult to support with an IR system. In our experiments abstracts have shown to be by far the most e ective eld in predicting the relevance of a document, however they are missing in 12.3% of all documents in the test data. We suppose that abstracts that are missing in our records were available to the researchers judging their relevance. Thus the probability that a document is relevant should not depend on the presence or absence of abstracts. Figure 7 shows that our model ful lls this property. Documents that lie on the x-axis do not have an abstract text in our data, but their frequency at the end of the ranking is similar to that of documents with an abstract text. Also, the length of the abstract text does not seem to in uence the relevancy in our model. 5
In this paper, we present a recall-centric learning-to-rank scheme accompanied by a statistical cuto criterion that identi es the optimal point for stopping human inspection of results by estimating the score-distribution of relevant documents in a biased sampling process. Our experiments show that this approach is able to reduce the size of the ranked list by more than half while retaining a recall close to 95% without using relevance feedback in the ranking step. This level of performance signi cantly exceeds the results obtained by traditional cuto methods.
In the future, we will move beyond the currently employed simple Gaussian score distributions in favor of more accurate approximations of the true distribution of relevance. Additionally, we plan to further evaluate this method in other recall-driven domains such as e-discovery. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Position in ranking in % 11. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Neural and Information Processing System (NIPS) (2013), https://papers.nips.cc/paper/5021-distributedrepresentations-of-words-and-phrases-and-their-compositionality.pdf 12. Petersen, H., Poon, J., Poon, S.K., Loy, C.: Increased workload for systematic review literature searches of diagnostic tests compared with treatments: Challenges and opportunities. JMIR Medical Informatics 2(1), e11 (may 2014), https://doi.org/10.2196/medinform.3037 13. de Vet HCW, Eisinga A, R.I.A.B.P.D.: Chapter 7: Searching for studies. in:
Cochrane handbook for systematic reviews of diagnostic test accuracy (2008) 14. Vincent, S., Greenley, S., Beaven, O.: Clinical evidence diagnosis: developing a sensitive search strategy to retrieve diagnostic studies on deep vein thrombosis: a pragmatic approach. Health Information & Libraries Journal 20(3), 150{159 (aug 2003), https://doi.org/10.1046/j.1365-2532.2003.00427.x 15. Wu, Q., Burges, C.J.C., Svore, K.M., Gao, J.: Adapting boosting for information retrieval measures. Information Retrieval 13(3), 254{270 (sep 2009), https://doi.org/10.1007/s10791-009-9112-1 16. Xu, J., Li, H.: Adarank: A boosting algorithm for information retrieval. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 391{398. SIGIR '07, ACM, New York, NY, USA (2007), http://doi.acm.org/10.1145/1277741.1277809