s for inclusion in the review. Seven teams participated in the task, with a total of 12 runs submitted for subtask 1 and 19 runs for subtask 2. This paper reports both the methodology used to construct the benchmark collection, and the results of the evaluation.
Evidence-based medicine has become an important pillar in health care and
policy making. In order to practice evidence-based medicine, it is important to
have a clear overview over the current scientific consensus. These overviews are
provided in systematic review articles, that summarize all available evidence
regarding a certain topic (e.g., a treatment or a diagnostic test). To write a
systematic review, researchers have to conduct a search that will retrieve all the
studies that are relevant to the topic. The large and growing number of published
studies, and their increasing rate of publication, makes the task of identifying
relevant studies in an unbiased way both complex and time consuming to the
extent that jeopardizes the validity of their findings and the ability to inform
policy and practice in a timely manner. Hence, the need for automation in this
process becomes of utmost importance. Finding all relevant studies in a corpus
is a difficult task, known in the Information Retrieval (IR) domain as the “total
recall” problem [
To this date, the retrieval of studies that contain the necessary evidence to inform systematic reviews is being conducted in multiple stages: 1. Identification: At the first stage a systematic review protocol, which describes the rationale, hypothesis, and planned methods of the review, is prepared. The protocol is used as a guide to carry out the review. Beyond other information, it provides the criteria that need to be met for a study to be included in the review. Further, a Boolean query that attempts to express these criteria is constructed by an information specialist. The query is then submitted to a medical database containing titles, abstracts, and indexing terms of a controlled vocabulary of medical studies. The result is a set, A, of potentially relevant studies. 2. Screening: At a second stage experts are screening the titles and abstracts of the returned set and decide which one of those hold potential value for their systematic review, a set D. If screening an abstract has a cost Ca, screening all |A| abstracts has a cost of Ca jAj. 3. Eligibility: At a third stage experts are downloading the full text of the potentially relevant abstracts, D, identified in the previous phase and examine the content to decide whether indeed these studies are relevant or not. Examining a document has typically a larger cost than the cost of examining an abstract, Cd > Ca. The result of the second screening is the set of studies to be included in the systematic review.
Unfortunately, the precision of the Boolean query is typically low, hence reviewers often need to manually examine many thousands of irrelevant titles and abstracts in order to identify a small number of relevant ones. Further, there is no guarantee that the Boolean query will retrieve all relevant studies, jeopardizing the validity of the reviews. To overcome some of the limitations of the Boolean search, researchers have been testing the effectiveness of machine learning and information retrieval methods. O’Mara-Eves et al. [15] provide a systematic review of the use of text mining techniques for study identification in systematic reviews.
The focus of the CLEF 2018 e-Health Technology Assisted Reviews in
Empirical Medicine (TAR), similar to last year [
The goal of the lab is to bring together academic, commercial, and government researchers that will conduct experiments and share results on automatic methods to retrieve relevant studies with high precision and high recall, and release a reusable test collection that can be used as a reference for comparing different retrieval and mining approaches in the field of medical systematic reviews.
This paper is organized as follows: Section 2 describes the two subtasks of the lab in detail, Section 3 describes the constructed benchmark collection, and Section 4 the evaluation measures used; in Section 5 we briefly describe the participating systems, and in Section 6 we discuss the results of the evaluation.
2
In this section we describe the two subtasks of the TAR lab, the input provided to participants for each one of the subtasks and the expected participant’s output submitted to the lab for evaluation. 2.1
Prior to constructing a Boolean Query researchers have to design and write a systematic review protocol that in detail defines what constitutes a relevant study for their review. In this experimental task of the TAR lab, participants are provided with the relevant pieces of a protocol, in an attempt to complete search effectively and efficiently by-passing the construction of the Boolean query.
In particular, for each systematic review that needs to be conducted (also referred to as topic in the IR terminology), participants are provided with the following input data:
Participants, the Index Tests, the Target Conditions, and the Reference Standards; 4. the PubMed database, provided by the National Center for Biotechnology Information (NCBI), part of the U.S. National Library of Medicine (NLM).
Participants are provided with 30 topics on Diagnostic Test Accuracy (DTA) reviews. For each one of these topics participants are asked to submit: (a) a ranked linked of PubMed articles, and (b) a threshold over this ranked list. Participant can submit up to 3 submissions (“runs”). A run is the output of the participants’ algorithm for all the topics, in the form of a text file, with each line of the file following the format:
TOPIC-ID
THRESHOLD
PMID
RANK
SCORE
Each line represents a PubMed article in the ranked list for a given topic, with RANK indicating the index of this article in the ranked list. TOPIC-ID is the id of the topic for which the document has been retrieved, and THRESHOLD is either 0 or 1, with 1 indicating that the given rank is the rank of the threshold. PMID is the PubMed Document Identifier of the article ranked at that position, SCORE is the score the algorithm gives to the article, and RUN-ID is an identifier for the submitted run. Participants are allowed to submit a maximum of 5,000 ranked PMIDs per topic, i.e. a total maximum of 150,000 lines per run. Given the results of the Boolean Search from the first stage of the systematic review process as the starting point, participants are asked to rank the set of abstracts. The task has two goals: (i) to produce an the efficient ordering of the documents, such that all of the relevant abstracts are retrieved as early as possible, and (ii) to identify a subset which contains all or as many of the relevant abstracts for the least effort (i.e. total number of abstracts to be assessed).
In particular, for each systematic review that needs to be conducted (also refereed to as topic in the IR terminology), participants are provided with the following input data:
4. the set of PubMed Document Identifiers (PMID’s) returned by running the query in MEDLINE.
Participants are provided with 30 topics on Diagnostic Test Accuracy (DTA)
reviews, which are the same topics as those provided in subtask 1. As in subtask
1 participants are asked to submit: (a) a ranked linked of the PubMed articles in
the given set, and (b) a threshold over this ranked list. Participant can submit
up to 3 submissions, and the format of each submission follows the format of
subtask 1 submissions. Further, given that subtask 2 was the main task of the
CLEF 2017 e-Health Technology Assisted Reviews in Empirical Medicine [
In what follows we describe the collection of articles used in the task, the topics released to participants, and how they were developed, as well as the relevance labels used in the evaluation. 3.1
The collection used in the lab is PubMed Baseline Repository last updated on November 28, 2017, and available on the NCBI FTP site under the ftp:// ftp.ncbi.nlm.nih.gov/pubmed/baseline directories. PubMed comprises more than 27 million citations for biomedical literature from MEDLINE, life science journals, and online books. Citations may include links to full-text content from PubMed Central and publisher web sites. NLM produces a baseline set of MEDLINE/PubMed citation records in XML format for download on an annual basis. The annual baseline is released in December of each year. The complete baseline consists of files pubmed18n0001 through pubmed18n0928. 3.2
To construct the benchmark collection, the organizers of the task considered 30 systematic reviews on Diagnostic Test Accuracy already conducted by Cochrane researchers. These reviews are publicly available and can be found in the Cochrane Library4; they can be identified by setting the topic filter in the library to "Diagnosis" and "Diagnostic Test Accuracy" and the stage filter to "Review".
At the time of the topic construction 88 such systematic reviews were
available; 50 of them were used in the 2017 task [
Participants were provided with two sets of topics: (a) a development set, and (b) a test set. The development set consisted of 42 topics out of the 50 topics provided in the 2017 version of the lab. 5 The 50 topics released in 2017 were re-examined by our Cochrane information specialist, and co-author of this paper, Rene Spijker, and 8 of them were found not reliable for training or testing purposes, and hence removed from the development set. In particular, the search strategies used within these reviews had a different objective than the objective of the lab. For this task we set-out to use searches that are sensitive in nature to inform a specific question for one review. Some of the reviews we removed were part of an overarching project where one search query was used to inform multiple reviews. We believe that including these would not reflect our intended practice and would misinform the algorithms and strategies developed. Other reviews had a different approach where a local registry was built on a broad topic (dementia) which would inform the review and the MEDLINE search was only intended as a highly specific top-up search, again not the intended approach for this task. So the reviews themselves were reliable but the methods used deviated from this task making them unsuitable. The IDs of the 8 topics are the following: 4 http://www.cochranelibrary.com/ 5 For subtask 1, two topics, CD011548 and CD011984, were not provided to participants, resulting in 40 training topics.
CD007431 (10), CD010772 (41), CD010775 (2), CD010896 (39), CD010771 (45), CD011145 (42), CD010783 (56), CD010860 (57), where in parenthesis is the filename of the topic in the 2017 release of the data.
Topic Description for Subtask 1: In subtask 1 each topic file was generated through the following procedure: First, the topic ID was extracted from the DOI of the systematic review. Then, the title of the systematic review was considered. Last, for each systematic review, the corresponding protocol was identified, and the objective of the review as described in the protocol was also considered. These three elements, topic ID, title and objective constitute the topic provided to participants. An example can be seen below: Title: Rapid diagnostic tests for diagnosing uncomplicated P. falciparum malaria in endemic countries Objectives: To assess the diagnostic accuracy of RDTs for detecting clinical P. falciparum malaria (symptoms suggestive of malaria plus P. falciparum parasitaemia detectable by microscopy) in persons living in malaria endemic areas who present to ambulatory healthcare facilities with symptoms of malaria, and to identify which types and brands of commercial test best detect clinical P. falciparum malaria.
Cochrane DTA review titles follow a particular structure [
The description of these relevant parts of the protocol as described in the
Cochrane Handbook for DTA reviews [
Types of studies: Identifiable design features of eligible studies must be stated. Review authors should describe the design as well as using a design name, as there is no universal terminology for diagnostic study designs. Key aspects include whether only prospective or both prospective and retrospective studies are to be included, to describe how and where participants were recruited (e.g. as a consecutive series of new presentations in primary care), and whether the study was cross-sectional or included longitudinal assessment for the reference standard. Authors should always state whether they included or excluded diagnostic case-control studies or the strategy used to make this decision. Any restrictions based on a minimal quality standard, minimal sample sizes, or numbers of diseased cases should be stated, but there is no clear guidance on how these limitations should be determined. In reviews that include comparisons between tests, alternative study designs which make within-study comparisons of tests may be sought, notably studies where all individuals receive all tests, and those where all individuals receive the reference standard but are randomized to receive different index tests. These latter studies should be described as randomized trials of test accuracy. Some reviews which compare tests may restrict study inclusion only to studies of these designs which make within-study comparisons, but others may include studies that evaluate one or other of the tests individually (particularly where few such published studies exist). Any such restrictions should be stated. Randomized trials of patient outcomes are rarely eligible for inclusion. They can only be included if individuals received both the index test and a reference standard – occasionally this information is available.
Participants: Review authors should specify the participants for whom the test would be applicable, including any restrictions on diagnoses, age groups and settings. Planned subgroup analyses related to participant characteristics should not be listed here – they should be listed under the sources of heterogeneity in the secondary objectives.
Index tests: Review authors should specify the test(s) to be evaluated in the review. If multiple tests are being reviewed and compared with each other details for each test should be given. In the first Cochrane DTA protocols and reviews tests were separated into new index tests or existing comparator tests. However it is often difficult to distinguish index from comparator tests and tests are no longer divided into these two categories. However, where it is clear that some tests are new experimental tests and others are existing standard comparative tests this should be noted.
Target conditions: The target condition is a particular disease or disease stage that the index test is intended to identify. Some reviews may evaluate the ability of tests to differentiate between several target conditions – if this is the case, the multiple target conditions should all be listed here.
Reference standards: Describe the clinical reference standards required to establish the presence or absence of the target condition in the tested population. If any reference standards are commonly used but considered inadequate this should be stated here as an exclusion criteria. If the review covers multiple target conditions, the reference standard for each should be stated.
Topic Description for Subtask 2: In subtask 2 each topic file was generated through the following procedure: For each systematic review, we reviewed the search strategy from the corresponding study in Cochrane Library. A search strategy, among other things, consists of the exact Boolean query developed and submitted to a medical database, at the time the review was conducted, and typically can be found in the Appendix of the study. Rene Spijker, a co-author of this work and a Cochrane information specialist examined the grammatical correctness of the search query and specified the date range which dictated the valid dates for the articles to be included in this systematic review. The date range was necessary because a study published after the systematic review should not be included even though it might be relevant, since that would require manually examining its content to quantify its relevance.
A number of medical databases, and search interfaces to these databases is available for search, and for each one information specialists construct a different variation of their query that better fits the data and meta-data of the database. For this task, we only considered the Boolean query constructed for the MEDLINE database, using the Wolters Kluwer Ovid interface. Then we submitted the constructed Boolean query to the OVID system at http: //demo.ovid.com/demo/ovidsptools/launcher.htm and collected all the returned PubMed document identification numbers (PMID’s) which satisfied the date range constraint. This step was automated by a Python script we put together and through an interface available to the University of Amsterdam.
The topic file is in a text format and contains four sections, Topic, Title, Query, and PMID’s. PMID’s are the PubMed document IDs returned by the Boolean query. The PMIDs can be used to access the corresponding document through the National Center for Biotechnology Information (NCBI)6. An example of a topic file can be viewed below. 6 https://www.ncbi.nlm.nih.gov/books/NBK25497/ Pids: 19164769 9557953 7688346 18509532 ... 3.3
The original systematic reviews written by Cochrane experts included a reference section that listed Included, Excluded, and Additional references to medical studies. Included are the studies that are relevant to the systematic review. Excluded are the studies that in the abstract and title screening stage were considered relevant, but at the article screening phase were considered irrelevant to the study and hence excluded from it. Additional are the studies that do not impact the outcome of the review, and hence irrelevant to it. The union of Included and Excluded references are the studies that were screened at a Title and Abstract level and were considered for further examination at a full content level. These constituted the relevant documents at the abstract level, while the Included references constituted the relevant documents at the full content level.
The majority of the references included their corresponding PMID, but not all of them. For those references missing the PMID, the title was extracted from the reference, and it was used as a query to Google Search Engine over the domain https://www.ncbi.nlm.nih.gov/pubmed/. The top-scored document returned by Google was selected, and the title of the study contained in landing page, as identified in the metadata extracted. The title was compared then with the title of the study used as search query. If the Edit Distance between the two titles was up to 3 (just to account for spaces, parentheses, etc.) then the study reference was replaced by the PMID also extracted from the metadata of the landing page. If (a) the title had an edit distance greater than 3 but less than 20, or (b) the study was an included study, or (c) no title was contained in the Google result metadata, or (d) no Google results were returned, then the query was submitted at https://www.ncbi.nlm.nih.gov/pubmed/ and the results were manually examined. All other studies were discarded under the assumption that they are not contained in PubMed. The format of the qrels followed the standard TREC format:
Relevance where Topic is the topic ID of the systematic review, Iteration in our case is a dummy field always zero and not used, Document is the PMID, and Relevancy is a binary code of 0 for not relevant and 1 for relevant studies. The order of documents in the qrel files is not indicative of relevance. Studies that were returned by the Boolean query but were not relevant based on the above process, were considered irrelevant. Those are studies that were excluded at the abstract and title screening phase. All other documents in MEDLINE were also assumed to be irrelevant, given that they were not judged by the human assessor.
Note that, as mentioned earlier, the references of a systematic review were produced after a number of Boolean queries were submitted to a number of medical databases, and their titles and abstracts were screened. The PMID’s provided however were only those that came out of the MEDLINE query. Therefore, there was a number of abstract-level relevant studies (the gray area in the Venn diagram below) that were not part of the result set of the Boolean query provided to the participants. Studies that were cited in the systematic review but did not appear in the results of the Boolean query were excluded from the label set for Subtask 2, but included for Subtask 1. Hence, the total number of relevant abstracts in the test set for Subtask 1 is 4,656, while in Subtask 2 it is 3,964; further the total number of relevant studies in Subtask 1 is 759, while for Subtask 2 it is 678.
CD010438 CD007427 CD009593 CD011549 CD011134 CD008686 CD011975 CD009323 CD009020 CD011548 CD011984 CD010409 CD008054 CD009591 CD008691 CD010632 CD007394 CD008643 CD009944 CD008803 CD008782 CD009647 CD009135 CD008760 CD009519 CD009372 CD010276 CD009551 CD012019 CD008081 CD009185 CD010339 CD010653 CD010542 CD010023 CD010705 CD010633 CD010173 CD009786 CD010386 CD009579 CD009925 CD008122 CD012599 CD009175 CD009694 CD009263 CD010502 CD010680 CD010864 CD011431 CD011602 CD011420 CD011686 CD012179 CD012281 CD011053 CD011515 CD008587 CD011926 CD012165 CD012083 CD008892 CD011126 CD010657 CD008759 CD010296 CD010213 CD012009 CD011912 CD012010 CD012216
Table 1 and Table 2 show the distribution of the relevant documents at abstract or document level for all the topics in the development set and the test set. The total number of unique PMID’s released for the training set was 241,669 (an average of 5754 per topic) and for the test set 218,496 (an average of 7283 per topic). The average percentage of relevant documents at Abstract level in the training set is 3.8% of the total number of PMID’s released, and in the test set 4.7%, while at the content level the average percentage is 1.5% in the training set, and 1% in the test set. In [17], a test collection was developed based on a random selection of 93 Cochrane systematic reviews (not just DTAs), and reported a slightly higher rate of relevance ( 111549 = 1:2%). However, compared with the TREC campaign, the rate of relevant documents is 5.45% and 2.78% for the Adhoc track of TREC 8, and the Web track of TREC 2002, respectively. Overall, the number of relevant documents is not very high in this lab, making locating them quite a difficult task. 4
Evaluation within the context of using technology to assist in the reviewing process is very much dependent on how the users interact with the system, and on the goal of the technology assistance. For example, if the goal of the assistance is to autonomously predict which studies should be assessed by the end-user at a document level, then the problem can be viewed as a classification problem; the system screens all abstracts and returns a subset of them as relevant. If the goal of the assistance is to identify all the relevant documents as quick as possible but let the human decide when to stop screening, then the problem can be viewed as a ranking problem. There are, of course, many other possible variations. For the purposes of the 2018 lab, we consider the problem as a ranking problem - that is, to rank the set of documents associated with the topic in decreasing order of relevance.
Furthermore, the two subtasks although very similar in terms of evaluation, i.e. in both subtasks participants’ runs are rankings of article, with a designated threshold, they also differ: in subtask 2 the set of articles to be prioritized contains all the relevant articles, while in subtask 1 the relevant articles need to be found within the entire PubMed database, and hence there is no guarantee that all relevant articles will appear in the top 5000. Further, in subtask 1, the length of the ranked lists vary significantly across different topics.
For the evaluation of runs employ a number of standard IR measures, along
with measures that have been developed for the particular task of technology
assisted reviews [
The lab organizers developed an evaluation software similar to trec_eval for the easy evaluation of the submitted runs, also provided to participants. The code of the tar_eval software is available at https://github.com/CLEF-TAR/tar. 5
The 2018 task received submissions from 7 teams, including one team from Canada (UWA), one team from the USA (UIC/OHSU), one team from the UK (Sheffield), one team from China (ECNU), one team from Greece (AUTH), one team from Italy (UNIPD), one team from France (Limsi-CNRS). The participating teams are:
For the subtask 1, we received 12 runs from 4 teams. For the subtask 2, we received 19 runs from 7 teams.
The 7 teams used a variety of learning methods including batch supervised learning, continuous active learning, a variety of learning algorithms including logistic regression, support vector machines, and neural networks, as well as unsupervised retrieval methods, such as TT-IDF, BM25, with or without traditional relevance feedback methods, such as the Rocchio’s Algorithm, and a variety of text representation methods including simple count-based methods and neural embeddings.
Tables 3 and 4 categorize the participating runs in the two subtasks along five dimensions: (a) automatic vs manual runs; (b) use of the development set; (c) use of supervised and semi-supervised learning algorithms, and (d) use of Run auth_run1 auth_run2 auth_run3 ECNU_RUN1 ECNU_RUN2 ECNU_RUN3 shef-bm25 shef-tfidf shef-bool UWA UWG UWX
Automatic Development Supervision
X X X X X X X X X X x x X X X X X X X x x X x x X x x X x X x x X x x X relevance feedback, and (e) the type of thresholding used. The categorization has been performed by the lab coordinators – not by the participants – based on the submitted participants description of their algorithms. Hence, there is always a chance of mis-classifying some run. In subtask 1 participants employed both supervised and unsupervised methods for ranking articles. A total of 5 runs were trained over the provided development set, and their generalization was tested against the test topics, while 7 made no explicit use of it; it may be the case that participants tried different models and algorithms over the development set, and selected to submit the best performing ones, hence there may be a flavor of model selection, however we did not consider this as use of the development set. Participants represented the textual data in a variety of ways, including document-topic features, bag-of-words, topic model distributions, embeddings, metadata. Out of the 19 runs submitted for subtask 2, 6 trained over the development set, 12 used the relevance feedback provided per topic, either at an abstract or content level, while 6 runs used a fixed threshold, 2 an automatic thresholding method, and the rest did not threshold the ranking at all. Below we provide a short description of the submitted runs for both subtask 1 and 2.
AUTH took a learning-to-rank approach, using both batch and active
learning. Their model consists of two parts: an inter-topic model which utilizes
XGBoost and is trained over the entire development corpus (for subtask 1 it is 2500
articles returned by PubMed search, and for subtask 2 the articles provided
by the organizers) and an intra-topic model, an iteratively-built SVM, trained
over relevance feedback provided partially in the test topics. For the inter-topic
model a total of 48 for subtask 1 and 31 for subtask 2 topic-document (or solely
topic) features were computed over the title and the abstract of the articles and
the query. For the intra-topic model a TF-IDF vectorization of the articles was
used [
CNRS trained a logistic regression model on a large number (> 500,000)
of features over the development set. The logistic regression model is intended
to capture features that are related to DTA studies independent of the topic.
They further used an active learning approach which continuously learn to find
relevant articles within each topic. A model that combines the two using a
feedforward neural network was also used [
ECNU used the BM25 algorithm for subtask 1 to acquire a baseline. Furthermore, query expansion based on MeSH terms and pseudo relevance feedback (PRF) was used to improve the results. In sub-task 2, they employed Paragraph2Vector to represent query and documents for similarity calculation [18].
UIC/OHSU first applied a clustering algorithm over a large number of
PubMed articles to identify 6 publication types, including DTA studies, but
also Randomized Controlled Trials, Cross-sectional Studies, Cross-over Studies,
Cohort Studies, and Case-Control Studies. The clusters were then represented
by a feature vector of the centroid with each article in the cluster represented
by 300 weighted terms most associated to the words in the article. Then, each
article in the provided dataset was compared to the 6 clusters and a number of
similarity measures were computed. These were used as features to be used by
an SVM to classify articles against the 6 clusters. [
UNIPD used a two-dimensional probabilistic version of BM25 to rank articles, using relevance feedback up to a certain number of articles shown to the user, and switched to a Naive Bayes classifier for the remaining of the articles until a fixed threshold point [14].
Sheffield used RAKE [16] keyword extraction algorithm for subtask 1 to
interpret protocols, extract keywords and form them into queries designed to
retrieve relevant documents, while Apache Lucene was used as the IR engine.
Their approach to subtask 2 was to enrich queries with terms designed to identify
diagnostic test accuracy studies and also by making use of relevance feedback [
UWA applied the Baseline Model Implementation (BMI) from the TREC
Total Recall Track (2015-2016) and the CLEF 2017 eHealth. They further
applied their "knee-method" stopping criterion to BMI to determine how many
abstracts should be examined for each topic. The difference between different
submissions came from the selection of feedback to be used to retrain the model
with the options being abstract-level, content-level, or manual feedback provided
by the participants themselves [
In this section we provide the results of the evaluation for both subtasks.
Fig. 1. Average precision using the document level relevance judgments. d o e c h
t s i n e i c d n e a d v u e l l e c r In ; 1 re k a tsa ta b th u S se
i in d s tu n s ru 95 e l h a t t r to o f n s I e . r t o n c a s v
e e l g e a r r . id 5 s
n d 7 P e o l c n b a re uR T a k R 0 0 0 5 R 0 0 0 1 R 0 0 5 R 0 0 1 R 0 5 A P a R
L
F T
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 5 5 5 0 0 0 5 6 6 5 2 1
5 5 5 3 3 3 6 9 7 2 8 5 6 6 9 1 2 1 1 0 8 7 7 8 2 6 9 5 6 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 2 3
N N N 1 2 3 U U U n n n R R R 52 f l u d o r ru ru _ _ _ m fi o _ _ _ U U U b t b h h h N N N f, a l v ev l e e s a i h
t e c s n ie a v ud le t e s r ;1 566 k 4 s a o n t c ru to s
w ed In ie e e th ed ts o d 0 f i
s 3 s reo con teh e a
g g s ra ie in e
d en v u e A ts rc . s 6 ed t e d c a k R 0 0 0 5 R 0 0 0 2 R 0 0 0 1 R 0 0 5 R 0 0 4 R 0 0 2 R 0 0 1 R 0 5
R lb lcu trs un a x b R T E a 8 8 6 5 6 4 5 2 6 9 9 9 1 6 8 6 2 7 4 1 1 0 0 0 .6 .
5 .
4 .
2 .
2 .
2 .
3 .
2 .
2 .
9 .
9 .9 .6 .
6 . 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 .3 .
3 . 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 v P
A M
F l l
T
Figure 2 shows the recall-effort curves for the participats’ runs, that is the recall value at different percentage of documents shown to the user. The green curve with the square marker corresponds to the Oracle run, which achieves an optimal recall at the different effort levels.
Figure 3 presents the recall obtained by the participants’ runs at the point of the threshold as a function of the number of documents presented to the user. As expected the more documents presented to the user (the lower the threshold) the higher the achieved recall. Nevertheless, there are still algorithms that dominate others. The figure present the Pareto frontier.
Figure 4 demonstrates the bar plot of average precision values per topic; the dashed blue line in the box plots designates the average Average Precision (AAP) for each topic, a measure that can be seen as a proxy for topic difficulty.
Fig. 4. Average Average Precision at document level relevance labels. R y t i l i b a i l e R 0 0 1 S S 5 9 W W R R R P A M n I l ten con
s m u e c i
w o e c h s i n e i c d e l l e c r In ; 2 re r t o n c a s v
e e l g e a r r e o l c b a re T a
S a m S
e n e 0 a d 2 v u b th u S se . id n 7 s u n R 5 7 7 5 5 5 4 5 4 0 7 1 5 5 5 2 8 5 8 4 7 7 4 4 4 6 6 6 7 8 6 4 4 4 4 7 3 3 2 8 8 2 2 2 4 4 4 8 5 1 2 2 2 1 7 5 7 7 7 7 7
1 2 7 7 7 2 1 1 2 0 1 0 0 0 0 7 2 9 2 2 4 0 0 0 4 6 3 9 0 8 8 0 0 0 8 3 7 2 6 7 0 0 0 0 3 6 9 .0 .
9 .
9 .
0 .
0 .
0 .
5 .
4 .
5 .
9 .
9 .
9 .
0 .
0 .
0 .
7 .
6 .
9 .9 1 0 0 1 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 1 5 6 0 0 0 4 5 6 3 0 1 0 0 0 5 7 8 9 1 4 4 2 2 2 4 1 4 6 6 2 2 2 2 3 5 5 2 .7 .
2 .
2 .
7 .
7 .
7 .
4 .
5 .
4 .
2 .
3 .
4 .
7 .
7 .
7 .
2 .
2 . 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
3 1 1 6 2 6 1 1 1 3 3 4 4 4
WW _ ee l a r y e e
l7 l8 d en re e g f f
d d u o o q 1 S S W R % 0 1 A M
T u lIcn teh r n t e s t b o a p s a i h e c s n ie a v ud le t e s r k 3 e a
g g s ra ie in
d en e v u e A ts rc e d c l lu r b c ts a x b T E a . s n 8 ed t u a R 3 1 1 3 3 3 5 6 5 4 1 8 3 3 3 3 1 6 4 8 8 8 8 8 8 6 6 6 7 0 8 8 8 8 5 2 2 6 2 8 8 2 2 2 4 4 4 8 6 1 2 2 2 7 1 9 7 7 7 7 7
1 2 7 7 7 1 2 2 1 0 4 3 0 0 0 0 1 4 6 0 5 0 0 0 3 6 0 7 0 4 4 0 0 0 2 7 3 5 2 4 0 0 0 1 7 9 2 .0 .
9 .
9 .
0 .
0 .
0 .
5 .
3 .
5 .
8 .
9 .
9 .
0 .
0 .
0 .
5 .
5 .
9 .9 1 0 0 1 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 4 1 2 8 8 8 7 2 9 9 1 0 8 8 8 6 6 9 7 9 7 7 9 9 9 2 0 0 0 4 7 9 9 9 2 7 8 8 .3 .
1 .
1 .
3 .
3 .
3 .
4 .
5 .
4 .
2 .
2 .
2 .
3 .
3 .
3 .
3 .
2 . 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 9 5 7 9 7 0 1 7 3 8 4 7 1 8 4 9 9 8 7 8 8 6 2 1 0 7 6 8 4 0 7 9 8 9 .8 .
8 .
8 .
7 .
7 .
7 .
4 .
4 .
5 .
8 .
7 .
6 .
6 .
4 .
WW l a r y e e
l8 l7 _ ee d en re e
d d u o o q m m % a o 5 5 5 9 0 8 2 5 7 3 2 1 2 0 2 0 5 6 9 5 1 1 1 1 0 1 0 5 3 1 1 1 7 7 4 0 0 1 2
Figure 6 shows the recall-effort curves for the participats’ runs, that is the recall value at different percentage of abstracts shown to the user.
Figure 7 presents the recall obtained by the participants’ runs at the point of the threshold as a function of the number of abstracts presented to the user. As expected the more abstract presented to the user (the lower the threshold) the higher the achieved recall. Nevertheless, there are still algorithms that dominate others. The figure present the Pareto frontier.
Figure 8 demonstrates the bar plot of average precision values per topic; the dashed blue line in the box plots designates the average Average Precision (AAP) for each topic, a measure that can be seen as a proxy for topic difficulty. 7
The CLEF 2018 e-Health Lab Task 2 constructed a benchmark collection of 30 Diagnostic Test Accuracy systematic reviews to study the effectiveness and effi
Fig. 8. Average Average Precision at abstract level relevance labels. ciency of information retrieval and machine learning algorithms both in finding relevant articles in a large medical databse without explicitely constructing a Boolean query and in prioritizing the studies to be screened at the abstract and title screening stage, and providing a stopping criterion over the ranked list. The results demonstrate that automatic methods can be trusted for finding most, if not all, relevant studies in a fraction of the time manual screening can do the same. Further, many of the algorithms retrieved articles that were not in the results of the Boolean query, hence raising even concerns for the validity of the current practice in conducting systematic reviews. Given that across different runs many parameters change simultaneously it is not easy to come to certain conclusions about the relative performance of automatic methods.
Regarding the benchmark collection itself, there is a number of limitations to be considered: (a) Pivoting on the results of the the OVID MEDLINE Boolean query limits our ability to identify all relevant studies, i.e. relevant studies that are outputted by Boolean queries over different databases, and relevant studies that are actually not found by these Boolean queries. The former can be overcome by considering all the different queries submitted; for the latter extra manual judgments would be required. (b) Pivoting on abstract and title only we miss the opportunity to study the effect of automatic methods when applied to the full text of the studies, that would present an opportunity to completely overcome the multi-stage process of systematic reviews. However, most of the full text articles are protected under copyright laws that do not give all participants access to those. (c) The evaluation setup of ranking does not allows us to consider the cost of the process, since given a ranking a researcher would have to still go over all studies ranked. A more realistic setup, e.g. a double-screening setup, could be considered. (d) In the construction of relevant judgments we considered the included and excluded references of the systematic reviews under study, which prevented us to study the noise and disagreement between reviewers. (e) In our effort to allow iterative algorithms, e.g. active learning algorithms, to be submitted, we handed the test sets’ relevant judgments directly to the participants, which is rather unusual for this type of evaluation exercises. An alternative would be the setup used by the TREC Total Recall, where participants submitted their running algorithms to the organizers. (f) When it comes to evaluation measures there is a large variety of those, all of which take a different often useful view point on the effectiveness of algorithm, but which makes it difficult to decide upon a single golden measure to rank participants’ runs. 14. Nunzio, G.M.D., Ciuffreda, G., Vezzani, F.: Interactive sampling for systematic reviews. ims unipd at clef 2018 ehealth task 2. In: Working Notes of CLEF 2018 Conference and Labs of the Evaluation forum, Avignon, France, September 10-14, 2018. CEUR Workshop Proceedings, CEUR-WS.org (2018) 15. O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., Ananiadou, S.: Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic Reviews 4(1), 5 (2015) 16. Rose, S., Engel, D., Cramer, N., Cowley, W.: Automatic keyword extraction from individual documents. Text Mining: Applications and Theory pp. 1–20 (2010) 17. Scells, H., Zuccon, G., Koopman, B., Deacon, A., Geva, S., Azzopardi, L.: A test collection for evaluating retrieval of studies for inclusion in systematic reviews. In: To appear in Proceedings of the 40th international ACM SIGIR conference on Research and development in Information Retrieval. ACM (2017) 18. Wu, H., Wang, T., Chen, J., Chen, S., Hu, Q., He, L.: Ecnu at 2018 ehealth task 2: Technologically assisted reviews in empirical medicine. In: Working Notes of CLEF 2018 - Conference and Labs of the Evaluation forum, Avignon, France, September 10-14, 2018. CEUR Workshop Proceedings, CEUR-WS.org (2018) CD008122 CD012599 CD009175 CD009694 CD009263 CD010502 CD010680 CD010864 CD011431 CD011602 CD011420 CD011686 CD012179 CD012281 CD011053 Rapid diagnostic tests for diagnosing uncomplicated P. falciparum malaria in endemic countries First and second trimester serum tests with and without first trimester ultrasound tests for Down’s syndrome screening Clinical symptoms and signs for the diagnosis of Mycoplasma pneumoniae in children and adolescents with community-acquired pneumonia Computed tomography (CT) angiography for confirmation of the clinical diagnosis of brain death 123I-MIBG scintigraphy and 18F-FDG-PET imaging for diagnosing neuroblastoma Rapid antigen detection test for group A streptococcus in children with pharyngitis Ankle brachial index for the diagnosis of lower limb peripheral arterial disease D-dimer test for excluding the diagnosis of pulmonary embolism Rapid diagnostic tests for diagnosing uncomplicated nonfalciparum or Plasmodium vivax malaria in endemic countries Ultrasonography for diagnosis of alcoholic cirrhosis in people with alcoholic liver disease Lateral flow urine lipoarabinomannan assay for detecting active tuberculosis in HIV-positive adults Triage tools for detecting cervical spine injury in pediatric trauma patients Blood biomarkers for the non-invasive diagnosis of endometriosis Combination of the non-invasive tests for the diagnosis of endometriosis Imaging for the exclusion of pulmonary embolism in pregnancy 2011/08/25 2012/06/26 2012/08/31 2012/09/21 2013/02/01 2013/02/01 2013/12/12 2013/12/31 2015/01/31 2015/02/28 2015/02/28 2015/05/01 2015/05/31 2015/07/28 CD011515 CD008587 CD011926 CD012165 CD012083 CD008892 CD011126 CD010657 CD008759 CD010296 CD010213 CD012009 Diagnostic accuracy of different imaging modalities following computed tomography (CT) scanning for assessing the resectability with curative intent in pancreatic and periampullary cancer Cytology versus HPV testing for cervical cancer screening in the general population Molecular assays for the diagnosis of sepsis in neonates Endometrial biomarkers for the non-invasive diagnosis of endometriosis Ultrasonography for confirmation of gastric tube placement Rapid diagnostic tests for typhoid and paratyphoid (enteric) fever Three-dimensional saline infusion sonography compared to two-dimensional saline infusion sonography for the diagnosis of focal intracavitary lesions Dimercaptosuccinic acid scan or ultrasound in screening for vesicoureteral reflux among children with urinary tract infections Platelet count, spleen length, and platelet count-tospleen length ratio for the diagnosis of oesophageal varices in people with chronic liver disease or portal vein thrombosis Ultrasonography for endoleak detection after endoluminal abdominal aortic aneurysm repair Imaging modalities for characterising focal pancreatic lesions Amylase in drain fluid for the diagnosis of pancreatic leak in post-pancreatic resection Pulse oximetry screening for critical congenital heart defects Serum amylase and lipase and urinary trypsinogen and amylase for diagnosis of acute pancreatitis 18F PET with florbetapir for the early diagnosis of AlzheimerâĂŹs disease dementia and other dementias in people with mild cognitive impairment (MCI) 2016/01/19 2016/02/16 2016/02/28 2016/03/01 2016/03/30 2016/03/31 2016/06/30 2016/07/01 2016/07/19 2017/02/28 2017/03/30 2017/03/30 2017/05/01