Evidence-based medicine has become an important pillar in health care and policy making. In order to practice evidence-based medicine, it is important to have a clear overview over the current scientific consensus. These overviews are provided in systematic review articles, that summarize all available evidence regarding a certain topic (e.g., a treatment or diagnostic test). In order to write a systematic review, researchers have to conduct a search that will retrieve all the studies that are relevant. The large and growing number of published studies, and their increasing rate of publication, makes the task of identifying relevant studies in an unbiased way both complex and time consuming to the extent that jeopardizes the validity of their findings and the ability to inform policy and practice in a timely manner. Hence, the need for automation in this process becomes of utmost importance. Finding all relevant studies in a corpus is a difficult task, known in the Information Retrieval (IR) domain as the total recall problem.
To this date, retrieval of evidence to inform systematic reviews is being conducted in multiple stages:
1. Boolean Search: At the first stage information specialists build a broad
Boolean query expressing what constitutes relevant information. The query is then submitted to a medical database containing titles, abstracts, and indexing terms of a controlled vocabulary of medical studies. The result is a set, A, of potentially interesting studies. 2. Title and Abstract Screening: At a second stage experts are screening the titles and abstracts of the returned set and decide which one of those hold potential value for their systematic review, a set D. If screening an abstract has a cost C a , screening all |A| abstracts has a cost of C a * |A|. 3. Study Screening: At a third stage experts are downloading the full text of the potentially relevant abstracts, D, identified in the previous phase and examine the content to decide whether indeed these studies are relevant or not. Examining a document has typically a larger cost of C d > C a . The result of the second screening is a set of references to be included in the systematic review.
Unfortunately, the precision of the Boolean searches is typically low, hence reviewers often need to look manually through many thousands of irrelevant titles and abstracts in order to identify a small number of relevant ones. Furthermore, the recall of the searches is often assumed to be 100%, which may not be the case.
To overcome some of the limitations of the Boolean search, researchers have been testing the effectiveness of machine learning and information retrieval methods. O'Mara-Eves et al. [15] provide a systematic review of the use of text mining techniques for study identification in systematic reviews.
The goal of this lab is to bring together academic, commercial, and government researchers that will conduct experiments and share results on automatic methods to retrieve relevant studies with high precision and high recall, and release a reusable test collection that can be used as a reference for comparing different retrieval and mining approaches in the field of medical systematic reviews.
To construct the benchmark collection, the organizers of the task considered 58 systematic reviews on Diagnostic Test Accuracy conducted by the Cochrane researchers. These reviews are publicly available through the Cochrane Library 4and can be identified by setting the topic filter in the library to "Diagnostic" and "Diagnostic Test Accuracy" and the stage fitler to "Review". At the date of the publication of this article 79 such studies are available, however the last 22 were performed after the organizers put the collection together. The 58 systematic reviews considered can be found in the Appendix of this articles at Table 6.
Participants were provided with two data sets: (a) a development set, and (b) a test set. The development set consists of 20 topics for Diagnostic Test Accuracy (DTA) systematic reviews, while the test set consists of 30 topics. For both sets, one topic file and two files of relevance judgments at abstract and document level respectively are constructed (qrel's).
The topic file is generated through the following procedure. For each systematic review, we reviewed the search strategy from the corresponding study in Cochrane Library. A search strategy, among others, consists of the exact Boolean query developed and submitted to a medical database, at the time the review was conducted, and typically can be found in the Appendix of the study. Rene Spijker, a co-author of this work and a Cochrane information specialist examined the grammatical correctness of the search query and specified the date range which dictated the valid dates for the articles to be included in this systematic review. The date range was necessary because a study published after the systematic review should not be included even though it might be relevant, since that would require manually examining its content to quantify its relevance. Important note: A number of medical databases, and search interfaces to these databases is available for search, and for each one information specialists construct a different variation of their query that better fits the data and meta-data of the database. For this task, we only considered the Boolean query constructed for the MEDLINE database, using the Wolters Kluwer Ovid interface. Then we submitted the constructed Boolean query to the OVID system 5 and collected all the returned PubMed document identification numbers (PMID's) which satisfied the date range constraint. This step was automated by a Python script we put together and through an interface available to the University of Amsterdam6 . Out of the 58 reviews 8 were discarded since the provided Boolean query was not in the right format, which made it difficult if not impossible to reconstruct the set of PMID's, hence the 50 topics in the development and test set.
The topic file is in a text format and contains four sections, Topic, Title, Query, and PMID's, where Topic is the topic ID, a substring of DOI of the document (e.g. CD010438 for 10.1002/14651858.CD010438.pub2), and PMID's are the document IDs returned by the Boolean query. The PIDs can be used to access the corresponding document through the National Center for Biotechnology Information (NCBI) 7 . An example of a topic file can be viewed below. For the construction of the qrel files, we considered the reference section of the 50 systematic reviews. The references are split into three categories: Included, Exclude, and Additional. Included are the studies that are relevant to the systematic review. Excluded are the studies that in the abstract and title screening stage were considered relevant, but at the article screening phase were considered irrelevant to the study and hence excluded from it. Additional are additional references that do not impact the outcome of the study, and hence irrelevant to it. The included references were the relevant studies at the document-level qrels, while both the included and excluded references were considered relevant at the abstract-level qrels. The format of the qrels followed the standard TREC format:
where Topic is the topic ID of the systematic review, Iteration in our case is a dummy field always zero and not used, Document is the PMID, and Relevancy is a binary code of 0 for not relevant and 1 for relevant studies. The order of documents in the qrel files is not indicative of relevance. Studies that were returned by the Boolean query but were not relevant based on the above process, were considered irrelevant. Those are studies that were excluded at the abstract and title screening phase. All other documents in MEDLINE were also assumed to be irrelevant, given that they were not judged by the human assessor. Important Note: Note that, as mentioned earlier, the references of a systematic review were produced after a number of Boolean queries were submitted to a number of medical databases, and their titles and abstracts were screened. The PMID's provided however were only those that came out of the MEDLINE query. Therefore, there was a number of abstract-level relevant studies (the gray area in the Venn diagram below) that were not part of the result set of the Boolean query provided to the participants. For the development set, the qrel file contained those additional PMIDs, for those participants that would decide to search the entire MEDLINE database, and not only consider the studies provided to them in the Topic files. To the best of our knowledge, no one submitted such a system, hence to avoid any bias we excluded those relevant studies from the test set.
Table 1 shows the distribution of the relevant documents at abstract or document level for all the topics in the development set and the test set. The total number of unique PMID is 149,405 for the development set and 117,562 for the test set. Their percentages of relevant documents at abstract level are quite close, which is 1.88% for the development set and 1.58% for the test set. This is not true at document level, however, where the relevant documents in the test set is almost twice as large as in the development set, even though there are 0.52% and 0.33% of relevant studies, respectively. In [17], a test collection was developed based on a random selection of 93 Cochrane systematic reviews (not just DTAs), and reported a slightly higher rate of relevance ( 14 1159 = 1.2%). However, compared with the TREC campaign, the rate of relevant documents is 5.45%, 2.78% for the Adhoc track of TREC 8 and the Web track of TREC 2002. Overall, the number of relevant documents is not very high in this lab, making locating them quite a difficult task.
Important Note: As one can observe in Table 1, there are topics for which the output of the Boolean query is rather narrow, with as few as 64 studies to be reviewed for topic CD008760. Cochrane is conducting systematic reviews on a regular basis, in an attempt to update each review every two-three years. Some of the reviews considered for the construction of the benchmark collection, such as the CD008760 review, are updates to previous reviews. These updates, only specify a query for a time range that starts after the last review on the topic was conducted. Hence, the 64 studies, are the output of the Boolean query for this short time range, hence its small number. If the Boolean query were to run against the entire MEDLINE database, the number of studies would be in the range of tens of thousands, as is the case for some other reviews considered, e.g. CD008782.
The CLEF 2017 e-Health Lab [8], task 2, focused on retrieving studies for conducting Diagnostic Test Accuracy (DTA) systematic reviews. Retrieval in this area is generally considered very difficult, where sensitive searches result in large quantities of references to be screened manually, and a breakthrough in this field would likely be applicable to other areas as well. The task has a focus on the second stage of the process, i.e. given the results of a Boolean search how to make abstract and title screening more effective and efficient. Currently a typical number needed to read (NNR), the number of studies to screen to identify 1 eligible study, for DTA systematic reviews is approximately 80 when applied to potential abstracts that need further full text assessment. With an average of 7000 results to be screened, which would take approximately 120 hours to screen (1 minute per abstract [18]), a huge benefit can be made in reducing the workload in this process.
Given the results of the Boolean search from stage 1 as the starting point, participants were asked to rank the set of the provided abstracts. The task had two goals: (i) to produce an efficient ordering of the documents, such that all the relevant abstracts are retrieved above the irrelevant ones, and (ii) to identify the relevant subset of abstracts to be shown to a user, that is a stopping point in the ranked list of abstract, where a researcher could confidently stop screening abstracts and titles. Therefore, we solicited two types of submissions: (i) ranking submission: automatic or manual methods that rank all abstracts, with the goal of retrieving relevant abstracts as early in the ranking as possible, and (ii) thresholding submission: thresholding can be performed in a batch, or iterative manner as well.
We also considered two evaluation frameworks, (a) a simple evaluation, and (b) a cost-effective evaluation. The assumption behind the simple evaluation framework is the following: The user of your system is the researcher that performs the abstract and title screening of the retrieved articles. Every time an abstract is returned (i.e. ranked) there is an incurred cost/effort of CA, while the abstract is either irrelevant (in which case no further action will be taken) or relevant (and hence passed to the next stage of document screening) to the topic under review. The assumption behind the cost-effective evaluation is the following: The user that performs the screening is not the end-user. The user can interchangeably perform abstract and title screening, or document screening, and decide what PMIDs to pass to the end-user. Every time an abstract is returned the user can either (a) read the abstract (with an incurred cost of CA) and decide whether to pass this PMID to the end-user, or (b) read the full document (with an incurred cost of CA+CD) and decide whether to pass this PMID to the end-user, or (c) directly pass the PMID to the end user (with an incurred cost of 0), or (d) directly discard the PMID and not pass it to the end user (with an incurred cost of 0). For every PMID passed to the end-user there is a cost of attached to it: CA if the abstract passed on is not relevant, and CA + CD if the abstract passed on is relevant (that is, we assume that the end-user completes a two-round abstract and document screening, as usual, but only for the PMIDs the algorithm+feedback user decided to be relevant). Although a small number of teams participated in the cost-effective sub-task, the lab focused on the simple evaluation sub-task, and this is what is described in the remaining of this report.
Evaluation within the context of using technology to assist in the reviewing process is very much dependent on how the user(s) interact with the system -and the goal of the technology assistance. For example, is the goal of the assistance to automate the screening process -where the system assess all the abstracts and returns a subset of the initial set to be screened by the end-user (i.e. screened in batch mode). Or, it could be used to identify all the relevant documents as soon as possible, in an iterative manner -where the system asks for feedback from the end-user to help improve the ranking. Of course, then the an open problem is decide when to stop requesting feedback, and when to stop assessing abstracts. In which case a subset of abstracts is identified, which consist of abstracts have been screened during the feedback cycles and the remainder that are screened but are not used for feedback (i.e. in batch mode). There are, of course, many other possible variations. For the purposes of this initial track/task, we consider the problem as a ranking task -that is to rank the set of documents associated with the topic in decreasing order of relevance. We consider a document relevant if the abstract passed the abstract screening phase (regardless of whether it was included or excluded from the study).
For this task we employ a number of standard measures, typically used in IR ranking evaluations, along with other measures from related tracks and some new measures we have developed.
• Average Precision (AP)
• Normalized cumulative gain @ 0% to 100% of documents shown; for the simple case that judgments are binary, normalized cumulative gain @ % is simply Recall @ % of shown documents To calculate the cost based measured, we considered three possible interactions to support a range of different ways to screen the items and to utilize feedback when ranking. We consider the follow possibilities:
1. suppose we have an ranking algorithm, which uses no feedback from the user, simply ranks the list of abstracts. The list is then presented to the end-user, who evaluates them in a batch. In this case, no feedback is requested, and abstracted are marked, NF. 2. suppose we have a ranking algorithm which uses feedback (i.e. abstract(s) are presented to the user, feedback on their relevance is obtained, which is then used by the algorithm, thus simulating online feedback from the user). In this case, for each document where feedback from the users is requested, abstracts are marked AF, but if no feedback is requested it is marked NF. Abstracts marked NF, are then presented to the end-user to evaluate in a final batch. 3. for either above option, the algorithm may decided that an abstract is not relevant, and thus it does not need to be shown to a user, and so are marked NS.
To calculate the total cost (TC), we calculated:
where C a is the cost of assessing the abstract, C f is the cost of asking for feedback #N F is the number of NF items, #AF is the number of AF items. We also created two additional cost measures which included a penalty for missing relevant abstracts (a) with a uniform penalty and (b) a weighted penalty. The uniform penalty was calculated as follows:
where C p is the cost of the penalty of missing a relevant abstract, N is the total number of documents in the set for the topic. The assumption behind this penalty is that the end-user would need to continue examining abstracts before they would from the remaining (R − r) relevant items, and encounters them at a uniform rate in the remaining N − n abstracts which were not shown. So if half the relevant items were missing, then the penalty component would be
If no relevant items were missing the penalty component would be zero.
The weighted penalty was calculated as follows:
where the assumption is that the end user would been to examine half of the remaining documents to find the next relevant abstract, per missing relevant abstract. So if all relevant items were missing, then the summation would tend to one, and the penalty component tends to (N − n) * C p , while if only one relevant item is missing then, the penalty component is
To compute these measures we set C a = 1,C f = 2 and C p = 2, to represent the relative costs of the different actions. Note that these are not based on any empirical data and used as a way to regulate penalize feedback and no shows.
Fourteen groups from eleven countries submitted a total of 68 runs for this task: 2 categorizes the participating runs along five dimensions: (a) automatic vs manual runs; (b) use of the development set; (c) use of supervised and semi-supervised learning algorithms, (d) use of relevance feedback; and (e) thresholding the ranked list of articles. The categorization has been performed by the lab coordinators -not by the participants -based on the submitted participants description of their algorithms. Hence, there is always a chance of mis-classifying some run. Out of the 68 runs submitted, 52 focused on the simple evaluation framework, while 16 on the cost-effective one. Out of the 52 submitted runs for the simple sub-task, 35 ranked all the PMIDs that were returned by the Boolean query, while 17 tested different stopping criteria over the ranking. Participants employed both supervised and unsupervised methods, for ranking articles. A large number of runs were trained over the provided development set, and their generalization was tested against the test topics. 26 runs used the development set in some fashion, while 26 made no explicit use of it; it may be the case that participants tried different models and algorithms over the development set, and selected to submit the best performing ones, hence there may be a flavor of model selection, however we did not consider this as use of the development set. Participants represented the textual data in a variety of ways, including document-topic features, bag-of-words, topic model distributions, embeddings, metadata. In the remainder of section, by article we mean the abstract and the title of an article. We are not aware of any participant that worked on the full text of these articles.
In particular, AMC took a batch supervised approach, training a Random Forest over a topic model representation of the articles. A 75-topic model was fitted over all articles in the collection, and the Topic-to-Document matrix was used to extract features [2].
AUTH took a learning-to-rank approach, using both batch and active learning. Their model, HybridRankSVM, consists of two parts: an inter-topic model which utilizes XGBoost and is trained over the entire development corpus and an intra-topic model, an iteratively-built SVM, trained over relevance feedback provided partially in the test topics. For the inter-topic model a total of 24 topicdocument (or solely topic) features were computed over the title, abstract and mesh terms of the articles and the query. For the intra-topic model a TF-IDF vectorization of the articles was used [3].
CNRS trained a logistic regression model on n-gram features from the titles and abstracts and structured data from the Medline citations. One of their models was trained using stochastic gradient descent on the majority of the features, and one on the principal components of a subset of the features. Class imbalance was handled by reweighting and undersampling, while two approaches for relevance feedback were investigated [13].
ECNU took a learning-to-rank approach, using BM25, PL2, and BB2 as features. The trained model was also combined with a vector space model [4].
ETH used a LAMBDA-Mart model trained on features, such as BM25, Fuzzy search, Vector content representation, publishing data. This model was used to experiment with different stopping criteria. One of the approaches taken was to use minimal relevance feedback to estimate the distribution of positive samples by score. This was done by sampling from the articles, preferring articles with higher score. A Gaussian distribution was fitted on the positive samples and the resulting biased distribution was corrected. The correction worked by first adapting the mean and then iteratively finding the standard deviation matching the sampled data the best. For more details the reader can refer to [9]. NCSU adopted a continuous active learning framework for this task. An SVM classifier was trained on the relevance feedback labels and undersampling of the negatively labeled articles removing those furthest from the SVM decision hyperplane was employed. Different runs made use of different weights on the labels depending on whether the abstract or the full text was considered relevant [20].
NTU examined the role of convolutional neural networks for classifying medical articles for systematic reviews [12].
Padua used a two-dimensional probabilistic version of BM25 to rank articles. The parameters were tuned using the development set. Further, the top abstract returned by BM25 was provided to two non-experts who generated one additional query each. The tree queries were then used to re-rank articles. Different approaches for relevance feedback and thresholding were investigated [14].
QUT trained a learning-to-rank model using domain specific features. As domain specific features, PICO annotations (Population, Intervention, Control, Outcome) were used; these were extracted automatically from articles and manually from the Boolean queries [16].
Sheffield automatically parsed the Boolean queries to extract both the terms and MeSH heading,s and used TF-IDF cosine similarity to calculate the similarity score between document title and abstracts [1].
UOS explored two methods: (i) topic models, where they used Latent Dirichlet Allocation to identify topics within the set of retrieved articles, and then ranking articles by the topic most likely to be relevant to the query, and (ii) relevance feedback, where they used Rocchio's algorithm to update the query model for subsequent rounds of interaction. A third approach combined the topic model and relevance feedback approaches to quickly identify the relevant articles. For the thresholding task, they applied a score threshold over BM25 [11].
UCL took a supervised approach and trained a deep model architecture to identify studies pertaining to a given review topic [19].
Waterloo applied the Baseline Model Implementation (BMI) from the TREC Total Recall Track (2015-2016). They further applied their "knee-method" stopping criterion to BMI to determine how many abstracts should be examined for each topic [7].
Tables 7, 8, 9, 10 provide the results of a selection of the evaluation measures for all participating runs, both against the abstract and the document level relevance judgments, for the simple evaluation scenario. Figure 1 shows the corresponding box plots for Average Precision, with the Mean Average Precision against the abstract and document level judgments respectively denoted with a blue rectangle over the box plot.
In the following subsections we present results separately for ranking and thresholding runs, so that comparisons can be more meaningful.
Table 3 presents a number of evaluation measures for those runs that ranked the entire set of articles provided by the original Boolean queries; no thresholding has been applied. Some runs, as it may appear from Tables 7, 8, 9, 10, even though they applied no stopping criterion, still missed a number of documents. There may be multiple reasons for that, e.g. missing some topic, or not being able to download the abstract text, since participants were provided by PIDs only. The number of documents for which feedback was requested appears in the second column of the table, while the remaining of the columns report different measures of performance.
Figure 2 shows the recall-effort curves for the participating runs, that is the recall value at different percentage of documents shown to the user. The straight pink line with the triangular markers on x=y is the results of the Boolean query randomly shuffled, and it serves as a naive baseline, provided by the UOS team. The brown curve with the triangular markers is the BM25 retrieval function, also provided by the UOS team as a baseline; it ranks abstracts by BM25 over the Boolean query terms, with the default BM25 parameters setting. the effectiveness of the ranking algorithms, as expected, however it may come with additional cost in terms of assessing the relevance of abstract (based on the screening setup considered). Fig. 3. Box-plots of Mean Average Precision for runs that do not make use of relevance feedback and those that do make use.
Table 4 presents a number of evaluation measures for those runs that applied a threshold criterion. The total number of shown to the user abstracts can be found in the second column of the table, the number of documents for which feedback was requested in the third, while the remaining of the columns report different measures of performance. The cost measures account both for the cost of presenting a document to the user and for the additional cost of requesting feedback for a document, while they also account for the cost one would need to pay to reach 100% recall, under certain assumptions. Reliability considers the cost of not finding all relevant documents but makes no discrimination between the documents returned to the user and those for which feedback is requested. Average precision is well defined under the stopping criterion but hard to be used for comparing runs that use different thresholds. An easy to understand measure is the achieved recall at the rank of the threshold.
Figure 5 presents recall at the point of the threshold as a function of the number of documents presented to the user; that is at different stopping criteria, but also with different ranking and thresholding algorithms. As expected the more documents presented to the user (the lower the threshold criterion) the higher the achieved recall. Nevertheless, there are still algorithms that dominate others. The figure present the Pareto frontier. Figure 5 presents recall at the point of the threshold as a function of the feedback documents requested. As it can be viewed, although feedback documents, are in principle helpful towards achieving a high recall, there are algorithms that used no relevance feedback and still achieved high recall at a threshold.
Table 5 provides statistics on the topics used in the test set, along with the average Average Precision (AAP) for each topic, a measure that can be seen as a proxy of the difficult of each topic. The Pearson correlation coefficient between AAP and the percentage of relevant documents, the total number of documents, and the total number of relevant documents is -0.4868 (p-value = 0.006), 0.1295 (p-value = 0.495), and 0.8994 (p-value = 0). Figures 6 and 7 visually demonstrate this correlation. The CLEF 2017 e-Health Lab Task 2 constructed a benchmark collection of 50 Diagnostic Test Accuracy systematic reviews to study the effectiveness and efficiency of information retrieval and machine learning algorithms in prioritizing the studies to be screened at the abstract and title screening stage, and providing a stopping criterion over the ranked list. The results demonstrate that automatic methods can be trusted for finding most, if not all, relevant studies in a fraction of the time manual screening can do the same. Given that across different runs many parameters change simultaneously it is not easy to come to certain conclusions about the relative performance of automatic methods.
Regarding the benchmark collection itself, there is a number of limitations to be considered: (a) Pivoting on the results of the the OVID MEDLINE Boolean query limits our ability to identify all relevant studies, i.e. relevant studies that are outputted by Boolean queries over different databases, and relevant studies that are actually not found by these Boolean queries. The former can be overcome by considering all the different queries submitted; for the latter extra manual judgments would be required. (b) Pivoting on abstract and title only we miss the opportunity to study the effect of automatic methods when applied to the full text of the studies, that would present an opportunity to completely overcome the multi-stage process of systematic reviews. However, most of the full text articles are protected under copyright laws that do not give all participants access to those. (c) The evaluation setup of ranking does not allows us to consider the cost of the process, since given a ranking a researcher would have to still go over all studies ranked. A more realistic setup, e.g. a double-screening setup, could be considered. (d) In the construction of relevant judgments we considered the included and excluded references of the systematic reviews under study, which prevented us to study the noise and disagreement between reviewers. (e) In our effort to allow iterative algorithms, e.g. active learning algorithms, to be submitted, we handed the test sets' relevant judgments directly to the participants, which is rather unusual for this type of evaluation exercises. An alternative would be the setup used by the TREC Total Recall, where participants submitted their running algorithms to the organizers. (f) When it comes to evaluation measures there is a large variety of those, all of which take a different often useful view point on the effectiveness of algorithm, but which makes it difficult to decide upon a single golden measure to rank participants' runs.
10.1002/14651858.CD010438.pub2/full 10.1002/14651858.CD009551.pub3/full 10.1002/14651858.CD010775.pub2/full 10.1002/14651858.CD012019/full 10.1002/14651858.CD009175.pub2/full 10.1002/14651858.CD008686.pub2/full 10.1002/14651858.CD011984/full 10.1002/14651858.CD009020.pub2/full 10.1002/14651858.CD009786.pub2/full 10.1002/14651858.CD011548/full 10.1002/14651858.CD008643.pub2/full 10.1002/14651858.CD010896.pub2/full 10.1002/14651858.CD009579.pub2/full 10.1002/14651858.CD010023.pub2/full 10.1002/14651858.CD009925/full 10.1002/14651858.CD010772.pub2/full 10.1002/14651858.CD009944.pub2/full 10.1002/14651858.CD011145.pub2/full 10.1002/14651858.CD007431.pub2/full 10.1002/14651858.CD010409.pub2/full 10.1002/14651858.CD007427.pub2/full 10.1002/14651858.CD008054.pub2/full 10.1002/14651858.CD008803.pub2/full 10.1002/14651858.CD010771.pub2/full 10.1002/14651858.CD008122.pub2/full 10.1002/14651858.CD009694.pub2/full 10.1002/14651858.CD009593.pub3/full 10.1002/14651858.CD010705.pub2/full 10.1002/14651858.CD008782.pub4/full 10.1002/14651858.CD010633.pub2/full 10.1002/14651858.CD009647.pub2/full 10.1002/14651858.CD010173.pub2/full 10.1002/14651858.CD009135.pub2/full 10.1002/14651858.CD009591.pub2/full 10.1002/14651858.CD008760.pub2/full 10.1002/14651858.CD010386.pub2/full 10.1002/14651858.CD011549/full 10.1002/14651858.CD011021.pub2/full 10.1002/14651858.CD009263.pub2/full 10.1002/14651858.CD008691.pub2/full 10.1002/14651858.CD009519.pub2/full 10.1002/14651858.CD010632.pub2/full 10.1002/14651858.CD009372.pub2/full 10.1002/14651858.CD007394.pub2/full 10.1002/14651858.CD011134.pub2/full 10.1002/14651858.CD010783.pub2/full 10.1002/14651858.CD010079.pub2/full 10.1002/14651858.CD010860.pub2/full 10.1002/14651858.CD010276.pub2/full 10.1002/14651858.CD007424.pub2/full 10.1002/14651858.CD008081.pub3/full 10.1002/14651858.CD011431/full 10.1002/14651858.CD009185.pub2/full 10.1002/14651858.CD010339.pub2/full 10.1002/14651858.CD011975/full 10.1002/14651858.CD010653.pub2/full 10.1002/14651858.CD009323.pub2/full 10.1002/14651858.CD010542.pub2/full Table 6. The DOI's of the studies considered for the construction of the benchmark collection