s should be examined for each topic. According to thresholdbased evaluation, the knee method identi ed every article that should have been included (100% recall), while examining 2; 659 abstracts, on average, per topic|72.8% of the 3; 655 abstracts, that would have required examination, on average, had a manual approach been used instead. While our results suggest that TAR can substantially improve the e ciency of abstract screening without compromising recall, there remains room for improvement both in ranking and stopping criterion, as well as important factors that were not addressed in the CLEF eHealth 2017 framework: the completeness of the universe of abstracts gathered using keyword search, and the accuracy of the human assessments of the collected abstracts.
The University of Waterloo participated in Task 2, Technologically Assisted Reviews in Empirical Medicine [10], of the CLEF 2017 eHealth Evaluation Lab [12]. Task 2 simulates the second phase|screening |in a prototypical three-phase work ow to identify studies for inclusion in a systematic review: 1. Search: First, Boolean queries are used to identify as many articles as possible that may describe studies that should be included; 2. Screening: Second, titles and abstracts of the articles identi ed in the search phase are examined to eliminate those which could not possibly describe studies that should be included; and 3. Selection: Finally, articles that survived the screening phase are read in full to determine whether or not they meet the systematic review inclusion criteria. The overall objective of our research is to improve the human e ciency, as well as the e ectiveness, of work ows to identify studies for inclusion in systematic reviews. The results of our CLEF experiments support the hypothesis that continuous active learning (\CAL") can substantially improve the human e ciency of screening, without substantially compromising its e ectiveness. The results also are consistent with the further hypothesis that CAL actually improves e ectiveness by identifying articles missed in the search phase, or articles mistakenly eliminated during the screening phase. While this hypothesis cannot be tested immediately within the framework of Task 2, we have identi ed a set of articles that, were it determined that they describe one or more studies that should have been included in the review, would demonstrate CAL's superior e ectiveness. Task 2 is essentially the Technology-Assisted Review (\TAR") task addressed by the TREC 2015 and TREC 2016 Total Recall Tracks [11, 8]. For our participation in CLEF, we reprised our Total Recall e orts using the same apparatus.
At TREC, the systems under test were given, at the outset, a corpus of documents and a set of topics. For each topic, a system under test repeatedly submitted documents from the corpus to a server, and in return, was given a simulated human assessment of \relevant" or \not relevant" for each document.
The objective was to identify as many relevant documents as possible, while submitting as few non-relevant documents as possible. The tension between these Algorithm 1 The AutoTAR Continuous Active Learning (\CAL") Method, as Implemented by the TREC Baseline Model Implementation (\BMI") and deployed by Waterloo for the CLEF Technologically Assisted Review Task. 1. The initial training set consists of a synthetic document containing only the topic title, labeled as \relevant." 2. Set the initial batch size B to 1. 3. Temporarily augment the training set by adding 100 random documents from the collection, provisionally labeled as \not relevant." 4. Apply logistic regression to the training set. 5. Remove the random documents added in step 3. 6. Select the highest-scoring B documents that have not yet yet been screened. 7. Label each of the B documents as \relevant" or \not relevant" by consulting: (a) Previous \abstract" assessments supplied by CLEF [Method A]; or, (b) Previous \document" assessments, once the rst \relevant" document assessment is encountered [Method B]. 8. Add the labeled documents to the training set. 9. Increase B by 1B0 . 10. Repeat steps 3 through 10 until either: (a) All documents have been screened [for ranked evaluation]; or, (b) The \knee-method" stopping criterion is met [for threshold evaluation]. two criteria was evaluated using rank-based measures (e.g., recall as a function of the number of documents submitted), as well as set-based measures (e.g., recall at a point when a certain number of documents, speci ed contemporaneously by the system, had been submitted).
Prior to TREC, we made available a Baseline Model Implementation
(\BMI"),1 to illustrate the client-server protocol, as well as to provide
baseline results for comparison. BMI, which encapsulates our AutoTAR Continuous
Active Learning (\CAL") method [
Task 2 di ered operationally from the TREC Total Recall Track in that a
list of document identi ers, rather than a corpus, was supplied at the outset,
and a complete set of relevance assessments, rather than an assessment server
were used to simulate human assessments. Task 2 also di ered substantively
from the Total Recall Track in that the corpus for each topic was narrowed
by a search phase speci c to that topic, and therefore yielded a much smaller
set that was richer in relevant documents. Task 2 di ered further in that two
sets of relevance assessments were available: the assessments from a previously
conducted screening phase, and the assessments from a previously conducted
selection phase, raising the question of which assessments (or combination of
assessments) should be used to simulate relevance feedback, and which should
be used to evaluate the results (cf. [
Task 2 provides no method equivalent to TREC's \call your shot" for a system under test to specify a stopping criterion (for threshold-based evaluation), while at the same time continuing until every document in the corpus has been submitted for assessment (for rank-based evaluation).
Task 2, however, unlike TREC, a orded participants the opportunity to conduct task-speci c tuning and con guration, by supplying 20 training topics (with corresponding corpora and assessments) in advance of the exercise, followed by 30 test topics, which were used for evaluation.
The corpus for each topic consisted of abstracts from MEDLINE/Pubmed2 identi ed by PMID. On March 8, 2017, we fetched the entire MEDLINE dataset consisting of 27,348,935 XML les, each containing the titles, abstracts, and metadata for an article. We used the raw XML les as documents in the corpora that were supplied at the outset to BMI.
http://cormack.uwaterloo.ca/trecvm. 2 See https://www.nlm.nih.gov/bsd/pmresources.html.
Public
License at
Our original intent had been to apply BMI to the entire corpus of 27,348,935 les, thus combining the search and screening phases. When we employed this strategy in a pilot experiment on the test topics, we found that no assessments were available for many, if not most, of the highly ranked documents returned by BMI. To our eye, these documents were indistinguishable from those for which \relevant" assessments were provided. We investigated, without success, the reasons why these documents were not retrieved by the previously conducted search phase. For example, the documents in question were neither newer nor older than those for which assessments were available, and appeared to contain relevant terms from the search query. As we were unable to reproduce the results of the CLEF search phase, we chose to ignore|for the purpose of relevance feedback and evaluation|documents for which no assessments were available. Ignoring these unjudged documents, our pilot experiment yielded what appeared to be reasonable rank-based results.
Ignoring documents for feedback and evaluation yields a substantially different result from removing them from the corpus altogether. In a second pilot experiment, we constructed a separate corpus for each topic, consisting of only those documents for which relevance assessments were available. While BMI ran much faster on these reduced corpora than on the 27M dataset, results were apparently inferior. We conjecture that this inferior result can be explained by skewed term-frequency statistics in the reduced corpora.
As a compromise between the e ectiveness of searching the 27M dataset and the (computational) e ciency of searching the reduced corpora, we conducted a third pilot experiment using a common corpus consisting of all documents that were assessed for any of the 20 test topics. That is, for any given topic, the corpus consisted of all the documents assessed for that topic, as well as all the documents assessed for each of the other 19 topics. Our rationale was that including documents retrieved for all topics would introduce enough diversity to unskew su ciently the term-frequency statics. This approach appeared to achieve the e ciency of using reduced corpora and the e ectiveness of using the full dataset, and was chosen for our o cial tests: For the o cial tests, the corpus consisted of all documents assessed for any of the 30 test topics (less four documents whose PMIDs were not present in our MEDLINE database); from this corpus, we submitted and solicited feedback only for documents for which assessments were available. 3.2
We investigated three modes of relevance feedback, of which only two were selected for o cial testing: 1. Relevance feedback based on the screening-phase assessments (selected as
Method A for o cial testing); 2. Relevance feedback based on the selection-phase assessments (not selected for o cial testing); 3. Relevance feedback based on a hybrid of screening-phase and selection-phase assessments (selected as Method B for o cial testing).
The rst and second methods are straightforward: When BMI identi es a document for assessment, the judgment returned to BMI is that supplied by CLEF for either the screening phase (the \abstract qrels") or the selection phase (\the content qrels"). The third method operates in two phases: At the outset, the judgment returned to BMI is that of the abstract qrels. The abstract qrels continue to be used until BMI identi es one document that is relevant not only according to the abstract qrels, but also according to the content qrels. Thereafter, the judgment returned to BMI is that of the content qrels.
In our pilot experiments, we found that the rst method consistently yielded superior rank-based results, whether evaluated using the abstract qrels or the content qrels. The second method yielded consistently inferior results. The third method showed similar, but slightly inferior, results, to the rst method, when evaluated using the content qrels. Based on our pilot results, we selected the rst and third methods, denoted as Method A and Method B, respectively, for our o cial experiments. 3.3
For threshold-based evaluation, it was necessary to implement a stopping
procedure to terminate screening when the best compromise between recall and e ort
had been achieved, for some de nition of \best." In our opinion,
technologyassisted review should be considered a satisfactory alternative to manual
review only if it yields comparable or superior recall, with high probability.
Toward this end, we deployed our knee method with default parameters ( =
156 min(relret; 150), = 100 [
The Task 2 guidelines specify a plethora of run types and evaluation measures, which may be classi ed on two orthogonal dimensions: 1. Rank-based vs. threshold-based (or set-based) evaluation; and 2. Simple vs. cost-sensitive scoring.
The strategies to optimize these measures are incompatible, occasioning us to submit four versions of the output from each of our two runs, for a total of eight submissions, detailed in Table 1. The only di erence between the \rank" and \thresh" runs is that the latter are truncated using the knee-method stopping procedure; the only di erence between the \normal" and \cost" runs is that the \interaction eld" \AF" is replaced by \AFS" where the document receives a \relevant" assessment, and by \AFN" where the document receives a \nonrelevant" assessment.
In 2015, we published the details and rationale for AutoTAR [
The AutoTAR/BMI algorithm, as modi ed for CLEF, is detailed in
Algorithm 1, which is reproduced from [
Internally, BMI constructs a normalized TF-IDF ((1 + log tf ) log dNf ) wordvector representation of each document in the corpus (which, as noted in Section 3.1, consists of raw XML les), where a word is considered to be any sequence of two or more alphanumeric characters not containing a digit, that occurs at least twice in the corpus. Scoring is e ected by So a-ML3 with parameters \--learner type logreg-pegasos --loop type roc --lambda 0.0001 --iterations 200000." As noted above, these parameters were xed when BMI was created in 2015. 5
We present separately the results for our threshold-based and rank-based runs, reporting only simple threshold-based and simple rank-based measures for each, computed using the content qrels. At the time of writing, cost-sensitive evaluation was not available to CLEF participants. 5.1
Our threshold-based results are shown in Table 2. Perhaps the most important result is shown in the rst three lines: Across 30 topics, Method A identi ed all 607 articles referencing studies that should have been included, thus achieving 100% recall. Method B, on the other hand, identi ed 575 of the articles, achieving 97.9% recall. Method A, however, entailed the review of 79,765 (72.8%) of the
109,560 abstracts identi ed by the search phase, while method B entailed the review of only 52,934 (48.3%) of the documents.
In other words, Method A was more e ective, but Method B was more efcient. According to the combined loss measure which considers both factors, Method B was superior. 5.2
Our rank-based results are shown in Table 3. Work saved over sampling (\WSS")|a measure commonly reported for systematic review|re ects how many fewer documents would have been needed to have been reviewed to achieve a particular level of recall, if it were somehow known exactly when that level had been achieved. Thus, WSS, along with all other rank-based measures, is a measure of what might have been, rather than achieved e ectiveness. According to WSS, Method A is marginally inferior to Method B at 95% recall (0.815 vs. 0.824), and at 100% recall (0.823 vs. 0.830).
Conversely, Method A is marginally superior to Method B in terms of the number of documents that had to be examined per topic before 100% recall was achieved (461 vs. 469, representing 12.6% and 12.8%, respectively, of the average number of documents per topic). In other words, Method A could have achieved 100% recall with roughly on-sixth the review e ort, had a stopping procedure been able to determine when 100% recall had occurred. Similarly, Method B could have achieved 100% recall with roughly four times less e ort that it actually required to achieve 97.9% recall, had a stopping procedure been available.
The Normalized Cumulative Gain (\NCG") results|which report the recall achieved when a speci ed fraction (between 10% and 100%) of the documents have been reviewed|tell much the same story: Very high recall could have been achieved at a fraction of the review e ort, had it been know when high recall had been achieved.
In our opinion, cumulative measures like norm area and average precision yield very little insight into the actual or hypothetical e ectiveness of technologyassisted review for screening purposes. 6
We believe that both sets of the CLEF assessments are incomplete with respect to the overall objective of identifying all studies that should be included in the review: The screening assessments are available only for documents retrieved by the search phase; the selection assessments are available only for documents retrieved by the search phase, and judged relevant during the screening phase. Therefore, from the assessments, it is impossible to determine whether an article not retrieved by the search phase, or an article eliminated during the screening phase, describes a study that should have been included in the review. The Task 2 architecture tacitly assumes that no such articles exist; in other words, that the search and screening phases used to generate the relevance assessments were infallible, and each attained 100% recall.
Such an assumption is unrealistic, and limits the recall of any simulated TAR
method to that of the manual review to which it is compared [
Our hypothesis that our TAR runs found relevant articles that were missed
by the search phase, or incorrectly discarded in the screening phase, is based
on results from other domains [