Screening articles for studies to include in systematic reviews is an application of technology-assisted review (\TAR"). In this work, we applied the Baseline Model Implementation (\BMI") from the TREC Total Recall Track (2015-2016) to the CLEF eHealth 2018 task of screening MEDLINE abstracts to identify articles reporting studies to be considered for inclusion. We employed exactly the same approach for Sub-Task 1 and Sub-Task 2, which was in turn exactly the same approach employed for the CLEF 2017 eHealth Lab. The only di erence was that for SubTask 1, the entire Pubmed/MedLine database was searched; whereas for Sub-Task 2, the only records searched were those identi ed by CLEF using Boolean queries.
Task 2 is essentially the Technology-Assisted Review (\TAR") task addressed by
the TREC 2015 and TREC 2016 Total Recall Tracks [
At TREC, the systems under test were given, at the outset, a corpus of documents and a set of topics. For each topic, a system under test repeatedly submitted documents from the corpus to a server, and in return, was given a simulated human assessment of \relevant" or \not relevant" for each document.
The objective was to identify as many relevant documents as possible, while submitting as few non-relevant documents as possible. The tension between these two criteria was evaluated using rank-based measures (e.g., recall as a function of the number of documents submitted), as well as set-based measures (e.g., recall at a point when a certain number of documents, speci ed contemporaneously by the system, had been submitted).
Prior to TREC, we made available a Baseline Model Implementation
(\BMI"),1 to illustrate the client-server protocol, as well as to provide
baseline results for comparison. BMI, which encapsulates our AutoTAR Continuous
Active Learning (\CAL") method [
Sub-Task 2, which was the only task for CLEF 2017, di ered operationally
from the TREC Total Recall Track in that a list of document identi ers, rather
than a corpus, was supplied at the outset, and a complete set of relevance
assessments, rather than an assessment server were used to simulate human
assessments. Sub-Task 2 also di ered substantively from the Total Recall Track in
that the corpus for each topic was narrowed by a search phase speci c to that
topic, and therefore yielded a much smaller set that was richer in relevant
documents. Sub-Task 2 di ered further in that two sets of relevance assessments were
available: the assessments from a previously conducted screening phase, and the
assessments from a previously conducted selection phase, raising the question of
which assessments (or combination of assessments) should be used to simulate
relevance feedback, and which should be used to evaluate the results (cf. [
Sub-Task 1, new for CLEF 2018, resembles the TREC Total Recall Track, in that no topic-speci c culling of the document set is done; each search applies to the entire 30M-document Pubmed/MEDLINE collection.
http://cormack.uwaterloo.ca/trecvm. General
Public
License at
The corpus for each topic consisted of abstracts from MEDLINE/Pubmed2 identi ed by PMID. On April 1, 2018, we fetched the entire MEDLINE dataset consisting of 28,256,688 XML les, each containing the titles, abstracts, and metadata for an article. We used the raw XML les as documents in the corpora that were supplied at the outset to BMI.
For Sub-Task 1, we applied BMI to the entire corpus of 28,256,688 les, thus combining the search and screening phases. In a pilot experiment on the test topics, we found that no assessments were available for many, if not most, of the highly ranked documents returned by BMI. To our eye, these documents were indistinguishable from those for which \relevant" assessments were provided. We investigated, without success, the reasons why these documents were not retrieved by the previously conducted search phase. For example, the documents in question were neither newer nor older than those for which assessments were available, and appeared to contain relevant terms from the search query. Nonetheless, for Sub-Task 1, we treated any document for which no qrel was available to be \not relevant" for the purpose of feedback.
In a separate manual run, the authors used their own judgement to assess the relevance of abstracts returned by BMI, in order to provide relevance feedback.
For Sub-Task 2, we used a common corpus consisting of all documents that were assessed for any of the 30 test topics. That is, for any given topic, the corpus consisted of all the documents assessed for that topic, as well as all the documents assessed for each of the other 29 topics. Our rationale was that including documents retrieved for all topics would introduce enough diversity to unskew su ciently the term-frequency statics. This approach appeared to achieve the e ciency of using reduced corpora and the e ectiveness of using the full dataset, and was chosen for our o cial tests: For the o cial tests, the corpus consisted of all documents assessed for any of the 30 test topics; any unassessed document was considered \not relevant." 3.2
For Sub-Task 2, we used two modes of relevance feedback: 1. Relevance feedback based on the screening-phase assessments (Method
UWA.Task2 for o cial testing); 2. Relevance feedback based on a hybrid of screening-phase and selection-phase assessments (Method UWB.Task2 for o cial testing).
The rst method is straightforward: When BMI identi es a document for assessment, the judgment returned to BMI is that supplied by CLEF for either the screening phase (the \abstract qrels"). The second method operates in two
phases: At the outset, the judgment returned to BMI is that of the abstract qrels. The abstract qrels continue to be used until BMI identi es one document that is relevant not only according to the abstract qrels, but also according to the content qrels. Thereafter, the judgment returned to BMI is that of the content qrels.
For Sub-Task 3, we used three modes of relevance feedback: 1. Relevance feedback based on the screening-phase assessments (Method UWA .Task1 for o cial testing); 2. Manual feedback based on the authors' relevance assessments (Method
UWG.Task1 for o cial testing); 3. Manual feedback based on the authors' relevance positive assessments, followed by relevance feedback based on the screening-phase assessments (Method UWX.Task1 for o cial testing). 3.3
For threshold-based evaluation, it was necessary to implement a stopping
procedure to terminate screening when the best compromise between recall and e ort
had been achieved, for some de nition of \best." In our opinion,
technologyassisted review should be considered a satisfactory alternative to manual
review only if it yields comparable or superior recall, with high probability.
Toward this end, we deployed our knee method with default parameters ( =
156 min(relret; 150), = 100 [
In 2015, we published the details and rationale for AutoTAR [
The AutoTAR/BMI algorithm, as modi ed for CLEF, is detailed in
Algorithm 1, which is reproduced from [
Algorithm 1 The AutoTAR Continuous Active Learning (\CAL") Method, as Implemented by the TREC Baseline Model Implementation (\BMI") and deployed by Waterloo for the CLEF Technologically Assisted Review Task. 1. The initial training set consists of a synthetic document containing only the topic title, labeled as \relevant." 2. Set the initial batch size B to 1. 3. Temporarily augment the training set by adding 100 random documents from the collection, provisionally labeled as \not relevant." 4. Apply logistic regression to the training set. 5. Remove the random documents added in step 3. 6. Select the highest-scoring B documents that have not yet yet been screened. 7. Label each of the B documents as \relevant" or \not relevant" by consulting: (a) Previous \abstract" assessments supplied by CLEF [Method A]; or, (b) Previous \document" assessments, once the rst \relevant" document assessment is encountered [Method B]. 8. Add the labeled documents to the training set. 9. Increase B by 1B0 . 10. Repeat steps 3 through 10 until either: (a) All documents have been screened [for ranked evaluation]; or, (b) The \knee-method" stopping criterion is met [for threshold evaluation]. Internally, BMI constructs a normalized TF-IDF ((1 + log tf ) log dNf ) wordvector representation of each document in the corpus (which, as noted in Section 3.1, consists of raw XML les), where a word is considered to be any sequence of two or more alphanumeric characters not containing a digit, that occurs at least twice in the corpus. Scoring is e ected by So a-ML3 with parameters \--learner type logreg-pegasos --loop type roc --lambda 0.0001 --iterations 200000." As noted above, these parameters were xed when BMI was created in 2015. 5
the review: The screening assessments are available only for documents retrieved by the search phase; the selection assessments are available only for documents retrieved by the search phase, and judged relevant during the screening phase. Therefore, from the assessments, it is impossible to determine whether an article not retrieved by the search phase, or an article eliminated during the screening phase, describes a study that should have been included in the review. The CLEF architecture tacitly assumes that no such articles exist; in other words, that the search and screening phases used to generate the relevance assessments were infallible, and each attained 100% recall.
Such an assumption is unrealistic, and limits the recall of any simulated TAR
method to that of the manual review to which it is compared [
Our hypothesis that our TAR runs found relevant articles that were missed
by the search phase, or incorrectly discarded in the screening phase, is based
on results from other domains [