Systematic reviews are widely considered as the highest form of medical evidence, since they aim to be a repeatable, comprehensive, and unbiased summary of the existing literature. Because of the high cost of missing relevant studies, review authors go to great lengths to ensure all relevant literature is included. It is not atypical for a single review to be conducted over the course of months or years, with multiple authors screening thousands of articles in a multi-stage triage process; rst on title, then on title and abstract, and nally on full text. Figure 1a shows a typical literature screening process for systematic reviews.
In the last decade, the information retrieval (IR) and machine learning (ML)
communities have shown increasing interest in literature searches for systematic
reviews [1{3]. Literature screening for systematic reviews can be characterised
as a classi cation task with two de ning features; a requirement for near perfect
recall on the class of relevant studies (the high cost of missing relevant evidence),
and highly imbalanced training data (review authors are often willing to screen
thousands of citations to nd less than 100 relevant articles). Previous attempts
at automating literature screening for systematic reviews have primarily focused
on two questions; how to build a suitably high recall model for the target class
in a given review under the conditions of highly imbalanced training data [
When screening articles, reviewers exclude studies for a number of reasons (animal populations, incorrect disease etc.). Additionally, in any given triage stage a study may not be relevant but still progress to the next stage as the authors have insu cient information to exclude it (i.e. the title may not indicate a study was performed with an animal population, however this may become apparent upon reading the abstract). We meet the requirement for near perfect recall on relevant studies by inverting the classi cation task and identifying subsets of irrelevant studies with near perfect precision. We attempt to identify such studies by training the classi er using the labels assigned at the previous triage stage (see Figure 1c). The seamless integration with the existing manual screening process is an advantage of our approach.
The classi er is built by rst selecting terms from the title and abstracts with the greatest information gain on labels assigned in the rst triage stage. Articles Reviewer 1 Screens on Ful Text
Reviewer 2 Screens on
Ful Text (a) { 'neutropenia', but 'infection' or 'thorax'
not { 'skin' but not 'thorax' { 'immunoglobulin g' { 'animals' { 'drug therapy', but not 'risk' or 'infection' (b)
Exclude Obtain Title and Abstract
Initial Screening on Title Alone Obtain Title and Abstract Reviewer 1 Screens on Title and Abstract
Build and Run
Classifier
Resolve Both Exclude Exclude One or more Include
Reviewer 2 Screens on Title and Abstract DisRcreespoalvnecies Exclude Obtain Ful
Text (c) are then represented as Boolean statements over these terms, and interpretable rules are then generated using Boolean minimisation (examples of rules are given in 1b Review authors can then re ne the classi er by selecting only those rules most likely to describe non-relevant studies, maximising overall precision.
Preliminary experiments simulating the process outlined in Figure 1c on a previously conducted systematic review indicate that as many as 25% of articles can be safely eliminated without the need for screening by a second reviewer. The evaluation does assume that all false positives (studies erroneously excluded by the generated rules) were included by the rst reviewer. Such an assumption is reasonable; the reason for multiple reviewers is that even human experts make mistakes. A study comparing the precision of our classi er to human reviewers is planned. In addition, future work will focus on improving the quality of the generated rules by trying to better capture reasons for excluding studies matching those used by human reviewers.