This paper describes Sheffield University's submission to CLEF 2017 eHealth Task 2: Technologically Assisted Reviews in Empirical Medicine. This task focusses on the identification of relevant evidence for systematic reviews in the medical domain. Participants are provided with systematic review topics (including title, Boolean query and set of PubMed abstracts returned) and asked to identify the abstracts that provide evidence relevant to the review topic. Sheffield University participated in the simple evaluation. Our approach was to rank the set of PubMed abstracts returned by the query by making use of information in the topic including title and Boolean query. Ranking was based on a simple TF.IDF weighted cosine similarity measure. This paper reports results obtained from six runs: four submitted to the official evaluation, an additional run and a baseline approach.
Systematic reviews attempt to identify, synthesise and summarise evidence available to
answer a research question. They form the backbone of evidence-based approaches to
medicine where they are used to answer complex questions such as “How effective are
statins for heart attack survivors?" [
The process of creating a systematic review is time-consuming with a single review
often requiring 6 to 12 months of effort from expert reviewers [
This paper is organised as follows: Section 2 introduces CLEF eHealth Task 2. Section 3 describes our approach to this task. Section 4 discusses the results obtained from applying this approach to both the development and test datasets. Finally, Section 5 presents the conclusions and potential future work. 1. Boolean Search: Experts construct a boolean query designed to identify all evidence relevant to the review question. This query is run against a medical database such as PubMed and set of titles and abstracts returned. 2. Title and Abstract Screening: Experts screen the titles and abstracts retrieved to identify those that are potentially relevant for inclusion in the review. 3. Document Screening: The full document content is then retrieved for any title and abstract that has been identified as being relevant in the previous stage. These are then examined in a second round of expert screening to form a final decision about their relevance to the review.
In CLEF eHealth 2017 [
Participants are provided with two datasets: a development set and a test set. The development dataset contains 20 topics and the test dataset contains 30 topics. All reviews focus on Diagnostic Test Accuracy (DTA). The queries were manually constructed by expert reviewers from the Cochrane collaboration1. For each topic, participants are provided with topic id, review title, boolean query and a list of PubMed documents identifiers retrieved by the query. The collection contains a total of 266,967 abstracts.
Figure 1 shows examples of two topics from the development dataset. Two different
formulations were used for the Boolean queries: OVID and PubMed. The queries are
generally complex and contain multiple operators. Table 1 shows operators commonly
used in both types of query [
Participants also provided with files that indicate which of the titles and abstracts returned by the Boolean query were indicated as being relevant after the Title and Abstract Screening and Document Screening stages (see Section 2), referred to as the abstract qrels and content qrels respectively. 3.2
The University of Sheffield’s submission to Task 2 ranked the list of PubMed abstracts retrieved for each topic with the intention of returning relevant ones as early as possible. The approach is completely automatic since queries are processed algorithmically and without manual intervention2. In addition, relevance feedback is not used.
Our method makes use of three pieces of information from the topic: (1) the title, (2) terms extracted from the Boolean query and (3) MeSH terms extracted from the Boolean query. Information for (2) and (3) are extracted from the Boolean query using a simple parser designed to interpret both OVID and PubMed style queries. Terms
Topic: CD009591 Title: Imaging modalities for the non-invasive diagnosis of endometriosis Query: exp magnetic resonance imaging/ or exp ultrasonography/ or exp Imaging, Three-Dimensional/ or exp radiography/ ultraso$.tw. or magnetic resonance imaging.tw. or MRI.tw. or imag$.tw. diagnos$.tw. ... (animals not (humans and animals)).sh. 8 not 9 PubMed Topic: CD008643 Title: Red flags to screen for vertebral fracture in patients presenting with low-back pain Query: 1 Index test: clinical red flags "Medical History Taking"[mesh] OR history[tw] OR "red flag"[tw] OR "red flags" OR Physical examination[mesh] OR "physical examination" [tw] OR "function test"[tw] OR "physical test"[tw] ... 1 AND 2 AND 3 NOT 4 and MeSH terms modified by certain operators (e.g. not and adj) are not extracted. Figure 2 shows examples of terms extracted from the query for topic CD008643 (see Figure 1). Some MeSH terms (e.g. Spine) are also standard English words that could appear as a term in an abstract. To avoid false matches all MeSH terms extracted from a query are prefixed with the string Mesh. In addition, MeSH terms are preprocesssed to remove whitespace and punctuation (e.g. Lumbar vertibrae becomes MeshLumbarvertibrate). Example MeSH terms extracted from the same query are shown in Figure 3.
’history’, ’red flag’, ’physical examination’, ’function test’, ’physical test’,’clinical’, ’clinically’,’diagnosis’ ’MeSHMedicalHistoryTaking’, ’MeSHPhysicalexamination’, ’MeSHra’, ’MeSHri’, ’MeSHWoundsandInjuries’ The abstracts returned by the Boolean query for each topic defined as the list of PMIDs (PubMed identifier) provided with the topic are downloaded from PubMed3. The text of the title, abstract and MeSH terms are extracted and the MeSH terms preprocessed using the same approach that was applied to the Boolean query.
Pre-processing is applied to both the PubMed abstracts and information extracted from the topics. The text is tokenised, converted to lower case, stop words/punctuation are removed and the remaining tokens stemmed4.
The information extracted from the topic and each of the abstracts are converted into tf.idf-weighted vectors. The similarity between the topic and each of the abstracts is then generated by computing the cosine metric for the pair of vectors5. Abstracts are ranked based on this similarity score.
Results are output in the TREC format shown in Table 2 where: – TOPIC-ID: topic identifier provided by CLEF 2017. – INTERACTION: this field is assigned the value NF in all our runs to indicate that relevance feedback is not used – PID: PubMed document identifier – RANK: rank of the document according to the cosine similarity score – SCORE: cosine similarity score described above – RUN-ID: run identifier 3.3
Four runs were officially submitted for the official evaluation: Sheffield-run-1, Sheffieldrun-2, Sheffield-run-3, and Sheffield-run-4. In addition, a baseline run (Sheffield-baseline) and additional approach (Sheffield-run-5) were also implemented and evaluated. A description of each run is presented below.
4 NLTK’s tokenize and LancasterStemmer packages are used for tokenisation and stemming. The list of stop words provided by scikit-learn (scikit-learn.org/stable/) is used for most runs. 5 Scikit-learn’s TfidfVectorizer and linear_kernel packages were used for these steps – Sheffield-baseline In this run the list of PubMed abstracts are randomly ordered.
This is intended to represent the scenario in which the results of the Boolean query are simply evaluated in the order in which they are retrieved without any attempt to identify those most likely to be relevant. This situation simulates common practise within many systematic review projects in which reviewers examine each of the retrieved abstracts in turn. The score of each abstract is calculated using the following equation:
n r + 1 score = (1) n where n is the total number of abstracts returned by the Boolean query and r the abstract’s rank in the random ordering. – Sheffield-run-1 Abstracts returned by the Boolean query are ranked by comparing them against only the topic title. – Sheffield-run-2 Abstracts are compared with the topic title and terms extracted from the Boolean query. – Sheffield-run-3 Abstracts are compared with the topic title and both terms and
MeSH terms extracted from the Boolean query.
– Sheffield-run-4 This run is the same as Sheffield-run-2 except that the PubMed
stop-words list [
Task 2 consists of two formal evaluations: simple evaluation and cost-effective evaluation. The University of Sheffield participated only in the simple evaluation setup and did not attempt to optimise the approach for the cost-effective evaluation. Evaluation was carried out using the script provided by the task organisers6. 4.1
The development dataset contains of 20 DTA topics (see Section 3.1). Tables 3 and 4 present the results for the approaches described in Section 3.3 applied to this dataset for the abstract and content qrels respectively.
As expected, all of the implemented methods outperform the simple baseline
approach. This demonstrates that even straightforward ranking techniques provide
potential benefit to systematic reviewers by ensuring that documents more likely to be
relevant are placed higher in the rankings. We have previously demonstrated a similar
results for a single systematic review [
The best result of the submitted runs for the abstract qrels (Table 3) was achieved by Sheffield-run-4 which achieved the average precision (ap) score of 0.223, an improvement of 0.173 against the baseline. It also achieved the best results for work saved over sampling (wss) and area under the cumulative recall curve normalized by the optimal area (norm_area) metrics. It is also close to the best result for the average of the minimum number of abstracts returned to retrieve all relevant ones (last_rel) metric.
For the content qrels (Table 4), both Sheffield-run-4 and Sheffield-run-5 are strong. Sheffield-run-4 produced the best scores for last_rel and norm_area and close to the best result of wss. Sheffield-run-5 achieved the best score for ap and wss_95.
Results from the development dataset suggest that including terms extracted from the Boolean query is beneficial (e.g. compare Sheffield-run-1 and Sheffield-run-2). However, the usefulness of MeSH terms extracted is less clear. Performance decreases when these are added to the title and query terms (e.g. compare Sheffield-run-2 and Sheffield-run-3). Results are mixed when they are used instead of query terms (e.g. compare Sheffield-run-1 and Sheffield-run-5), there is no improvement for the abstract evaluation but some benefit for the content evaluation. The development dataset contains of 30 DTA topics (see Section 3.1). Tables 5 and 6 show the results for the abstract and content qrels respectively.
The highest ap scores were achieved using Sheffield-run-2 and Sheffield-run-4 for both the abstract and content qrels (Tables 5 and 6). The overall pattern of results suggest that Sheffield-run-4 is the best performing run on the test data.
Results from the development and test datasets indicate the strong relative performance of Sheffield-run-4. This indicates that including terms extracted from Boolean query and using the PubMed stop-words list are benefical for this task.
There were some relevant documents in the test data set for which our approach assigned a score of 0 and this caused NCG@100 scores to be less than 1. This was observed at both the content and abstract level for the development and test datasets. The scoring script treats these documents as not being included in the ranking. The problem could be resolved by adding a small delta value to each score. 5
This paper described the University of Sheffield’s approach to CLEF 2017 Task 2. Information from the review title and Boolean query was used to rank the abstracts returned by the query using standard similarity measures. The title and terms extracted from the Boolean query were found to be the most useful information for this task. All of the submitted runs outperform a baseline approach based on random ordering.
In future we plan to refine the techniques for extracting terms and MeSH terms from the Boolean query (Section 3.2) by taking account of the query structure and MeSH hierarchy. We also plan to develop techniques to minimise the cost of identifying relevant evidence and make use of ActiveLearning to improve the ranking based on feedback from reviewers.