This paper describes the University of She eld's approach to the CLEF 2019 eHealth Task 2: Technologically Assisted Reviews in Empirical Medicine. This task focuses on identifying relevant studies for systematic reviews. The University of She eld participated in subtask 2 (Abstract and Title Screening). Our approach used lexical statistics (LogLikelihood, Chi-Squared and Odds-Ratio) to identify terms that retrieve speci c types of evidence. A total of 12 o cial runs were submitted.
Systematic reviews aim to collect, synthesise and summarise all available evidence that answers a speci c research question. Medical practitioners and decision makers rely on the information they contain to guide treatment decisions.
Cochrane is one the key producers of medical systematic reviews. Its library
contains 7,987 reviews1 which fall into ve categories [
Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CLEF 2019, 9-12 September 2019, Lugano, Switzerland.
Systematic reviews are time-consuming to create, it may take up to a year
to conduct a single review [
CLEF eHealth 2019 Task 2 Subtask 2 [
This paper is structured as follows: Section 2 describes the datasets and approaches used, Section 3 the experiments conducted and Section 4 states and discusses the results obtained. 2 2.1
CLEF2019 dataset is partitioned into training and testing datasets. The training dataset contains two types of review (72 DTA and 20 Intervention) while the test dataset contains four types (eight DTA, 20 Intervention, two Qualitative and one Prognosis). For each review participants are provided with the review title, Boolean query, set of PMIDs and relevance judgements for both title/abstract and content level screening. 2.2
She eld's submission extended the approach that had been developed for our
previous entries to the task. The core of our approach extracts terms from the
Boolean query and uses them to rank the studies [
Our submissions to the 2019 task extended this approach to multiple review
types by developing lists of terms that indicate the relevant evidence for a
speci c review type. Lexical statistics were used to automatically derive these lists
of key terms. Three lexical statistics were applied: Log-Likelihood, Chi-Squared
and Odds-Ratio [
Erel = Nrel ;
Eirrel = Nirrel
where Nrel and Nirrel represent sub-corpus size (e.g. relevant and irrelevant documents). Terms are assigned high Log-Likelihood scores for a particular corpus when their observed frequency is (much) higher than the expected frequency. Chi-Squared is computed as Odds-Ratio is computed as
Chi-Squared = (Orel
2 Erel) + (Oirrel
2
where Orel and Oirrel are the observed values and Erel and Eirrel are expected values calculated using equation 2.
Odds-Ratio =
(Nirrel (Nrel
where Orel and Oirrel are the frequency counts of the term in the relevant and irrelevant sub-corpus and Nrel and Nirrel are the total number of terms in each of those sub-corpus. (1) (2) (3) (4)
Four o cial runs were submitted for DTA and Intervention reviews: she eldbaseline, she eld-Log Likelihood, she eld-Chi Squared and she eld-Odds Ratio. Two o cial runs were submitted for Prognosis and Qualitative reviews: she eldbaseline and she eld-relevance feedback. 3.1
she eld-baseline
A baseline query was formed using the review title and terms extracted from
the Boolean query. Studies were ranked using this query and BM25 [
Studies in the training dataset were partitioned into relevant and irrelevant sets depending upon whether they were included in the systematic review. The three lexical statistics described in Section 2 were calculated and the terms with the highest scores added to the baseline query. The number of terms added was determined from experiments conducted using the training data2. The studies in the test dataset are ranked by matching terms from the expanded queries against those in the abstracts using BM25. Note that sets of additional terms were generated for each review type separately, i.e. once for DTA reviews and again for Intervention reviews. 3.3
she eld-relevance feedback No training data was provided for Prognosis and Qualitative reviews. Consequently it was not possible to apply the lexical statistics and a relevance feedback approach was used instead. Studies in the test dataset are ranked using BM25 and the top 5% extracted. The Chi-Squared statistic was then applied using relevance judgements to divide the studies into relevant and irrelevant sets. The top 20 terms were added to the query which is then used to re-rank the remaining 95% of the studies. 2 For DTA reviews, 10 terms were added when the Log-Likelihood and Chi-Squared lexical statistics were used and 50 when Odds-Ratio was used. For Intervention reviews, 20 terms were added for the Log-Likelihood statistic, 5 for Chi-Squared and 50 for Odds-Ratio.
Results for Intervention reviews are shown in Table 3. Log-Likelihood performs strongly, with the best results for all metrics using content level judgements and using MAP for abstract level judgements. However, it is noteworthy that the baseline approach achieves the best performance using the WSS@95 metric and abstract level judgements.
Tables 4 and 5 show the results produced by applying the baseline and relevance-feedback approaches to the Qualitative and Prognosis reviews, respectively. The use of relevance feedback produced a slight improvement in the results, particularly MAP. The modest level of improvement may be down to the small number of relevant studies found in the top 5% of the ranked documents. For example, for the Qualitative review (CD011558) only two of the 2,168 (0.09%) studies are relevant. 3 https://github.com/leifos/tar
This paper presented the University of She eld's approach to CLEF2019 task 2 subtask 2. Studies were ranked by supplementing terms extracted from the Boolean query with ones speci c to the review type. Three lexical statistics were used to generate these list of supplementary terms. Results demonstrated that adding these additional terms improved performance although there was no clear picture of which lexical statistics was most e ective.