=Paper=
{{Paper
|id=Vol-1866/paper_124
|storemode=property
|title=SiS at CLEF 2017 eHealth TAR Task
|pdfUrl=https://ceur-ws.org/Vol-1866/paper_124.pdf
|volume=Vol-1866
|authors=Leif Azzopardi,Vassil Kalphov,Georgios Georgiadis
|dblpUrl=https://dblp.org/rec/conf/clef/AzzopardiKG17
}}
==SiS at CLEF 2017 eHealth TAR Task==
https://ceur-ws.org/Vol-1866/paper_124.pdf
SiS at CLEF 2017 eHealth TAR Task
Vassil Kalphov1 , Georgios Georgiadis1 , Leif Azzopardi1
vassil.kalphov.2013@uni.strath.ac.uk,
georgios.georgiadis.2013@uni.strath.ac.uk, and
Leif.Azzopardi@strath.ac.uk
University of Strathclyde, Glasgow, UK
Abstract. This paper presents Strathclyde iSchool’s (SiS) participation
in the Technological Assisted Reviews in Empirical Medicine Task. For
the ranking task, we explored two ways in which assistance to review-
ers could be provided during the assessment process: (i) topic models,
where we use Latent Dirichlet Allocation to identify topics within the set
of retrieved documents, ranking documents by the topic most likely to be
relevant and (ii) relevance feedback, where we use Rocchio’s algorithm to
update the query model for subsequent rounds of interaction. A third ap-
proach combines the topic and relevance feedback to quickly identify the
relevant abstracts. For the thresholding task, we apply a score threshold,
and exclude documents which did not exceed the threshold given BM25.
1 Introduction
CLEF 2017 introduced a new eHealth retrieval problem - that of providing tech-
nological assistance to reviewers of systematic reviews - where the goals of the
task were to explore how Information Retrieval techniques could be used to: (i)
identify relevant material more quickly in the ranking challenge and (ii) identify
when reviewers could stop processing documents in the thresholding challenge [3,
2]. During the review process, reviewers will routinely examine hundreds to thou-
sands of abstracts to decided if the document (and evidence it contains) could
be included in the systematic review that they are conducting [5]. Once they
have identified a subset of abstracts, which are potentially relevant, they exam-
ine the document’s contents to decide whether the document should be included
or excluded. The track focused on the first part, identifying potential relevant
documents during the, so called, screening phase.
In this work we considered two different approaches - one which uses topic
modelling and the other which uses relevance feedback. In selecting these ap-
proaches we thought that such techniques could be used in the following way.
For topic modelling, we envisaged that the download abstracts could be seman-
tically clustered - and the different clusters could be presented to the reviewer -
the reviewer could then start the review process by selecting a cluster that they
felt was most likely to contain the relevant documents. Since we did not have
recourse to reviewers, we explored a number of different ways to automatically
select the best cluster. For relevance feedback, we envisaged that as the reviewer
starts to examine documents, the query could be updated to bring back the next
most relevant documents, so that they would quickly find all the relevant mate-
rial as soon as possible. Obviously, if the aim is to reduce the workload of the
reviewers, then we need to be able to select a point where the reviewer can stop
assessing documents - however, this runs the risk of losing relevant documents.
To this end, we explore various heuristics to select the threshold such that we
minimize effort and maximise recall (but ideally obtain total recall).
2 Experimental Set-up
Data: Given the list of topic descriptions the PubMed IDs were extracted, and
a scripted fetched the Abstract and associated Metadata from the PubMed API.
From the topics, we extracted the title for each topic and use that as the query.
Indexing and Retrieval System: We used Lucene 6.2 to create a separate
index for each of the topic (where stop words were removed, no stemming was
applied). A Lucene Document was created where the following fields were index:
Title, Abstract, Author, and Publication Name. The baseline retrieval algorithm
we employed was fielded BM25 with standard parameters settings i.e. b = 0.75,
and equal weights between fields (denoted as BM25).
Relevance Feedback: We implemented Rocchio’s Algorithm [4] in Lucene -
where feedback was used to provide relevance information. In each round of
feedback, 30 documents were examined, and the query model updated, to pro-
vide a re-ranking of the subsequent documents. This was performed on the first
10%, 20%, etc of documents associated with the topic. Here, we only report
the 30% runs (AL30) as these generally performed the best and little change in
performance was observed on the training set using more feedback.
Topic Modelling: We used MALLET toolkit, and thus Latent Dirichlet Allo-
cation [1] to semantically cluster the documents within each topic. We set the
number of latent topics to 5, and α =. To rank the documents we selected one of
the latent topics zi , and ordered the documents by the probability p(zi |d|). In an
attempt to select the cluster that provides the best ranking, we took the BM25
ranking from above, and use the top 100 ranked documents as pseudo-relevance
feedback. Then we ranked each latent topic (given the ordering by p(zi |d|)) and
select the one with the highest overlap with BM25 (TMBM).
Combined: Since our topic modelling approaches do not use feedback, we de-
cided to see whether we could start the process of relevance feedback using the
topic modelling run, and refine the query model accordingly. Thus, we selected
the best performing method TMAL, given the training data, and used this as
starting point for the active learning. Again we considered obtained feedback for
the first 10%,20% and 30% of the documents per topic.
Thresholded Runs To create thresholded runs, we took the BM25 run and
applied a simple score based threshold. Using Lucene the scores for a query
given BM25 are from 1.0 or greater, so we used thresholds 1.0, 1.5, 2.0 and
2.5. This led to a reasonable reduction in the number of documents without
sacrificing much recall.
3 Results and Discussion
Tables 1 and 3 report the performance on the Ranking Task, while Tables 2
and 4 report the performance on the Threshold Task. Our best performing run
on the Ranking Task, in terms of normalized area under the gain curve, was
AL30, which used 30% of the documents as feedback. From our results, it is
clear that the topic modelling approach, as we have employed it, has not lead
to significantly better improvements over the BM25 baseline. However, when
inspecting the individual topic based ranking, the most probably topic, was not
always the best performing topic. So we will direct more research into topic
selection. This is because, when formulating queries, reviewers could use topic
modelling to understand the space of documents retrieved, and then refine their
query further - and thus make savings a priori rather than have to trawl through
hundreds and thousands of documents. Another factor that could significantly
improve our results is that for these initial runs we used the title as the query,
as opposed to the boolean query provided within the topics. It is quite possible
that the more complex and verbose boolean queries could lead to a better initial
ranking - and so when used in conjunction with relevance feedback the relevant
items could be found sooner. We leave these directions for further work.
Table 1: Results on Training Data for Ranking Task
Run Random BM25 AL30 TMBM TMAL30
NumRels 2494 2494 2494 2494 2494
NumFeed 0 0 44939 0 44940
RelsFound 2494 2494 2494 2494 2494
AP 0.044 0.171 0.273 0.15 0.24
MinRel 6947 5174 3210 4221 2835
WSS100 0.05 0.25 0.33 0.30 0.37
Area 0.51 0.83 0.91 0.80 0.90
NCG10 0.07 0.30 0.41 0.30 0.41
NCG20 0.18 0.51 0.73 0.57 0.73
NCG30 0.29 0.67 0.86 0.74 0.86
TotalCost 7453 7453 11947 7453 11947
TotalCostUniform 7453 7453 11947 7453 11947
TotalCostWeighted 7453 7453 11947 7453 11947
loss er 0.38 0.38 0.38 0.38 0.38
loss r 0.0 0.0 0.0 0.0 0.0
loss e 0.38 0.38 0.38 0.38 0.38
References
1. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. Journal of Machine
Learning Research 3, 993–1022 (2003)
Table 2: Results on Training Data for Threshold Task
Run BM25 T1 T1.5 T2 T2.5
NumRels 2494 2949 2494 2494 2494
NumFeed 0 0 0 0
RelsFound 2494 2439 2419 2318 2187
AP 0.17 0.171 0.17 0.17 0.17
MinRel 5174 4531 3930 3453 3236
WSS100 0.25 0.23 0.22 0.19 0.19
Area 0.83 0.83 0.83 0.82 0.80
NCG10 0.30 0.30 0.30 0.30 0.30
NCG20 0.51 0.51 0.51 0.51 0.51
NCG30 0.67 0.67 0.67 0.67 0.67
TotalCost 7453 6010 5391 4700 4144
TotalCostWeighted 7453 6811 6978 6911 6985
TotalCostUniform 7453 6049 5482 5008 4714
loss er 0.38 0.30 0.24 0.20 0.16
loss r 0.0 0.0 0.001 0.006 0.016
loss e 0.38 0.29 0.24 0.19 0.15
Table 3: Results on Test Data for Ranking Task
Run Random BM25 AL30 TMBM TMAL30
NumRels 1857 1857 1857 1857 1857
NumFeed 0 0 35730 0 35432
RelsFound 1857 1857 1857 1857 1857
AP 0.05 0.17 0.22 0.12 0.16
MinRel 3722 2851 2290 3124 2305
WSS100 0.04 0.29 0.41 0.27 0.40
Area 0.48 0.81 0.86 0.73 0.84
NCG10 0.09 0.45 0.62 0.31 0.54
NCG20 0.19 0.65 0.79 0.55 0.77
NCG30 0.28 0.75 0.88 0.68 0.86
TotalCost 3918 3918 6300 3918 6280
TotalCostUniform 3918 3918 6300 3918 6280
TotalCostWeighted 3918 3918 6300 3918 6280
loss er 0.54 0.54 0.54 0.54 0.54
loss r 0.00 0.00 0.00 0.00 0.00
loss e 0.54 0.54 0.54 0.54 0.54
2. Goeuriot, L., Kelly, L., Suominen, H., Névéol, A., Robert, A., Kanoulas, E., Spijker,
R., Palotti, J., Zuccon, G.: CLEF 2017 eHealth evaluation lab overview. In: CLEF
2017 - 8th Conference and Labs of the Evaluation Forum, Lecture Notes in Computer
Science (LNCS). Springer (September 2017)
3. Kanoulas, E., Li, D., Azzopardi, L., Spijker, R.: echnologically assisted reviews in
empirical medicine. In: CLEF 2017 Evaluation Labs and Workshop: Online Working
Notes, CEUR-WS (2017)
4. Rocchio, J.J.: Relevance feedback in information retrieval (1971)
Table 4: Results on Test Data for Threshold Task
Run BM25 T1 T1.5 T2 T2.5
NumRels 1857 1857 1857 1857 1857
NumFeed 0 0 0 0 0
RelsFound 1857 1828 1809 1784 1758
AP 0.17 0.17 0.17 0.17 0.17
MinRel 2851 2503 2333 2068 1877
WSS100 0.29 28 0.27 0.23 0.22
Area 0.81 0.81 0.80 0.80 0.79
NCG10 0.45 0.45 0.45 0.45 0.45
NCG20 0.65 0.65 0.65 0.65 0.65
NCG30 0.75 0.75 0.75 0.75 0.75
TotalCost 3918 3435 3165 2824 2536
TotalCostUniform 3918 3786 3865 3748 3902
TotalCostWeighted 3918 3454 3280 3117 2905
loss er 0.54 0.54 0.38 0.33 0.27
loss r 0.00 0.001 0.005 0.01 0.014
loss e 0.54 0.54 0.38 0.32 0.26
5. Shemilt, Khan, Park, Thomas: Use of cost-effectiveness analysis to compare the
efficiency of study identification methods in systematic reviews. Systematic reviews
(2016)