This paper describes the methods used for our participation to the CLEF (Conference and Labs of the Evaluation Forum) eHealth 2016 Task 3: Patient-Centered Information Retrieval, IRTask 1: ad-hoc search. For this participation, we evaluate the e ectiveness of three different retrieval strategies. In particular, we deploy PL2 with Boolean Fallback as our baseline system. In another approach, we deploy the collection enrichment approach, where the original query is expanded with additional terms from an external collection (collection not being searched). To deliver an e ective ranking, we combine the rst two rankers using data fusion techniques.
In this paper, we describe the methods used for our participation to the CLEF
eHealth 2016 Task 3: Patient-Centered Information Retrieval, IRTask 1: ad-hoc
search. Detailed task description is available in the overview paper of Task 3 [
This paper is structured as follows. Section 2 contains a background on algorithms used. Section 3 describes the 3 runs submitted by team ub-botswana. In Section 4, we describe the experimental environment. Section 5 reports our results.
In this section, we begin by presenting a brief but essential background on the di erent algorithms used in our experimental investigation and evaluation. We start by describing the PL2 term weighting model in Section 2.1. In Section 2.2, we describe the Bose-Einstein 1 (Bo1) model for query expansion, followed by a description of the CombSUM for data fusion in Section 2.3. A description of the runs is provided in Section 3. 2.1
For our baseline system and all our experimental investigation and evaluation, we used the PL2 term weighting model to score and rank medical documents. For a given query Q, the relevance score of a document d based on the PL2 Divergence from Randomness (DFR) term weighting model is expressed as follows [8]: scoreP L2(d; Q) = Pt2Q tfqtnf+n1 tf n log2 tfn + ( tf n) log2 e + 0:5 log2(2 tf n) (1) where score(d; Q) is the relevance score of a document d for a given query Q.
= tNfc is the mean and variance of a Poisson distribution, tf c is the frequency of the term t in the collection C while N is the number of documents in the collection. The normalised query term frequency is given by qtf n = qtqftmfax , where qtfmax is the maximum query term frequency among the query terms and qtf is the query term frequency. tf n is the Normalisation 2 of the term frequency tf of the term t in a document d and is expressed as: (2) (3) tf n = tf log2 1 + b
; (b > 0) avg l l In the above expression, l is the length of the document d, avg l is the average document length in the collection and b is a hyper-parameter. 2.2
Bose-Einstein 1 (Bo1) Model for Query Expansion
In our experimental investiagtion and evaluation, we used the Terrier-4.0
Divergence from Randomness (DFR) Bose-Einstein 1 (Bo1) model to select the most
informative terms from the topmost documents after a rst pass document
ranking on an external collection. The DFR Bo1 model calculates the information
content of a term t in the top-ranked documents as follows [
Pn(t) w(t) = tf x log2
+ log2(1 + Pn(t)) tf c Pn(t) = (4)
N where Pn(t) is the probability of t in the whole collection, tf x is the frequency of the query term in the top x ranked documents, tf c is the frequency of the term t in the collection, and N is the number of documents in the collection. 2.3
In another approach, we used data fusion to combine document rankings of two di erent rankers. In particular, we used CombSUM, which is a data fusion technique that sums the scores of each document in the constituent ranking based on the following equation: score(d; Q) = X scorer(d; Q) r2R (5) where r is a ranking in R, R being the set of ranking being considered. scorer(d; Q) is the score of document d for query Q in ranking r. If a document d is not in ranking r, the scorer(d; Q) = 0. Hence, a document scored highly in many rankings is likely to be scored highly in the nal ranking. In contrast, a document with low scores, or that is present in less rankings is less likely to end up highly in the nal ranking. 3
Description of the Di erent Runs
ub-botswana EN Run1: This is the baseline system. We used PL2 Divergence
from Randomness term weighting model in Terrier-4.0 IR platform to score and
rank the documents in the ClueWeb 12 B13 document collection. In order to
improve the retrieval e ectiveness of our system, we deployed a boolean fallback
score modi er. With this score modi er, if any of the retrieved documents
contain all undecorated query terms (ie query terms without any operators), then
we remove from the result set documents that do not contain all undecorated
query terms. Otherwise, we do nothing. The intuition is that when we combine
this ranker with another ranker using any data fusion technique, documents
retrieved and ranked by this score modi er are likely to be ranked higher when
they appear in both rankings.
ub-botswana EN Run2: We used the baseline system without boolean fallback.
As improvement, we used the collection enrichment approach [
FAQ Retrieval Platform: For all our experimental evaluation, we used Terrier4.01[7], an open source Information Retrieval (IR) platform. All the documents (ClueWeb 12 B13) used in this study were rst pre-processed before indexing and this involved tokenising the text and stemming each token using the full Porter stemming algorithm [9]. Stopword removal was enabled and we used Terrier stopword list. The hyper-parameter for PL2 was set to its default value of b = 1.0. 5
These working notes were compiled and submitted before the relevance judgments were released. However, results for our di erent runs and how our approaches performed as compared to other participating teams are presented in the Task 3: Patient-Centered Informational Retrieval overview paper entitled \The IR Task at the CLEF eHealth Evaluation Lab 2016: User-centered Health Information Retrieval" [12]. 1 http://terrier.org/ 7. I. Ounis, G. Amati, Plachouras V., B. He, C. Macdonald, and Johnson. Terrier Information Retrieval Platform. In Proceedings of the 27th European Conference on IR Research, volume 3408 of Lecture Notes in Computer Science, pages 517{519, Berlin, Heidelberg, 2005. Springer-Verlag. 8. Vassilis Plachouras and Iadh Ounis. Multinomial Randomness Models for Retrieval with Document Fields. In Proceedings of the 29th European Conference on IR Research, pages 28{39, Berlin, Heidelberg, 2007. Springer-Verlag. 9. M.F. Porter. An Algorithm for Su x Stripping. Readings in Information Retrieval, 14(3):313{316, 1997. 10. I. Stanton, S. Ieong, and N. Mishra. Circumlocution in Diagnostic Medical Queries.
In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval, pages 133{142. ACM, 2014. 11. G. Zuccon, B. Koopman, and J. Palotti. Diagnose This If You Can: On the Effectiveness of Search Engines in Finding Medical Self-Diagnosis Information. In Advances in Information Retrieval (ECIR 2015), pages 562{567. Springer, 2015. 12. G. Zuccon, J. Palotti, L. Goeuriot, L. Kelly, M. Lupu, P. Pecina, H. Mueller, J. Budaher, and A. Deacon. The IR Task at the CLEF eHealth Evaluation Lab 2016: User-centred Health Information Retrieval. In CLEF 2016 Evaluation Labs and Workshop: Online Working Notes. CEUR-WS, September 2016.