CLEF 2018 eHealth Consumer Health Search task aims to investigate the e ectiveness of the information retrieval systems in providing health information to common health consumers. Compared to previous years, this year's task includes ve subtasks and adopts new data corpus and set of queries. This paper presents the work of University of Evora participating in two subtasks: IRtask-1 and IRtask-2. It explores the use of learning to rank techniques as well as query expansion approaches. A number of eld based features are used for training a learning to rank model and a medical concept model proposed in previous work is re-employed for this year's new task. Word vectors and UMLS are used as query expansion sources. Four runs were submitted to each task accordingly.
CLEF 2018 eHealth Consumer Health Search (CHS) task is a continuation of
the previous CLEF eHealth information retrieval (IR) tasks that started on
2013 [
This paper describes the University of Evora (UEvora) approach to CLEF 2018 eHealth CHS subtasks IRTask-1 and IRtask-2. While IRTask-1 is a standard ad-hoc search that aims at retrieving relevant information to people seeking health advice on the web, IRTask-2 is a personalized search task. This task develops on top of the IRTask-1 and aims to personalize the retrieved list of search results to match user expertise.
The following questions were investigated by conducting experiments on
CLEF 2018 eHealth CHS Task:
1. How does a model learned from data based on 2016 and 2017 CLEF eHealth
IR task [
To answer the questions proposed in the previous section, di erent approaches
were employed in this work. To answer the rst question, learning to rank
techniques were used: a number of features were explored and assessments results
from 2016 and 2017 CLEF eHealth IR task were used for training a model. For
the second question, a pre-trained word vectors model was used as a source of
query expansion and the result is compared to the one retrieved with query
expansion using the domain speci c United Medical Language System (UMLS)
thesaurus. Finally, to tackle the third question, the model proposed in previous
work [
All queries were pre-processed with characters lower-casing, stop words removing and Porter Stemmer stemming. The default stop words list available in the IR platform Terrier 4.21 was used. 2.2
The assessment results from 2016 and 2017 CLEF eHealth IR task were employed and used to train a learning to rank model where a number of elds based features were explored in this work.
Features extracted. In this work, a simple group of features on di erent elds
were extracted for training a learning to rank model. Three information elds
were considered: title, H1 and else. One kind of the features were using a normal
weighting model on a single eld. BM25 and PL2 were used as the weighting
model, with each weighting model for every eld. Query independent features
and eld length were also taken into account. DL weighting model implementing
a simple document length was used [
Training a model. Di erent learning to rank algorithms were previously
explored (logistic regression, random forests, LambdaMART, AdaRank and
ListNet) and among all, LabmdaMART algorithm presented the best performance [
The assessment results from 2016 and 2017 CLEF eHealth IRtask [
A medical concepts model employed [
Two di erent expansion sources were used: UMLS and word vectors model.
For UMLS based expansion, selected terms were added to the original query and
the approach employed our previous work [
Besides the query expansion techniques, the pseudo relevance feedback was also tested for automatic expansion during retrieval process. The number of words was set to 10 and the number of top-ranked documents from which those words were extracted was set to 3 in Terrier 4.2 platform. 3
This section rst brie y presents the IR platform employed in this work, the dataset and queries for the task, as well as the evaluation measures used for the assessments. Next is the description of the techniques employed in our experiments. 3.1
Terrier platform version 4.2 was chosen as IR model of the system. The Okapi BM25 weighting model was used with all the parameters set to default values.
3.2
The dataset used in CLEF 2018 CHS task consisted of web pages acquired from the CommonCrawl. By submitting the task queries to the Microsoft Bing APIs repeatedly over a period of time, an initial list of websites used for acquisition was returned. Some URLs domains were excluded and a number of know reliable health websites were added3. Totally, a number of 1,903 sites were included in the list. 3.3
The basic query set used for CLEF 2018 CHS task consisted of 50 queries written
in English and were issued by the general public to the search service [
For IRtask-1, this basic set of 50 queries was used as the input to participating systems. An example of a query is shown in Figure 1.
IRtask-2 was based on IRtask-1 with 7 variations for each query. The rst 4 variations were issued by people with no medical backgrounds while the remaining were issued by medical experts. An example for IRtask-2 is shown in Figure 2. 3.4
Di erent evaluation measures were used for IRtask-1 and IRtask-2. For IRtask-1 they were: Normalized Discounted Cumulative Gain at depth 10 (NDCG@10), binary preference-based measure (Bpref) and Rank Biased Precision (RBP) [10].
For IRtask-2, uRBPgr with alpha was be used for the assessment. Based on RBP, uRBPgr is calculated as uRBP = (1
K ) X k 1r(k)u(k) k=1 (1) where the u(k) function is a graded gain function for the understandability dimension. The parameter attempts to model user behaviour and was set to
task-3-consumer-health-search/ 0.8. The r(k) function is the standard RBP gain function: the value is 1 if the document at rank k is relevant and 0 if it is irrelevant [10].
In IRTask 2, each topic has 7 query variations. A parameter alpha
capturing user expertise is used when evaluating results for query variations.
Setting alpha to increasing values, an increasing level of medical expertise across
the query variations is modeled [
Four experiments were conducted for each sub-task. Next paragraphs discuss the techniques used.
Runs for IRTask-1. UEvoraIRTask1Run1 is based on the Medical Concepts
Model presented in previous work [
For UEvoraIRTask1Run2 the same techniques are used but the expansion is performed with the pre-trained word vectors model.
UEvoraIRTask1Run3 uses a ranking model trained with the topical assessments from 2016 and 2017 CLEF eHealth IR task (see sub-section 2.2).
UEvoraIRTask1Run4 uses the similar techniques to UEvoraIRTask1Run3, yet using ve folders cross validation to obtain a learning to rank model. Runs for IRTask-2. Similar techniques and parameters were used for IRTask2. UEvoraIRTask2Run1 uses the same approach employed UEvoraIRTask1Run1; the queries and their variations were processed and issued to the retrieval system. UEvoraIRTask2Run2 uses a similar approach to UEvoraIRTask2Run1 with queries expanded using a pre-trained word vectors model and for UEvoraIRTask2Run3 the understandability assessments from 2016 and 2017 CLEF eHealth IR task were used for training a learning to rank model. Finally and for UEvoraIRTask2Run4, the same techniques of UEvoraTask2run3 were employed and a ve cross validation was done when training the learning to rank model. 3.6
The assessments for the CLEF eHealth 2018 IR tasks are still being conducted, so they are not available at this time. 4
This working note reports the UEvora team participation in two di erent tasks of CLEF 2018 eHealth CHS. A number of eld based features were explored while applying learning to rank techniques. Based on previous work, both UMLS and a word vector model were applied for performing query expansion.
As the future work, the methods proposed in this paper will be further analyzed: di erent learning to rank features will be explored and an ensemble algorithm will be investigated.
This work was supported by EACEA under the Erasmus Mundus Action 2, Strand 1 project LEADER { Links in Europe and Asia for engineering, eDucation, Enterprise and Research exchanges.