This paper details methods, results and analysis of the CLEF 2015 eHealth Evaluation Lab, Task 2. This task investigates the e ectiveness of web search engines in providing access to medical information with the aim of fostering advances in the development of these technologies. The problem considered in this year's task was to retrieve web pages to support information needs of health consumers that are confronted with a sign, symptom or condition and that seek information through a search engine, with the aim to understand which condition they may have. As part of this evaluation exercise, 66 query topics were created by potential users based on images and videos of conditions. Topics were rst created in English and then translated into a number of other languages. A total of 97 runs by 12 di erent teams were submitted for the English query topics; one team submitted 70 runs for the multilingual topics.
This document reports on the CLEF 2015 eHealth Evaluation Lab, Task 2. The
task investigated the problem of retrieving web pages to support information
needs of health consumers (including their next-of-kin) that are confronted with
a sign, symptom or condition and that use a search engine to seek
understanding about which condition they may have. Task 2 has been developed within
the CLEF 2015 eHealth Evaluation Lab, which aims to foster the development
? In alphabetical order, JP, GZ led Task 2; LG, LK, AH, ML & PP were on the Task
2 organising committee.
of approaches to support patients, their next-of-kin, and clinical sta in
understanding, accessing and authoring health information [
The use of the Web as source of health-related information is a wide-spread
phenomena. Search engines are commonly used as a means to access health
information available online [
Searching for self-diagnosis information is another important type of health
information seeking activity [
The remainder of this paper is structured as follows: Section 2 details the task, the document collection, topics, baselines, pooling strategy, and evaluation metrics; Section 3 presents the participants' approaches, while Section 4 presents their results; Section 5 concludes the paper. 2 2.1
The goal of the task is to design systems which improve health search, especially in the case of search for self-diagnosis information. The dataset provided to participants is comprised of a document collection, topics in various languages, and the corresponding relevance information. The collection was provided to participants after signing an agreement, through the PhysioNet website7. 7 http://physionet.org/
Participating teams were asked to submit up to ten runs for the English queries, and an additional ten runs for each of the multilingual query languages. Teams were required to number runs such as that run 1 was a baseline run for the team; other runs were numbered from 2 to 10, with lower numbers indicating higher priority for selection of documents to contribute to the assessment pool (i.e. run 2 was considered of higher priority than run 3). 2.2
The document collection provided in the CLEF 2014 eHealth Lab Task 3 [
Queries were manually built with the following process: images and videos
related to medical symptoms were shown to users, who were then asked which
queries they would issue to a web search engine if they, or their next-of-kin, were
exhibiting such symptoms. Thus, these queries aimed to simulate the situation
of health consumers seeking information to understand symptoms or conditions
they may be a ected by; this is achieved using imaginary or video stimuli. This
methodology for eliciting circumlocutory, self-diagnosis queries was shown to be
e ective by Stanton et al. [
Following the methodology in [
Please provide 3 search queries that you would issue to find out what is wrong. Instructions: * You must provide 3 distinct search queries. * The search queries must relate to what you see below.
A total of 266 possible unique queries were collected; of these, 67 queries (22 conditions with 3 queries and 1 condition with 1 query) were selected to be used in this year's task. Queries were selected by randomly picking one query per condition (we called this the pivot query), and then manually selecting the query that appeared most similar (called most ) and the one that appeared least similar (called least ) to the pivot query. Candidates for the most and least queries were identi ed independently by three organisers and then majority voting was used to establish which queries should be selected. This set of queries formed the English query set distributed to participants to collect runs.
In addition, we developed translations of this query set into Arabic (AR), Czech (CS), German (DE), Farsi (FA), French (FR), Italian (IT) and Portuguese (PT); these formed the multilingual query sets which were made available to participants for submission of multilingual runs. Queries were translated by medical experts available at the organisers institutions.
After the query set was released, numbered qtest1-qtest67, one typo was found in query qtest62, which could compromise the translations. In order to keep consistency between the English query and all translations made by the experts, qtest62 was excluded. Thus, the nal query set used in the CLEF 2015 10 With exception of one condition, for which only one query could be generated. 11 Note that additional instructions were given to volunteers at the start and end of the task, including training and de-brie ng. eHealth Lab Task 2 for both English and multilingual queries consisted of 66 queries.
An example of one of the query topics generated from the image shown in
Figure 1 is provided in Figure 2. To develop their submissions, participants were
only given the query eld of each query topic, that is, teams were unaware of
the query type (pivot, most, least), the target condition and the image or video
that was shown to assessors to collect queries.
<topics>
...
<top>
<num>qtest.23</num>
<query>red bloodshot eyes</query>
<disease>non-ulcerative sterile keratitis</disease>
<type>most</type>
<query_index>22</query_index>
</top>
...
</topics>
Relevance assessments were collected by pooling participants' submitted runs
as well as baseline runs (see below for a description of pooling methodology
and baseline runs). Assessment was performed by ve paid medical students
employed at the Medizinische Universitat Graz (Austria); assessors used
Relevation! [
Target symptoms or conditions were used to provide the relevance criteria assessors should judge against; for example for query qtest1 { \many red marks on legs after traveling from US" (the condition used for generating the query was \Rocky Mountain spotted fever (RMSF)"), the relevance criterion read \Relevant documents should contain information allowing the user to understand that they have Rocky Mountain spotted fever (RMSF).". Relevance assessments were provided on a three point scale: 0, Not Relevant; 1, Somewhat Relevant; 2, Highly Relevant.
Along with relevance assessments, readability judgements were also collected
for the assessment pool. The notion of readability and understandability of
information is of important concern when retrieving information for health
consumers [
Readability assessments were collected by asking the assessors whether they believed a patient would understand the retrieved document. Assessments were provided on a four point scale: 0, \It is very technical and di cult to read and understand"; 1, \It is somewhat technical and di cult to read and understand"; 2, \It is somewhat easy to read and understand"; 3, \It is very easy to read and understand". 2.5
A di erent set of 5 queries was released to participants as example queries (called
training ) to help develop their systems (both in English and the other
considered languages). These queries were released together with associated relevance
assessments, obtained by evaluating a pool of 112 documents retrieved by a set
of baseline retrieval systems (TF-IDF, BM25, Language Model with Dirichlet
smoothing as implemented in Terrier [
The organisers generated baseline runs using BM25, TF-IDF and Language Model with Dirichlet smoothing, as well as a set of benchmark systems that ranked documents by estimating both (topical) relevance and readability. Table 1 shows the 13 baseline systems created, 7 of them took into consideration some estimation of text readability. No baselines were created for the multilingual queries.
The rst 6 baselines, named baseline1 -6, were created using either Xapian or
Lucene as retrieval toolkit. We vary the retrieval model used, including BM25
(with parameters k1 = 1; k2 = 0; k3 = 1 and b = 0:5) in baseline1, Vector Space
Model (VSM) with TF-IDF weighting (the default Lucene implementation) in
baseline2, and Language Model (LM), with Dirichlet smoothing with = 2; 000
in baseline3. Our preliminary runs based on the 2014 topics showed that
removing HTML tags from documents in this collection could lead to higher retrieval
e ectiveness when using BM25 and LM. We used the python package
BeautifulSoap (BS4)12 to parse the HTML les and remove HTML tags. Note that it
12 https://pypi.python.org/pypi/beautifulsoup4
does not remove the boilerplate from the HTML (such as headers or navigation
menus), being one of the simplest approaches to clean a HTML page and prepare
it to serve as the input of readability formulas [
For the baseline runs that take into account readability estimations, we used
two well-known automatic readability measures: the Dale-Chall measure [
is a hyperparameter and 0 1 (in readability1 2. Direct Multiplication: Score(d) = Rel(doc) Read(d)
Rel(doc) 3. Inverse Logarithm: Score(d) = log(Read(d)) ) Read(d), where is 0.9)
Table 1 shows the settings of retrieval model, HTML processing, readability measure and query expansion or score combination method that were considered to produce the 7 readability baselines used in the task. 2.7
In Task 2, for each query, the top 10 documents returned in runs 1, 2 and 3 produced by the participants14 were pooled to form the relevance assessment pool. In addition, the baseline runs developed by the organisers were also pooled with the same methodology used for participants runs. A pool depth of 10 documents was chosen because this task resembles web-based search, where often users consider only the rst page of results (that is, the rst 10 results). Thus, this pooling methodology allowed a full evaluation of the top 10 results for the 3 submissions with top priority for each participating team. The pooling of more submissions or a deeper pool, although preferable, was ruled out because of the limited availability of resources for document relevance assessment. 13 https://pypi.python.org/pypi/ReadabilityCalculator/ 14 With the exclusion of multilingual submissions, for which runs were not pooled due to the larger assessment e ort pooling these runs would have required. Note that only one team submitted multilingual runs.
Because only one team submitted runs for the multilingual queries and only limited relevance assessment capabilities were available through the paid medical students that performed the assessment of submissions for the English queries, multilingual runs were not considered when forming the pools for relevance assessments. However, additional relevance assessments were sought through the team that participated in the multilingual task: they were thus asked to perform a self-assessment of the submissions they produced. A new pool of documents was sampled with the same pooling methodology used for English runs (see the previous section). Documents that were already judged by the o cial assessors were excluded from the pool with the aim to limit the additional relevance assessment e ort required by the team.
Additional relevance assessments for the multilingual runs were then
performed by a medical doctor (native Czech speaker with uent English)
associated with Team CUNI [
System evaluation was conducted using precision at 10 (p@10) and
normalised discounted cumulative gain [
A separate evaluation was conducted using both relevance assessments and
readability assessments following the methods in [
The RBP parameter which attempts to model user behaviour15 (RBP
persistence parameter) was set to 0.8 for all variations of this measure, following
the ndings of Park and Zhang [
To compute uRBP, readability assessments were mapped to binary classes, with assessments 0 and 1 (indicating low readability) mapped to value 0 and assessments 2 and 3 (indicating high readability) mapped to value 1. Then, uRBP (up to rank K) was calculated according to
K ) X
k=1 uRBP = (1 k 1r(k)u(k) (1) where r(k) is the standard RBP gain function that is 1 if the document at rank k is relevant and 0 otherwise; u(k) is a similar gain function but for the readability dimension and is 1 if the document at k is readable (binary class 1), and zero otherwise (binary class 0).
To compute uRBPgr, i.e. the graded version of uRBP, each readability label was mapped to a di erent gain value. Speci cally, label 0 was assigned gain 0 (least readability, no gain), label 1 gain 0.4, label 2 gain 0.8 and label 3 gain 1 (highest readability, full gain). Thus, a document that is somewhat di cult to read does still generate a gain, which is half the gain generated by a document 15 High values of users.
representing persistent users, low values representing impatient that is somewhat easy to read. These gains are then used to evaluate the function u(k) in Equation 1 to obtain uRBPgr.
The readability-biased evaluation was performed using ubire16, which is publicly available for download. 3
This year, 52 groups registered for the task on the web site, 27 got access to the
data and 12 submitted any run for task 2. The groups are from 9 countries in 4
continents as listed in Table 2. 7 out of the 12 participants had never participated
in this task before.
Team CUNI [
Team FDUSGInfo explored query expansion methods that use a range of knowledge resources to improve the e ectiveness of a statistical Language Model baseline. The knowledge sources that have been considering for drawing expansion terms are MeSH and Freebase. Di erent techniques were evaluated to select the expansion terms, including manual term selection. Team FDUSGInfo, unfortunately, did not submit their working notes and thus the details of their methods are unknown.
Team GRIUM [
Team KISTI [
Team KUCS [
Team LIMSI [
Team Miracl [
Team HCMUS [
Team UBML [
Team USST [
Team YorkU [
A total of 8,713 documents were assessed. Of these, 6,741 (77.4%) were assessed as irrelevant (0), 1,515 (17.4%) as somewhat relevant (1), 457 (5.2%) as highly relevant (2). For readability assessments, the recorded distribution was: 1,145 (13.1%) documents assessed as di cult (0), 1,568 (18.0%) as somewhat di cult (1), 2,769 (31.8%) as somewhat easy (2), and 3,231 (37.1%) as easy (3).
Table 3 details the coverage of the relevance assessments with respect to the participant submissions, averaged over the whole query set. While in theory all runs 1-3 should have full coverage (100%), in practice a small portion of documents included in the relevance assessment pool were left unjudged because the documents were not in the collection (participants provided an invalid document identi er) or the page failed to render in the relevance assessment toolkit (for example because the page contained redirect scripts or other scripts that were not executed within Relevation17). Overall, the mean coverage for runs 1-3 was above 99%, with only run 3 from team KUCS being sensibly below this value. This suggests that the retrieval e ectiveness for runs 1-3 can be reliably measured. The coverage beyond submissions 3 is lower but always above 90% (and the mean above 95%); this suggest that the evaluation of runs 4-10 in terms of precision at 10 may be underestimated of an average maximum of 0.05 points over the whole query set.
Table 4 details the coverage of relevance assessment for the multilingual runs. As mentioned in Section 2.8, due to limited relevance assessment availability, only the English runs were considered when forming the pool for relevance assessment. The coverage of these relevance assessments with respect to the top 10 documents ranked by each participants' submissions is shown in the columns marked as Eng. in Table 4. An additional document pool, made using only documents in runs13 of multilingual submissions, was created to further increase the coverage of multilingual submissions; the coverage of the union of the original assessments and these additional ones (referred to as merged ) is shown in the columns marked as Merged in Table 4 for the multilingual runs. The merged set of relevance assessments was enough to provide a fairly high coverage for all runs, including those not in the pool (i.e., runs beyond number 3), with a minimal coverage of 97%; this is likely because only one team submitted runs for the multilingual challenge, thus providing only minimal variation in terms of top retrieval results.
Table 5 reports the evaluation of the participants submissions and the organisers baselines based on P@10 and nDCG@10 for English queries. The evaluation based on RBP and the readability measures is reported in Table 6.
Most of the approaches developed by team ECNU obtain signi cantly higher values of P@10 and nDCG@10 compared to the other participants, demonstrat17 Note that before the relevance assessment exercise started, we removed the majority of scripts from the pooled pages to avoid this problem. ing about 40% increase in e ectiveness in their best run compared to the runnerup team (KISTI). The best submission developed by the organisers and based on both relevance and readability estimates has been proved di cult to outperform by most teams (only 4 out of 12 teams obtained higher e ectiveness). The pooling methodology does not appear to have signi cantly in uenced the evaluation of non-pooled submissions, as demonstrated by the fact that the best runs of some teams are not those that were fully pooled (e.g. team KISTI, team CUNI, team GRIUM).
There are no large di erences between system rankings produced using P@10 or nDCG@10 as evaluation measure (Kendall = 0.88). This is unlike when readability is also considered in the evaluation (the Kendall between system rankings obtained with P@10 or uRBP is 0.76). In this latter case, while ECNU's submissions are con rmed to be the most e ective, there are large variations in system rankings when compared to those obtained considering relevance judgements only. In particular, runs from team KISTI, which in the relevance-based evaluation were ranked among the top 20 runs, are not performing as well when considering also readability, with their top run (KISTI EN RUN.7) being ranked only 37th according to uRBP.
The following considerations could be drawn when comparing the di erent methods employed by the participating teams. Query expansion is found to often improve results. In particular, team ECNU obtained the highest e ectiveness among the systems that took part in this task; this was achieved when query expansion terms are mined from Google search results returned for the original queries (ECNU EN Run.3). This approach indeed obtained higher e ectiveness compared to learning-to-rank alternatives (ECNU EN Run.10). The results of team UBML show that query expansion using the Bose-Einstein model 1 and the local collection works better than other query expansion methods and external collections. Team USST also found that query expansion was e ective to improve results, however they found that the Bose-Einstein models did not provide improvements over their baseline, while the Kullback-Liebler Divergence based query expansion provided minor improvements. Health-speci c query expansion methods based on the UMLS were shown to be e ective above common baselines and other considered query expansion methods by Team LIMSI and GRIUM (this form of query expansion was the only one that delivered higher e ectiveness than their baseline).Team KISTI found that the combination of concept-based document centrality (CBDC) and cluster- based external expansion model (CBEEM) improved the results best. Few teams did not observe improvements over their baselines; this was the case for teams KUCS, Miracl, FDUSGInfo and HCMUS.
Tables 7 and 8 report the evaluation of the multilingual submissions based on P@10 and nDCG@10; results are reported with respect to both the original qrels (obtained by sampling English runs only) and the additional qrels (obtained by sampling also multilingual runs, but using a di erent set of assessors); see Section 2.8 for details about the di erence between these relevance assessments. Only one team (CUNI) participated in the multilingual task; they also submitted to the English-based task and thus it is possible to discuss the e ectiveness of their retrieval system when answering multilingual queries compared to that achieved when answering English queries.
The evaluation based on the original qrels allows us to compare multilingual runs directly with English runs. Note that the original relevance assessments exhibit a level of coverage for the multilingual runs that is similar to those obtained for English submissions numbered 4-10. The evaluation based on the additional qrels (merged) allows analysis of the multilingual runs using the same pooling method used for English runs; thus submissions 1-3 for the multilingual runs can be directly compared to the corresponding English ones, at the net of differences in expertise, sensibility and systematic errors between the paid medical assessors and the volunteer, student self-assessor used to gather judgements for the multilingual runs.
When only multilingual submissions are considered, it can be observed that there is not a language in which CUNI's system is more e ective: e.g. submissions that considered Italian queries are among the best performing with original assessments and are the best performing with the additional assessments, but di erences in e ectiveness among top runs for di erent languages are not statistically signi cant. However, it can be observed that none of CUNI's submissions that addressed queries expressed in not European languages (Farsi and Arabic) are among the top ranked systems, regardless of the type of relevance assessments.
The use of the additional relevance assessments naturally translates in observing increased retrieval e ectiveness across all multilingual runs (because some of the documents in the top 10 ranks that were not assessed, and thus irrelevant, in the original assessments may have been marked as relevant in the additional assessments). However, a noteworthy observation is that the majority of the most e ective runs according to the additional assessments are those that were not fully sampled to form the relevance assessment pools (i.e. runs 4-10, as opposed to the pooled runs 1-3).
When the submissions of team CUNI are compared across English and
multilingual queries, it is possible to observe that the best multilingual runs do not
outperform English runs (unlike when the same comparison was instructed in
the 2014 task [
This paper has described methods, results and analysis of the CLEF 2015 eHealth Evaluation Lab, Task 2. The task considered the problem of retrieving web pages for people seeking health information regarding unknown conditions or symptoms. 12 teams participated in the task; the results have shown that query expansion plays an important role in improving search e ectiveness. The best results were achieved by a query expansion method that mined the top results from the Google search engine. Despite the improvements over the organisers' baselines achieved by some teams, further work is needed to sensibly improve search in this context, as only about half of the top 10 results retrieved by the best system were found to be relevant.
As a by-product of this evaluation exercise, the task contributes to the research community a collection with associated assessments and evaluation framework (including readability evaluation) that can be used to evaluate the e ectiveness of retrieval methods for health information seeking on the web. Queries, assessments and participants runs are publicly available at http://github.com/ CLEFeHealth/CLEFeHealth2015Task2.
This task has been supported in part by the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement no257528 (KHRESMOI), by Horizon 2020 program (H2020-ICT-2014-1) under grant agreement no 644753 (KCONNECT), by the Austrian Science Fund (FWF) project no I1094-N23 (MUCKE), and by the Czech Science Foundation (grant number P103/12/G084). We acknowledge the time of the people involved in the translation and relevance assessment tasks, in special we want to thank Dr. Johannes Bernhardt-Melischnig (Medizinische Universitat Graz) for coordinating the recruitment and management of the paid medical students that participated in the relevance assessment exercise.
R Run Name
Run Name 1 CUNI DE Run10 0.2985 2 CUNI DE Run7 0.2970 3 CUNI FR Run10 0.2833 4 CUNI FR Run7 0.2773 5 CUNI IT Run10 0.2758 6 CUNI IT Run1 0.2652 7 CUNI IT Run4 0.2621 8 CUNI PT Run6 0.2530 9 CUNI PT Run8 0.2515 10 CUNI DE Run8 0.2500 10 CUNI FR Run9 0.2500 12 CUNI FR Run8 0.2455 12 CUNI IT Run6 0.2455 14 CUNI DE Run9 0.2409 14 CUNI PT Run10 0.2409 16 CUNI IT Run2 0.2394 17 CUNI IT Run3 0.2348 17 CUNI PT Run7 0.2348 19 CUNI IT Run8 0.2333 20 CUNI CS Run10 0.2303 20 CUNI FA Run10 0.2303 20 CUNI PT Run1 0.2303 20 CUNI PT Run5 0.2303 24 CUNI PT Run4 0.2288 25 CUNI AR Run10 0.2273 25 CUNI FA Run4 0.2273 25 CUNI IT Run9 0.2273 28 CUNI FA Run1 0.2258 29 CUNI CS Run7 0.2242 30 CUNI FA Run3 0.2227 30 CUNI FA Run5 0.2227 32 CUNI AR Run5 0.2197 32 CUNI AR Run6 0.2197 34 CUNI FA Run2 0.2182 34 CUNI FA Run8 0.2182 1 CUNI IT Run10 0.3727 1 CUNI IT Run4 0.3727 3 CUNI IT Run1 0.3712 4 CUNI FR Run10 0.3682 4 CUNI FR Run7 0.3682 6 CUNI IT Run6 0.3606 7 CUNI PT Run2 0.3576 8 CUNI DE Run10 0.3561 9 CUNI DE Run7 0.3545 10 CUNI IT Run8 0.3515 11 CUNI PT Run1 0.3500 12 CUNI PT Run4 0.3485 13 CUNI IT Run2 0.3424 14 CUNI IT Run3 0.3394 15 CUNI PT Run10 0.3379 16 CUNI PT Run3 0.3364 17 CUNI FA Run10 0.3333 17 CUNI FR Run9 0.3333 17 CUNI PT Run6 0.3333 20 CUNI CS Run1 0.3318 20 CUNI CS Run7 0.3318 20 CUNI IT Run5 0.3318 20 CUNI IT Run7 0.3318 20 CUNI PT Run8 0.3318 25 CUNI CS Run10 0.3288 26 CUNI FA Run4 0.3273 26 CUNI FR Run3 0.3273 28 CUNI FA Run1 0.3258 29 CUNI FA Run3 0.3242 30 CUNI FA Run2 0.3227 31 CUNI FR Run4 0.3182 31 CUNI FR Run8 0.3182 31 CUNI PT Run5 0.3182 34 CUNI FR Run1 0.3121 35 CUNI IT Run9 0.3106 0.2498 0.2539 0.2661 0.2259 0.2343 0.2206 0.2669 0.2672 0.2613 0.2556 0.2569 0.2544 0.2397 0.2255 0.2426 0.2392 0.2493 0.2446 0.2122 0.2504 0.2256 0.2070 0.2327 0.2255 0.2403 0.2058 0.2101 0.2039 0.2178 0.2237 0.2211 0.2261 0.1984 0.1954 0.1811