This paper presents the results of task 3 of the ShARe/CLEF eHealth Evaluation Lab 2013. This evaluation lab focuses on improving access to medical information on the web. The task objective was to investigate the e ect of using additional information such as the discharge summaries and external resources such as medical ontologies on the IR e ectiveness. The participants were allowed to submit up to seven runs, one mandatory run using no additional information or external resources, and three each using or not using discharge summaries.
The goal of the ShARe/CLEF (Cross-Language Evaluation Forum) eHealth
Evaluation Lab is to evaluate systems that support laypeople in searching for
and understanding their health information [
Searching for health advice is a common and important task performed by
individuals on the web. Nearly 70% of search engine users in the US have
conducted a web search for information about a speci c disease or health problem [
The growing importance of health IR has provided the motivation for a
number of evaluation campaigns focusing on health information. For example, the
TREC (Text REtrieval Conference) Medical Records Tracks aim at identifying
patient cohorts from medical reports to recruit for user studies [
The rest of this paper is organized as follows: Section 2 outlines the main evaluation campaigns on health IR. Section 3 describes the creation of the CLEF eHealth dataset, that is, the document collection, query generation, and relevance assessment. Section 4 presents the result sets and their evaluation and Section 5 the approaches used by task participants. Finally Section 6 concludes the paper.
Previous research has considered the information needs of individuals seeking
health advice on the web, but these studies mainly analysed query logs from
large commercial search engines [
OHSUMED, published in 1994, was the rst collection containing medical
data used for IR evaluation [
The TREC Medical Records Track ran in 2011 and 2012 [
The data set provided to participants comprises a document collection of around
one million documents (web pages from medical web sites), 50 topics, which were
developed by medical experts, and the corresponding relevance information [
The data was provided to participants after signing an agreement, through the PhysioNet website. As test data, ve training topics together with corresponding relevance assessment were released.
We describe in this section each part of the task dataset. 7 http://www.ncbi.nlm.nih.gov/mesh/
A large web crawl of health resources is used as the corpus for this task. The
crawl contains about one million documents, which have been made available
to CLEF eHealth through the Khresmoi project [
Novel methods to generate contextualised statements of patient information needs were used. These are based on realistic short query statements created in the context of patient discharge summaries. The discharge summaries can be considered as a description of the context in which the patient has been diagnosed with a given disorder and has written a query. The discharge summaries originate from the de-identi ed MIMIC-II database13 (Multiparameter Intelligent Monitoring in Intensive Care, Version 2.5).
Discharge summaries are semi-structured reports with the following appearance: Admission Date : [ 2014 03 28 ] D i s c h a r g e Date : [ 2014 04 08 ] Date o f B i r t h : [ 1930 09 21 ] Sex : F S e r v i c e : CARDIOTHORACIC A l l e r g i e s : P a t i e n t r e c o r d e d as having No Known A l l e r g i e s to Drugs Attending : [ Attending I n f o 5 6 5 ] C h i e f Complaint : Chest pain Major S u r g i c a l or I n v a s i v e Procedure : Coronary a r t e r y bypass g r a f t 4 .
H i s t o r y o f P r e s e n t I l l n e s s : 83 year o l d woman , p a t i e n t o f Dr . [ F i r s t Name4 ( NamePattern1 ) ] [ Last Name ( NamePattern1 ) 5 0 0 5 ] , Dr . [ F i r s t Name ( S T i t l e ) 5 8 0 4 ] [ Name ( S T i t l e ) 2 2 7 5 ] , with i n c r e a s e d SOB with a c t i v i t y , l e f t s h o u l d e r b l a d e / back pain at r e s t , + MIBI , r e f e r r e d f o r c a r d i a c cath . This p l e a s a n t 83 year o l d p a t i e n t n o t e s becoming 8 http://www.healthonnet.org 9 http://www.hon.ch/HONcode/Patients-Conduct.html 10 http://www.drugbank.ca/ 11 http://www.diagnosia.com/ 12 http://www.tripanswers.org/ 13 http://mimic.physionet.org SOB when walking up h i l l s or i n c l i n e s about one year ago . This SOB has p r o g r e s s i v e l y worsened and she i s now SOB when walking [ 01 19 ] c i t y b l o c k ( f l a t s u r f a c e ) . [ . . . ] Past Medical H i s t o r y : a r t h r i t i s ; c a r p a l t u n n e l ; s h i n g l e s r i g h t arm 2 0 0 0 ; needs r i g h t knee r e p l a c e m e n t ; l e f t knee r e p l a c e m e n t i n [ 2 0 1 0 ] ; thyroidectomy 1 9 7 8 ; c h o l e c y s t e c t o m y [ 1 9 8 1 ] ; hysterectomy 2 0 0 1 ; h/o LGIB 2000 2001 a f t e r t a k i n g baby ASA; 81 QOD [ . . . ] 3.3
The queries used in the task aim to model those used by laypeople (i.e., patients,
their relatives or other representatives) to nd out more about their disorders,
once they have examined a discharge summary. Disorders have been identi ed
within discharge summaries and linked to the matching UMLS (Uni ed Medical
Language System) concept in the CLEF eHealth Task 1 [
Previous evaluation tasks in health IR have used MeSH entries (the MeSH ontology is contained in the UMLS meta-ontology) as queries (see Section 2). However, the queries considered by the task presented here are intended to be representative of real patients' information needs and statements. Thus the possibility of issuing concept-queries is discarded. Layperson queries tend to be short, with an average length less than two words. However, di erent patients can have di erent information needs associated with the same query statement. For example, a patient that receives a cancer diagnosis for the rst time would have a di erent information need than a patient at a terminal cancer stage. This type of contextual information related to the patient history is contained in the discharge summary. Thus, the discharge summaries can be used for contextually focused generation of queries. The information in a discharge summary can then be used to determine the relevance of retrieved information to the speci c user.
A query is generated for a given disorder and a discharge summary. To better structure the query generation process, patients' information needs have been grouped into three main scenarios: 1. the patient has a short-term disease, or has been hospitalised after an accident (little to no knowledge of the disorder, short-term treatment), 2. the patient has a chronic disease or a long-term disease that has just been diagnosed (little to no knowledge of the disorder, long-term treatment), and 3. the patient has a chronic or long-term disease, and this is the n-th diagnosis (potentially good knowledge of the disorder, long-term treatment).
Queries to be used in this task have been created by experts (each expert was a registered nurse and clinical documentation researcher) involved in the CLEF eHealth consortium. This solution has been chosen in place of recruiting patients because of the issues involved with recruitment and privacy. We believe that, being on a daily basis in contact with patients receiving treatments and discharge summaries, nurses are familiar with patients' information needs and patient pro les.
65 disorders have been randomly selected from the set of 1,006 disorders identi ed in the CLEF eHealth Task 1. For each disorder, a discharge summary containing the disorder itself has been randomly selected. Using the pairs of disorder and associated discharge summary, the experts have developed a set of patient queries (and criteria for judging the relevance of documents to the queries, for use in the relevance assessment task described in the next section). Queries are provided in a standard TREC format, consisting of a topic title (text of the query), description (longer description of what the query means), and a narrative (expected content of the relevant documents).
The following example outlines a query: <query> < t i t l e > thrombocytopenia treatment c o r t i c o s t e r o i d s
l e n g t h </ t i t l e > <desc> How l o n g s h o u l d be the c o r t i c o s t e r o i d s treatment
to c u r e thrombocytopenia ? </desc> <narr> Documents s h o u l d c o n t a i n i n f o r m a t i o n about t r e a t m e n t s o f thrombocytopenia , and e s p e c i a l l y c o r t i c o s t e r o i d s . I t s h o u l d d e s c r i b e the treatment , i t s d u r a t i o n and how the d i s e a s e i s cured u s i n g i t . <s c e n a r i o > The p a t i e n t has a s h o r t term d i s e a s e , or has been h o s p i t a l i s e d a f t e r an a c c i d e n t ( l i t t l e to no knowledge o f the d i s o r d e r , s h o r t term treatment ) </s c e n a r i o > <p r o f i l e > P r o f e s s i o n a l f e m a l e </ p r o f i l e > </narr> </query>
With this approach, ve training and fty test queries have been generated
for use in the task. 65 disorders have been selected (i.e. more than the targeted
number of queries) because some disorders/queries may not be answerable using
web pages from the document collection. During the query generation process,
the experts manually removed disorders from the list of 65 that do not allow
for realistic query generation. A real log containing queries issued by the general
public on the HON website has also been used to exclude candidate queries which
are unrealistic of the type of query that a patient would typically enter. For each
query, an IR system that implements a standard BM25 weighting scheme [
Relevance assessment has been performed by domain experts and IR experts. We
used the Relevation system14 for collecting relevance assessments of documents
contained in the assessment pools. Relevation is a system for performing
relevance judgements for Information Retrieval system evaluation [
As we received many run submissions, we had to limit the pool depth and distributed the relevance assessment workload between medical experts and IR experts. The relevance assessment for these test queries has been conducted by six Finnish nursing professionals and ve Australian nursing professionals or students in health sciences (domain experts), and three Irish, one Australian, and one Swedish senior researcher in clinical NLP (Natural Language Processing) and ML (Machine Learning), all technological experts. Each document was assessed by one person.
We pooled the top ten documents obtained from the participants' baseline runs, and both their top-priority run using discharge summaries and their toppriority run not using discharge summaries. This resulted in a pool of 6,391 documents. The relevance assessment was based on a four point scale, which is mapped to a binary scale. The graded relevance assessment yielded 0: 4,316, 1: 197, 2: 1,439, 3: 439 documents. The binary relevance assessments yielded 0: 4,513 non-relevant and 1: 1,878 relevant documents.
Thus, there have been 37.56 relevant documents per topic on average and 127.82 documents assessed per topic. Table 1 shows the relevance assessment coverage. (*) indicates runs for which results up to rank 10 were completely assessed.
Relevance assessments for the ve training queries were formed based on
pooled sets generated using the Vector Space Model [
For this task, the participants were allowed to submit up to seven runs, one mandatory run using no additional information or external resources (run 1), three using the discharge summary and any other external resource (runs 2-4), and three using external resources but not using the discharge summaries (run 5-7). Among each set of additional runs, one had to use only the title and the description elds of the query. Participants were also asked to rank their runs 2-4 and 5-7 according to their importance. 4.1
13 groups registered for task 3 and 9 groups submitted runs. The groups are from 5 countries and 4 continents: Asia (2), Australia (2), USA (4), Europe (1). The groups are listed in Table 2. Although CLEF has mainly been the focus of European groups, only one group from Europe submitted runs to this lab.
Teams submitted in total 48 runs, including: 9 baseline runs, 15 runs using discharge summaries (from 5 teams), and 24 runs not using them. We examined all documents in runs 1, 2 and 5 up to rank 10 for relevance. The two major evaluation metrics are therefore metrics at a cut-o of up to 10 documents, i.e. P@5, P@10, NDCG@5, and NDCG@10. In addition, we considered MAP as an evaluation metric, but we are aware that MAP is unreliable because only the top ten documents have been assessed. Nevertheless, we wanted to report a measure covering the full set of up to 1000 retrieved documents. We also report the number of relevant and retrieved documents in the top 1000 results as a more recall-oriented measure. Table 1 provides details on the coverage of assessments for each run up to rank 10.
Performance metrics are computed with the standard trec eval tool15 using the following options: { trec eval -c -M1000 { trec eval -c -M1000 -m ndcg cut
We are aware that the performance metrics for other runs might be unreliable compared to that of runs 1, 2, and 5. However, this situation is common for IR lab evaluations, where additional experiments on an existing data set typically do not include re-assessment of documents previously not retrieved or relevance assessment of additional documents. 4.3
For comparison, we created our own baseline experiment, which is based on the
BM25 retrieval model. This experiment does not incorporate any domain-speci c
adaptations. We used the JSoup library16 to extract the textual content from
the web pages and applied standard normalization (e.g. replacing named HTML
characters) and spelling correction based on a xed list of frequent spelling errors.
This approach has been employed before for experiments on medical records
retrieval [
This baseline system uses the BM25 retrieval model and standard blind
relevance feedback for BM25 [
The baseline system performs only two types of document preprocessing:
character normalization (i.e. mapping characters with diacritical marks to the
equivalent characters without) and word normalization (e.g. correcting frequent
spelling errors). Spelling correction is based on a list of 9533 spelling errors from
medical documents [
We generated two baseline experiments: one with standard retrieval using the BM25 model (BM25 baseline) and a second experiment using blind relevance feedback (BM25 FB baseline). For blind relevance feedback, the top T terms from the top ranked D documents are added to the query to retrieve the nal result set of documents. For our baseline experiments, we used T = 10 and D = 10. 4.4
The o cial results for all submitted runs and for our baseline experiments are shown in Table 3.
Comparing the participants' results wrt. P@10 and NDCG@10, we found that for 4 teams (ohsu, THCIB, teamAEHRC, uog) run 5 (using discharge summaries) achieves the best P@10, while for 2 teams (QUT-TOPSIG, TeamMayo), 15 http://trec.nist.gov/trec eval/ 16 http://jsoup.org/ run 2 (not using summaries) achieves the best P@10. For 3 teams (TeamKC, MEDINFO, UTHealth), the baseline runs (run 1) performs best.
Only 4 runs (from the same group) outperform our baseline experiment using standard blind relevance feedback wrt. P@10. Runs from two groups outperform the baseline experiment wrt. NDCG@10. This illustrates that our baseline system based on BM25 is a very strong baseline. 5
We describe in this section the approaches used by each team, and summarize ndings from their analysis.
Team THCIB [
Team TOPSIG [
Team SNUMEDINFO [
Team OHSU [
Team UThealth [
The best result overall was obtained by team Mayo, using a retrieval model and external resources for query expansion, as well as re-ranking based on concepts from the query and the discharge summary. However, this team, together with Topsig, are the only teams having improved their baseline with the use of the discharge summaries, and in both cases for re-ranking purposes.
Concept re-ranking has also been used by team THCIB and given their best results. For ve teams, the best scores are obtained with their baseline (coupled with pre-processing such as acronym expansion and spelling correction for team AEHRC). Combination of methods and external resources such as those employed by team Mayo seem to be e cient, but the overall results here show that research still needs to be conducted to make the best out of the external resources. 6
In this rst year of the ShARe/CLEF eHealth2013 evaluation lab Task 3, there was strong take-up in the community with 9 groups submitting runs to the task. The challenge of developing retrieval techniques for layperson medical queries proved di cult. Overall, BM25 is a very strong baseline, which proved hard for teams to beat. BM25 with relevance feedback, as opposed to standard BM25, proved to be the strongest baseline. Here P@10 of 0.4860 was obtained, and NDCG@10 of 0.4328. One team, Mayo, had four runs which performed better than BM25 for P@10. P@10 achieved for these runs ranged from 0.5180 { 0.4880. Of these runs, the best performing used information in the discharge summaries. The only other team that improved over the BM25 baseline was SNUBME for their baseline run, where NDCG@10 of 0.4377 was achieved.
Despite the results obtained being mostly lower than the baseline in this rst year of the task, many valuable insights into retrieval technique development for the domain were gained. This forms a good basis for future exploration in the domain in general, and speci cally for further technique development for the 2014 CLEF eHealth Task 3 lab.
Given the success of this year 1 of the task, we anticipate even more interest in next year's campaign. In the second year of the task, we will seek to remove noise from the document collection. This year, as task participants were informed, there were some non-English web pages in the collection and duplicate web pages. That is there were occurrences in the document collection of the same web page, with the same URL but di erent unique identi er. On top of this, we also identi ed several occurrences of the same web pages with di erent URL pre xes. This occurs for example, when a dropdown menu is expanded. We will also explore the possibility of removing or highlighting such duplication in the collection. In next year's campaign we also intend exploring new query generation techniques and means to improve the relevance assessment work ow. In query generation for example, we will look at using the main disease in the discharge summary the query (information need) stems from instead of using a randomly selected disease from within the discharge summary. The goal here being to increase the relevance of using discharge summaries.
Task 3 of the ShARe/CLEFeHealth2013 evaluation lab has been supported in part by the Khresmoi project, funded by the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement no 257528, and NICTA, funded by the Australian Government as represented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the ICT Centre of Excellence program. The relevance judgements were in part funded by the ESF project ELIAS. We acknowledge the time given to perform the relevance assessment task. We want to thank the following individuals: Maricel Angel (NICTA, Australia), Rmi Bois (Dublin City University), Riitta Danielsson-Ojala (University of Turku, Finland), Lotta Kauhanen (University of Turky, Finland), Hugo Mougard (Dublin City University), Laura-Maria Murtola (University of Turku, Finland), Heidi Parisod (University of Turku, Finland), Josh Robertson (Dublin City University, Ireland), Eriikka Siirala (University of Turku, Finland), Timothy Sladden (University of Queensland, Australia), Thomas Souchen (AEHRC, Australia), Sumithra Velupillai (Stockholm University, Sweden).