In this working notes paper we present our methodology and the results we obtained in task 3a of the CLEF eHealth lab 2014. In the set-up of our experiments we assumed that the discharge summary provides the context of the patient's query, and therefore may contain useful background information that can be used to retrieve more relevant results. The central component in our approach is the Indri search engine, with its Language Modelling-based retrieval model. We experimented with query expansion using terms extracted from the discharge summary, and using terms extracted from the UMLS thesaurus. We obtained a small positive e ect from expanding the Indri queries with terms from both sources. Our future work is directed at improving our term extraction and query expansion strategies.
In this working notes paper we present our methodology and the results we obtained
in task 3a of the CLEF eHealth lab 20141. The aim of the eHealth track is \to
evaluate systems that support laypeople in searching for and understanding their
health information" [
The data that was distributed for task 3 consists of: { the 2012 crawl of approximately one million medical documents made available by the EU-FP7 Khresmoi project2 in plain text form; { a set of English `layman' queries that individuals may realistically pose based on the content of their discharge summaries. The query creators used the disorder diagnosed in the discharge summary as main query term. The training set contained 5 queries and the test set 50; { (optionally, after following human subjects training and obtaining the certi cate) the collection of 299 discharge summaries from task 2 among which the discharge summaries belonging to the train and test queries for task 3.
In the set-up of our experiments we assumed that the discharge summary provides the context of the patient's query, and therefore may contain useful background information that can be used to retrieve more relevant results. Our research question is:
2 http://www.khresmoi.eu/ Does query expansion with terms from the medical context of a patient's query lead to better results?
The central component in our approach is the Indri search engine3, with its Language Modelling-based retrieval model. We experiment with query expansion using terms extracted from the discharge summary, and using terms extracted from the UMLS thesaurus. 2 2.1
Document collection We preprocessed the document collection by splitting the corpus les in separate documents and saving for each document its uid, date, url and content.
Discharge summaries We obtained the corpus of 299 discharge summaries that
was distributed for CLEF-eHealth task 2. We processed all discharge summaries that
are referred to by a query in the query set from task 3. We cleaned the discharge
summaries from all variables of the form [** ...**] (e.g. [**MD Number 2860**]),
from abbreviations (using the regular expression [a-z]\.([a-z]\.)+), and from
numbered lists (sequences of lines matching the regular expression ^[
We used the Indri API to index the CLEF collection and set up a query interface to the index. We applied a stopword list to the CLEF collection at indexing time. As ranking model, we use the Indri LM with Dirichlet smoothing and Pseudo-Relevance feedback (PRF) using the Ponte Expander. As parameters for the PRF, we used: number of feedback docs: 20, number of feedback terms: 3. We did not optimize these parameters for the current task but used the optimal settings from a previous task. 2.3
All characters that are not alphanumeric, no hyphen or whitespace are removed from the query and all letters are lowercased. The words in the query are concatenated into one string and combined using the combine function in the Indri query language. We used two types of queries: short queries, consisting of a concatenation of the title and description elds, and long queries, consisting of a concatenation of all four elds title, description, pro le and narr. For example, for this query: <id>QTRAIN2014.1</id> <discharge summary>08114-027513-DISCHARGE SUMMARY.txt </discharge summary> <title>MRSA and wound infection</title> <desc>What is MRSA infection and is it dangerous?</desc> <profile>This 60 year old lady has had coronary artery bypass grafting surgery and during recovery her wound has been infected. She wants to know how dangerous her infection is, where she got it and if she can be infected again with it.</profile> <narr>Documents should contain information about sternal wound infection by MRSA. They should describe the causes and the complications.</narr>
QTRAIN2014.2 QTRAIN2014.3 QTRAIN2014.4 mcg unit stay aortic dissection mute medical intensive care lake thrush end-stage renal disease type levothyroxine sodium department by nodule mesylate saturating right kidney the following short query is created: #combine(mrsa and wound infection what is mrsa infection and is it dangerous) 2.4
We used the discharge summaries to create more informative queries, covering some
of the medical background information of the patient. To this end, we aimed to
extract the most relevant terms from the discharge summary belonging to each
query. We extracted all n-grams with n = f1; 2; 3g from the discharge summary
and ranked them by their relevance. For calculating the relevance of each n-gram,
we used Kullback-Leibler divergence for informativeness and phraseness [
in which P (ui) is the probability of the ith unigram inside the n-gram t { informativeness (how informative is the term for the foreground corpus): kldivP = P (t) log Qn
i=1(P (ui)) kldivI = P (t)fg log
P (t) P (t)fg P (t)bg (1) (2) in which P (t)fg is the probability of the n-gram t in a foreground collection and P (t)bg is the probability of t in the background collection.
For expanding query q, we used the discharge summary belonging to q as foreground collection, and all 299 discharge summaries in the task2 corpus as background collection. The parameter is the weight of the informativeness score relative to the phraseness score:
T ermRelevance = kldivI + (1 ) kldivP (3) We used = 0:9, giving a higher weight to informativeness than to phraseness. We sort the n-grams by their T ermRelevance. Table 1 shows the top-5 terms extracted for a few training queries. Note that not all terms are qualitative, some seem too generic to give information about this speci c patient. This is caused by the relative sparseness of the data from which the terms are extracted, only one document.
In the runs that use the discharge summaries (run2{4), we added the top-k (k = f2; 5g) of terms extracted from the discharge summary to the query, again using unstructured queries with the #combine-operator. This implies that multiword terms are treated as separate words in the query, not as phrases. 271 C UMG6-MRSA-METHICILLIN-RESISTANT-STAPHYLOCOCCUS-AUREUS-INFECTION: *0 #{{{ S *0
MRSA - Methicillin resistant Staphylococcus aureus infection *0 Methicillin-resistant staphylococcus aureus (MRSA) *0 . -#}}}
. -#}}} C UMG9-METHICILLIN-RESISTANT-STAPHYLOCOCCUS-AUREUS: *0 #{{{
S *0
Methicillin resistant Staphylococcus aureus *0 methicillin resistant Staphylococcus aureus *0
MRSA *0 We use thesaurus expansion to enricht the queries with non-personalized information. The idea is that the patient might use layman's terminology and that the technical-medical synonyms of these terms might give better retrieval results. For this purpose, we use an old version of the ULMS-thesaurus that we have stored locally. In the UMLS, medical terms are ordered in synonym sets. An example is shown in Figure 1. We looked up all query titles from the train and test set in the ULMS: First we preprocessed the query titles so that only the rst content word is kept (e.g. \MRSA and wound infection" becomes \MRSA"). Then we processed all synonym sets that { contain one of the preprocessed query titles, or { in which a word from the query title that is between 3 and 5 uppercase characters (presumably an abbreviation) is used between brackets (e.g. Methicillinresistant staphylococcus aureus (MRSA)) For each query title, we created one expansion set by merging all synonym sets in which the preprocessed query title occurs, disregarding synonym sets with 30 or more terms in them (because generic synonym sets such as \UMG2-BODY-PARTORGAN-OR-ORGAN-COMPONENT" contain dozens and sometimes hundreds of terms). From the expansion set, we removed duplicates and near-duplicates (the only di erence is a plural -s, hyphens and underscores, or capitalization). Then we sort the expansion terms by the number of times they occur in synonym sets together with the query title and select the top-k terms to be added to the query.
In the runs that use the thesaurus (run2,3,5,6), we added the top-k (k = 5) of terms extracted from the thesaurus to the query, again using unstructured queries with the #combine-operator. This implies that multi-word terms are treated as separate words in the query, not as phrases. 3
1. Baseline: Indri LM retrieval with Pseudo-Relevance feedback (PRF). Queries are unstructured combination of title and description, using the #combine- operator in the Indri query language 2. Same as Run1, and: { We expanded each query with terms from the discharge summary (k=5) { We expanded each query with maximally 5 terms from the UMLS thesaurus 3. Same as Run2, but with k=2 for the terms from the discharge summary 4. Same as Run2 (k=5), but without the thesaurus expansion. Note that in our original submission, there was an error in this run, as a result of which its results were equal to Run 3. We reproduced the run and included the correct results in the results section below. 5. Same as Run1, and:
{ We expanded each query with maximally 5 terms from the UMLS thesaurus 6. Same as Run5 but with long queries: unstructured combination of title, description, pro le and narrative 7. Same as Run6 but without the thesaurus expansion
{ Run 7, with long queries and no thesaurus expansion gives the exact same results as Run 6, with long queries and maximally 5 terms from the thesaurus. This suggests that when using the long queries with the elds `pro le' and `narrative' fully included, makes it senseless to add a few thesaurus terms. Since runs 6 and 7 are the poorest of all our runs, using the longer queries suspectedly leads to more irrelevant results than the shorter queries. Even if the thesaurus terms are informative, they cannot compensate this. Since we did not do comprehensive analyses of our results, we can only draw preliminary conclusions. We worked with a Language Modelling approach, and although the Indri retrieval model can handle long queries (it will nd the best possible matches for the combination all query terms | not necessarily all terms are present in the retrieved documents), we found that expanding the query consisting of the elds title and description with the elds pro le and narrative has a negative e ect on the retrieval results. However, we did get a positive e ect from adding the 5 most informative terms from the discharge summary and maximally 5 synonyms from the thesaurus to the queries.
In the near future, we plan to do more analyses in order to nd out what factors play a role in the success of query expansion. Our current line of work focuses on personalization of IR through terms extracted from personal documents. The discharge summary is a good example of a document that may provide additional information about the context of a user's query. For that purpose, we aim to improve our term extraction and query expansion strategies. In addition, we will investigate how thesaurus expansion could be applied successfully (selecting the query terms to look up, selecting the synonym sets to expand with).
This publication was supported by the Dutch national program COMMIT (project P7 SWELL).