Task 3 of the 2013 ShARe/CLEF eHealth Evaluation Lab simulated web searches for health information by patients. The web searches were designed to be connected to hospital discharge summaries from the patient's Electronic Medical Record (EMR), thus effectually modeling a post-visit information need. We primarily investigated three research questions about the retrieval of medical information from the web: 1) to what degree retrieval techniques effective in searching Electronic Medical Records (EMRs) could aid in finding medical web documents; 2) to what degree medical web retrieval would benefit from natural language processing (NLP) techniques that extract information from text based on medical knowledge; and 3) how to leverage contextual information in the patient's discharge summaries to improve retrieval. We submitted seven runs to ShARe/CLEF eHealth. Our best run used effective EMR-based IR techniques, NLP-produced information, and information in patients' discharge summaries to achieve precision at 10 (P@10) scores at or above the CLEF median for all but 2 of 50 test queries.
Task 3 of the 2013 ShARe/CLEF eHealth Evaluation Lab [
In particular, we used a two-step ranking strategy. In Step 1, we performed retrieval in the text space, where the documents and queries were in their raw text form, by using the MRF, MRM, and MeSH-based query expansion — the first research question. In Step 2, we re-ranked the output from Step 1 in the concept space where everything was represented by the medical concepts. In addition, we produced contextual artifacts of the medical concepts in documents and discharge summaries by using a clinical NLP annotation tool called MedTagger, and incorporated those artifacts in our retrieval models.
To derive the MRM model, we used several external sources: the TREC 2011 Medical Records Track test collection, the TREC 2007 Genomics Track test collection, and a subset of Mayo Clinic clinical notes collected between 2001– 2010. To apply domain knowledge and construct the concept space, we adopted the Concept Unique Identifier (CUI) in the Unified Medical Language System (UMLS) for representing medical concepts.
We submitted seven runs to ShARe/CLEF eHealth. When evaluated by precision at 10 (P@10), our best run, which used the two-step ranking strategy and leveraged information in patients’ discharge summaries, were above or at the CLEF median for all but 2 of 50 test queries.
The rest of the paper proceeds as follows: Sect. 2 presents the overview of our system and follows it with detailed description about the underlying retrieval models and medical-specific features. Then, Sect. 3 describes different system settings for our submitted runs, and shows the official evaluation results. Finally, Sect. 4 concludes our research findings. 2
In this section, we first present an overview of our retrieval system, and then detail its underlying retrieval models and domain-specific features. 2.1
Indexing I
Index T Query 2
Retrieving I
2 1 4 2 5
Text
Space Concept Space
Rank List I 6 6 Reranking
Rank
List II Indexing II
Index C 5
Retrieving II 5 Discharge Summary 3 3 CUI Discharge
Summary
Docs 3 3 CUI Docs 5 1 3 3 4 CUI
Query
NLP Concept Extraction3
1. Indexing I: we used Indri3 [
However, we did not stem the CUIs since they are all unique. Due to the computational cost of NLP concept extraction on 1.6 million documents, we 3 http://www.lemurproject.org/indri/ 4 http://code.google.com/p/boilerpipe/ only indexed the 61,273 documents that were present in retrieved results of the 50 test queries. 5. Retrieving II: the query likelihood model was used to retrieve CUI-only documents based on CUI-only queries. The concepts from discharge summaries were also included for the relevant submissions. We will elaborate on this in Sect. 2.3 where we describe our retrieval models in the concept space. 6. Re-ranking: we weight the text-based retrieval rankings against the conceptbased retrieval rankings. Because retrieving in the concept space is done on the intermediate text-based output (Rank List 1), it is essentially a reranking procedure. 2.2
In this section, we describe our text-based retrieval models.
Query Likelihood Model. We used the query likelihood language model [
#combine(shortness breath swelling).
Note that we formulated the above query by removing the stop words ‘of’ and ‘and’ from its original version ‘shortness of breath and swelling’.
This model has been shown to be effective in web search and in medical records;
here, we test the intersection of the two domains — medical web texts.
State-ofthe-art performance on medical records is possible with language models when
augmented with advanced retrieval models, which we turn to now.
Markov Random Field Model. In the query likelihood model, it is a strong
assumption that query terms are generated independently from the document
language model. In reality, related terms are likely to occur in close proximity to
each other. The Markov random field (MRF) model [
P (QjD) ra=nk ∑
T fT (c) + ∑
OfO(c) + ∑
U fU (c);
(2)
c2T
c2O
c2O[U
where T is defined to be the set of 2-cliques containing the document node and
a query term node, O is the set of cliques involving the document node and two
or more query terms that appear contiguously in the query, and U is the set of
cliques involving the document node and two or more query terms that appear
non-contiguously within the query. f (c) is the feature function over clique c
and ’s are the feature weights. MRF model has been shown to consistently
outperform the standard unigram model across a range of TREC test collections [
Following Metzler and Croft [
In particular, we selected several external collections along with the target
collection (i.e., CLEF) for building our MRM model. Our preliminary work suggests
that the domain and characteristics of the supporting external collections have an
effect on retrieval performance, thus we choose the medically oriented resources
in Table 1, which are shown with their collection statistics. The Medical and
Genomics collections are from TREC 2011 Medical Records Track [
Query Expansion with MeSH. Previous work has shown that expanding
queries with related Medical Subject Headings (MeSH) enhances retrieval
performance significantly in an EMR cohort identification setting [
In this section, we describe several concept-based retrieval models. We used these models for re-ranking the output from the text-based retrieval. The motivation of this re-ranking process was our second research question, namely, whether NLP approaches for concept identification could incorporate some medical domain knowledge and further improve the results from text-based retrieval. Concept Unique Identi ers. In the concept space as shown in Fig. 1, we represented everything by UMLS CUI. These shallow semantics are a common normalization target for medical NLP systems, since extensive work has gone into the curation of the UMLS Metathesaurus in reconciling between many different source ontologies, terminologies, and vocabularies. We used the query likelihood model (see Eq. 1) for computing the relevance score. An Indri query using this model will look like:
#combine(C0225386 C0347940), where the C0225386 is the CUI related to breath or respiratory air, and C0347940 is related to chest swelling.
Discharge Summary. Since queries were generated based on discharge summaries (DS), we hypothesized that DS could contain ‘hidden’ concepts that did not appear in the query but were related to query concepts. Psychologically, if a patient finishes reading his/her medical record, these ‘hidden’ concepts might serve as semantically priming for the patient’s mind when he/she formulates the query. This may be valuable context that informs us about the patient’s actual information need.
To identify these ‘hidden’ concepts and use them for query expansion, we again used MedTagger to convert text DS to CUI DS, and then weighted each CUI by its term frequency, and finally took the top 20 CUIs with their weights for query expansion. An Indri query formulated based on CUI DS looks like: 20 C1269008 20 C0869781 20 C0205042 12 C0018802 11 C1278960 11 C0729538 11 C0398102 11 C0392907 11 C0042449 11 C0036186 10 C0030685 9 C2926602 9 C0460139 9 C0234222 9 C0012621 8 C1548828 ) ), where expansion CUI terms (e.g., those related to heart and artery: C0018787, C0003842, and C0205042) are potentially useful for retrieving information related to chest swelling and shortness of breath.
Attributes. In addition to content artifacts (CUIs), Medtagger also produced contextual attributes of CUIs, such as negation, uncertainty, semantic group, and experiencer. We used negation and uncertainty attributes in our submitted system. In particular, we experimentally assigned weights to different ‘values’ of CUI attribute, as illustrated in Table 2. Those weights embody the importance of specific attribute values of CUIs in terms of ranking documents. Note that Table 2 corresponds to the weight assignment when there are no negated CUIs in the original query and the status of each query CUI is ‘confirmed’. Thus, an Indri query using CUI attributes will look like: #weight(0.475 C0225386.sta0,pos, 0.0475 C0225386.sta1,pos, 0.095 C0225386.sta2,pos, 0.3325 C0225386.sta3,pos, 0.025 C0225386.sta0,neg, 0.0025 C0225386.sta1,neg, 0.005 C0225386.sta2,neg, 0.0175 C0225386.sta3,neg), where we applied the attributes values as field restrictions in Indri, e.g., only negated C0225386 with ‘probable’ status in a document can match the Indri query term ‘C0225386.sta3,neg’, and consequently contribute with a weight value of 0.0175 to the relevance score of that document. Note that we used only one CUI in the above example for a simple demonstration. Longer CUI attribute queries can be formulated in a similar way.
We submitted 7 runs based on different settings of our retrieval system. For all the runs, we experimentally set the Dirichlet smoothing parameter to 2500. In this section, we describe each run in more detail, and present the corresponding evaluation results. 3.1
Run 4: On top of the setting for run 3, we further applied the CUI attributes (see Sect 2.3) for re-ranking documents in the concept space. Tests the third research question (usefulness of discharge summaries) when compared to run #7.
Run 5: We used the same setting as run 3 for the text-based retrieval, i.e., we leveraged MRF, MRM, and MeSH-based query expansion. However, we skipped the re-ranking process and used the initial ranking (i.e, rank list 1 in Fig. 1) for submission. Tests the first research question (robustness of IR methods from medical records) when compared to the baseline.
Run 6: On top of the setting for run 5, we further re-ranked the top 1000 retrieved documents in the concept space by using CUIs only (see Sect. 2.3). Tests the second research question (usefulness of NLP methods) when compared to run #5.
Run 7: On top of the setting for run 6, we further exploited CUI attributes (see Sect. 2.3) for documents re-ranking. Tests the second research question (usefulness of NLP methods) when compared to run #5. 3.2
Table 3 shows the official evaluation results for our submitted runs on the primary evaluation metrics P@10 and NDCG@10. Run 2 obtained the best overall performance among the seven runs. We further compare the P@10 performance of run 2 with the medians of all submitted runs of CLEF. As we can see in Fig. 2, for 14 out of the 50 test queries, run 2 obtained the best P@10 scores. Run 2 is also at or above the median for all but two of the queries (Topic 32 and Topic 49).
In error analysis, we found that the top 10 retrieved for Topic 32 (which contains a single query term ‘sob’, i.e., shortness of breath) are all non-English documents which are automatically considered as irrelevant according to the task guideline. This points to possible improvement through modifying the preprocessing algorithm, including a language filter. Our system performed reasonably on Topic 49, but other systems in the Task 3 pool outperformed it.
The comparison between the pairs (run 3 vs. 4, and run 6 vs. 7) suggests that the CUI attributes hurt the performance. This might be because due to the heuristic attribute weight settings (see Table 2), an issue due to the limited training data (only 5 sample queries were available for training).
The comparison between runs 3 and 6 and between 4 and 7 are insignificantly pessimistic about incorporating information in the discharge summaries into our retrieval model.
Furthermore, the performance of run 5 is comparable to run 2, suggesting that the text-based retrieval model alone is quite competitive. It is unclear whether the improved performance in run 2 is due to the absence of the MeSH terms, or the presence of the concept space. 4
We have shown that the text-based retrieval leveraging the well-tested MRF and MRM presented relatively strong performance for this CLEF retrieval task. The best-performing system also included NLP results and query expansion through discharge summaries. We have also found, however, that named entity attributes should not be used with untested heuristics. Future work includes a more principled inclusion of named entity attributes, different preprocessing of web text, and cross-validation to determine the stability of the results. 5
The work was supported by ABI: 0845523 from United States National Science Foundation, R01LM009959 from United States National Institute of Health. The challenge was organized by the Shared Annotated Resources (ShARe) project, funded by the United States National Institutes of Health with grant number R01GM090187.