The National Center for Biomedical Ontology (NCBO) ontology recommender helps users choose a biomedical terminology by analyzing a submitted document. Submitting a single document might not be representative and result in poor recommendations, while submitting a large sample might be expensive, sometimes unfeasible. In this paper, we investigate the effectiveness of two well-researched automatic summarization techniques as an alternative: topic modeling using Latent Dirichlet Allocation and keyword extraction using TextRank. In our case study, both techniques proved to be extremely valuable, dramatically boosting performance without significantly affecting terminology recommendations (r = 0.83-0.98).
Selecting one or more domain terminologies that are best suited for a given
application has proved to be a hard task [
To help users choose a suitable terminology in text annotation applications,
the National Center for Biomedical Ontology (NCBO) [
Unfortunately, techniques that are widespread in the field of recommender
systems, such as collaborative filtering [
There are several limitations, however, when using a document submitted by a user as context for making recommendations. Firstly, the submitted document might not accurately represent the context of the user’s document collection, misleading the recommender. Secondly, getting recommendations is expensive: our experience shows that a single recommendation can take over 30 seconds when using a full clinical document as input. Thirdly, even with a substantially improved performance of the system, an intensive use with numerous submissions of full texts to the recommender web service might result in degraded performance. Therefore, it would be desirable to minimize both the number and size of submitted documents, while maintaining their informational value.
Summarizing the context of a collection before submitting it to a
recommender has proved to be a useful technique to improve efficiency without
substantially influencing recommendations [
Objectives The main objective of this paper is to study the effectiveness of (a) topic modeling using Latent Dirichlet Allocation and (b) keyword extraction using TextRank as summarization strategies in a context-based biomedical terminology recommender, the NCBO ontology recommender. 1 http://bioportal.bioontology.org/ontologies/SNOMEDCT, as of September 2014
The NCBO ontology recommender web service suggests the most appropriate
biomedical terminology for annotating biomedical documents. The recommender
analyzes a document submitted by the user and applies three criteria for making
recommendations: coverage, connectivity, and size of the candidate terminology,
taking into account all 370 terminologies from Bioportal [
As a representative document collection we selected discharge summaries
from an Intensive Care Unit (ICU), reporting events of a hospitalization (e.g.,
admitting and discharge diagnoses, physical examinations, and past and
followup medications). These text address a number of topics of interest in biomedical
informatics (e.g., anatomy, drugs, and diseases). The documents were obtained
from the Multiparameter Intelligent Monitoring in Intensive Care (MIMIC-II)
research database2, a collection of de-identified data from an ICU [
ADMISSION DIAGNOSIS: End stage renal disease, admitted for transplant surgery. HISTORY OF PRESENT ILLNESS: The patient is a 65 year-old woman with end stage renal disease, secondary to malignant hypertension. She was started on dialysis in (...) PAST MEDICAL HISTORY: End stage renal disease, secondary to malignant hypertension on dialysis. History of anemia following gastric angiectasia (...) ALLERGIES: No known drug allergies.
MEDICATIONS: Unknown.
SOCIAL HISTORY: Married, lives with her husband. She has a history of a half pack of cigarettes per day for 20 years. Occasional alcohol.
PHYSICAL EXAMINATION: The patient was afebrile. Vital signs were stable. Blood pressure was 124/58; heart rate 76; weight 160 pounds. Abdomen soft and nontender (...) HOSPITAL COURSE: On [**3389-7-7**], the patient went to the operating room for living donor kidney transplant, performed by Dr. [**Last Name (STitle) 593**] and assisting by (...) DIAGNOSES: End stage renal disease, status post renal transplant. Arterial thrombosis. Deep venous thrombosis. Resolving hypertension.
Table 1. Excerpt of a discharge summary from the MIMIC-II database.
For topic modeling our collection of discharge summaries, we used a topic modeling tool3 based on LDA. We used all 26,657 documents from MIMIC-II, 200 iterations of Gibbs sampling, and 10 topics, which we termed Topic A, Topic B... Topic J. For each document, we were only interested in the two most prevalent topics and their associated keywords (termed primary topic and secondary topic, respectively). As a per-document summarization strategy, we used an in-house improvement of TextRank. Table 2 shows the obtained keywords using both topic modeling and TextRank summarization when applied to the free text from Table 1.
Our main goal was to evaluate how effective topic modeling using LDA and keyword extraction using TextRank were, in comparison to submitting full texts 2 http://mimic.physionet.org/database.html 3 http://code.google.com/p/topic-modeling-tool/ Topic1 Topic2 TextRank blood patient continued neumonia arterial femoral q day post failure pulmonary blood good renal discharge postoperative fluid renal day history right history rate negative started disease lower transplant mg status patient tube extremity normal ultrasound Table 2. Keywords for document in Table 1 using topic modeling and TextRank. Topic1 and Topic2 represent the primary and secondary topics, respectively. Matching keywords using different methods are marked in bold. to the recommender. For that purpose, we considered the recommender as a black box and took a sample of 20 documents from the MIMIC-II database4. Figure 1 shows our approach for getting the recommendations in each case.
Topic Modeling Doc. Topic1 – W1 Topic2 – W2 1 | E – 0.382 | B – 0.251 2 | E – 0.419 | B – 0.374 ...
Documents Topic Keywords
A | bid daily day ...
...
E | blood day discharge ...
...
J | admission age blood ...
Topics Topic
A | E | J |
Recomm. Ontology
SNOMED CT ...
EHDA ...
EHDA Look-up Table
TextRank Full Text arterial blood day disease extremity good history (...) Ontology Recommender EHDA
EHDA
EHDA
For topic modeling, we submitted each topic’s keywords to the recommender and stored the top recommended terminology for each topic, storing an association between topics, keywords, and recommended terminologies in a look-up table for future use. For keyword extraction (TextRank), we submitted the keywords representing the summary of each document from the sample. As gold standard for comparison, we used the full text of each document. We applied a limit of 7,000 characters to every document submitted, as our preliminary experiments showed that the recommender was not able to process long documents. 4 The first 20 documents retrieved by our PostgreSQL installation. We recorded recommendation times, including pre-processing when applicable (e.g., time spent summarizing a document using TextRank). 3
The discovered keywords using topic modeling (Table 3) suggest several contexts in the documents, such as: medication administration (A), cardiology (C, I), and diagnostic tests (G). However, only two terminologies were recommended: SNOMED CT and EHDA. Surprisingly, EHDA, focused on developmental stagespecific anatomical structures of the human, was recommended for 7 of the 10 topics, including diagnostic tests (G). terminologies were recommended, with SNOMED CT and EHDA recommended for 85% of the documents. Using TextRank, a terminology not identified using the full texts was suggested5. 5 Bone Dysplasia Ontology – http://bioportal.bioontology.org/ontologies/BDO # Full Text Score TextRank Score # Full Text Score TextRank Score 1 EHDA 4378.80 EHDA 1303.18 11 SNOMED 1851.21 NCIT 124.93 2 SNOMED 2539.19 SNOMED 114.75 12 EHDA 1453.65 RH-MESH 97.72 3 SNOMED 1086.31 BDO 108.68 13 SNOMED 1306.00 EHDA 153.02 4 EHDA 5436.65 EHDA 1846.39 14 EHDA 423.34 NCIT 35.81 5 NCIT 183.62 SNOMED 37.12 15 RH-MESH 2068.63 EHDA 81.61 6 RH-MESH 222.07 SNOMED 39.80 16 SNOMED 1783.57 SNOMED 97.44 7 SNOMED 1983.63 SNOMED 146.16 17 EHDA 1734.18 EHDA 285.63 8 EHDA 5926.30 EHDA 688.57 18 EHDA 5949.76 EHDA 884.94 9 EHDA 2695.63 EHDA 612.06 19 SNOMED 1838.72 EHDA 137.71 10 EHDA 4054.92 EHDA 1096.61 20 SNOMED 1979.53 EHDA 1150.17 between terminology distributions respect the gold standard (full texts) was very high (r = 0.83–0.98). Topic modeling the MIMIC-II database took 5 minutes 17 seconds (11 ms per document) and 7 seconds per topic were spent getting a recommendation. When submitting documents, a recommendation took 27 seconds per full text, 11 seconds using TextRank (including summarization). TextRank, and Topic modeling (-w = weighted).
EHDA and SNOMED CT were recommended for the majority (85%) of
documents in the sample, EHDA being preferred. Why EHDA was the most
recommended terminology when submitting discharge summaries as context needs to
be carefully studied, as discharge summaries contain a broad range of topics
(discharge diagnoses, physical examinations, past and follow-up medications, etc.)
that are not covered by EHDA. Even in the case of anatomy, FMA [
When analyzing performance, our results suggest that it might not be feasible for users to submit a large number of documents as a representative context, as getting recommendations for a sample of 20 documents with full texts (limited to 7,000 characters) took nearly 10 minutes. The keywords obtained using topic modeling were less in number than the keywords using TextRank. This should, in principle, make recommendations using topic modeling less correlated to the ones obtained using the full texts, but the opposite was true. This fact might be explained because all 26,657 documents from MIMIC-II were used when modeling the topics, providing a much more accurate context. 5
In this study, we have proposed and evaluated two well-researched automatic summarization techniques for summarizing a large collection of clinical documents used as input to the NCBO ontology recommender: topic modeling the collection using LDA, and per-document TextRank keyword extraction. When comparing both approaches to our gold standard (full texts) in the evaluation, we found out that recommendation times improved considerably. In all cases, the distributions of recommended terminologies were highly correlated with the gold standard distribution (r = 0.83–0.98). The high correlation shows that both TextRank and topic modeling are valuable techniques to summarize the context provided by the full texts and boost recommendation performance without seriously affecting the overall recommendation results.
As future work, we plan to: (i) use a larger sample of documents to investigate if our results are consistent, (ii) select a collection of documents from other domain to generalize our results, and (iii) investigate potential quality issues in the recommender, given the prevalent but inexplicable recommendations of the EHDA terminology when submitting discharge summaries as input. The authors thank H. Ziak, A. Rexha, G. Hammer, C. Martínez-Costa, M. Kreuzthaler, and G. A. Uribe Gómez for their contributions; the NCBO for providing the ontology recommender and Bioportal; and the MIT and the Beth Israel Deaconess Medical Center for providing the MIMIC-II database. This work was developed within the EEXCESS project funded by the European Union FP7/2007-2013 under grant agreement number 600601.
Bibliography