=Paper=
{{Paper
|id=Vol-2482/paper4
|storemode=property
|title=A Relation Extraction Approach for Clinical Decision Support
|pdfUrl=https://ceur-ws.org/Vol-2482/paper4.pdf
|volume=Vol-2482
|authors=Maristella Agosti,Giorgio Maria Di Nunzio,Stefano Marchesin,Gianmaria Silvello
|dblpUrl=https://dblp.org/rec/conf/cikm/AgostiN0S18
}}
==A Relation Extraction Approach for Clinical Decision Support==
https://ceur-ws.org/Vol-2482/paper4.pdf
A Relation Extraction Approach for Clinical Decision
Support
Maristella Agosti Giorgio Maria Di Nunzio
maristella.agosti@unipd.it giorgiomaria.dinunzio@unipd.it
Stefano Marchesin Gianmaria Silvello
stefano.marchesin@unipd.it gianmaria.silvello@unipd.it
Department of Information Engineering
University of Padua, Italy
Via Giovanni Gradenigo, 6/b
literature from a large collection of medical publica-
tions. Due to severe time constraints, clinicians must
Abstract take fast decisions without having the possibility to
thoroughly read the literature; for this reason, case-
In this paper, we investigate how semantic re- based retrieval favors precision over recall [BDS+ 04].
lations between concepts extracted from medi- A key characteristic of the medical literature is the
cal documents can be employed to improve the large use of synonyms and context-specific expressions.
retrieval of medical literature. Semantic rela- To address this term heterogeneity, Knowledge Bases
tions explicitly represent relatedness between (KBs) have often been exploited by Information Re-
concepts and carry high informative power trieval (IR) systems. The current availability of med-
that can be leveraged to improve the effec- ical KBs offers us the opportunity to develop tech-
tiveness of retrieval functionalities of clinical niques that better capture the semantics of medical
decision support systems. We present prelimi- documents, leading to the following research question:
nary results and show how relations are able to
provide a sizable increase of the precision for How can we employ the rich semantic in-
several topics, albeit having no impact on oth- formation within medical case reports and
ers. We then discuss some future directions to related literature to boost retrieval perfor-
minimize the impact of negative results while mances and ease the clinical decision process?
maximizing the impact of good results. Semantic relations are a key aspect within the se-
mantics of a document. They have been mainly used to
1 Motivation find relevant concepts to expand a user query, but not
as semantic elements to be indexed and retrieved. We
The volume of medical literature published every year
hypothesize semantic relations can provide a higher se-
keeps growing at a very fast pace. The time required
mantic representation of medical cases and literature.
by clinicians to retrieve relevant information from such
In this work, we present an initial study on the ef-
an amount of literature using standard systems is often
fectiveness of the use of semantic relations for the re-
prohibitive. Therefore, there has been a strong inter-
trieval of medical literature. We define an approach
est in Clinical Decision Support (CDS) systems [Ber07]
comprising two methods: a rule-based method and a
designed to produce effective and timely information
learning method. In the rule-based method, we assign
that can help clinicians in the decision making process
a relation to a pair of concepts — contained within
for patient care. Within this context, we focus on case-
the same sentence — when it holds within a reference
based retrieval — i.e. given a medical case of interest,
KB. In the learning method, we train a sentence-level
the CDS system should retrieve highly related medical
relation extractor that is able to infer relation between
Copyright © CIKM 2018 for the individual papers by the papers' a pair of concepts given the sentence’s context.
authors. Copyright © CIKM 2018 for the volume as a collection We evaluated our approach by using the publicly
by its editors. This volume and its papers are published under
shared OHSUMED collection [HBLH94]. OHSUMED
the Creative Commons License Attribution 4.0 International (CC
BY 4.0).
provides rather short queries which represent a hard problem of finding human readable descriptions of a
task for our approach, since limited information — e.g. given relationship in a knowledge graph. [SRPD16] ap-
concepts and relations — can be extracted from them. ply supervised relation extraction to documents that
Testing with OHSUMED allow us to assess the poten- are relevant for an information need Q and study how
tial and limitations of the approach. The remainder of many of the extracted relations are indeed relevant for
the paper is organized as follows: Section 2 presents Q. [KD17] explores current state of the art in unsuper-
the background and related work, Section 3 describes vised relation extraction (OpenIE) for the task of find-
the proposed approach, Section 4 presents experiments ing support passages to complement an entity ranking
and results and Section 5 draws some conclusions and with human-readable explanations of how those re-
outlines future work. trieved entities are connected to the information need.
Conversely, our approach applies supervised relation
2 Related Work extraction to extract semantic relations that are used
in all stages of retrieval. Hence, relations play a pivotal
Concept-based IR aims at making use of external role in the actual retrieval of documents.
sources (like thesauri and ontologies) to provide addi-
tional knowledge and context that may not be explicit 3 Methodology
in a document collection and users’ queries. Concept-
based methods can be categorized in two types: (i) We present a new approach that uses semantic re-
methods that use concepts in both indexing and re- lations for case-based retrieval. The methodology is
trieval stages [EMG11], and (ii) methods that apply composed of the information extraction step (Subsec-
concept analysis in one specific stage, such as concept- tion 3.1) and the information retrieval step (Subsection
based query expansion [GVDW06]. The approach we 3.2).
adopt extends the use of concepts to relations and uses
them in all stages of retrieval. This is more challeng- 3.1 Information Extraction
ing, but it allows for a finer semantic representation of The information extraction step is divided into an en-
documents and queries. tity linking component and a relation extraction com-
In the biomedical domain — where there are au- ponent.
thoritative and curated ontologies — concept-based The entity linking component extracts entity men-
approaches demonstrate consistent improvements over tions within the text and links them to a reference
classic keyword-based systems. In [KZN+ 12], ‘is-a’ re- KB; this reduces the high number of synonyms, ab-
lationships between concepts are used to weight doc- breviations and context specific expressions that are
uments containing concepts subsumed by the query’s present in the medical literature. For entity linking we
concepts. [LMO13b] proposes a method to represent adopt MetaMap,1 the most authoritative tool to detect
medical records and queries by focusing only on med- medical entity mentions in free-text. MetaMap analy-
ical concepts essential for the information need of a ses biomedical free-text and identifies concepts belong-
medical search task. In [LMO13a], queries are ex- ing to the Unified Medical Language System (UMLS),
panded by inferring additional conceptual relation- associating each mention with a number of concepts
ships from domain-specific resources as well as by from the UMLS Metathesaurus2 — which comprises
extracting informative concepts from the top-ranked more than 3 million distinct concepts. Within UMLS,
medical records. a substantial understanding of the medical domain is
The field of Biomedical Information Extraction included, comprising medical concepts, relations, def-
(BioIE) is highly relevant for CDS. [LCJY16] reviews initions and so on.
the recent advances in learning-based approaches for The relation extraction component detects seman-
BioIE tasks. BioIE tasks comprise entity linking tic relation between pairs of concepts within a sen-
[ZHZ+ 15], event identification [APTK10] and relation tence. To be consistent with concepts extracted with
extraction [USSD11, WF14]. Being targeted to CDS MetaMap, we consider semantic relations from UMLS
— i.e. voted to the extraction of key relations that Metathesaurus as well. Furthermore, since our task
can facilitate clinical decision making — our problem requires a high coverage of the medical domain, con-
setup is fundamentally different from the conventional sidering UMLS Metathesaurus relations — which are
biomedical setups. Most of state-of-the-art biomedical coarse-grained relationships that span to a high num-
relation extraction techniques are developed for spe- ber of concepts — allows us to increase the recall of
cific relations, like protein-protein interactions, gene- extracted relations.
disease interactions and so on — which cover only a 1 https://metamap.nlm.nih.gov/
fraction of the biomedical domain. 2 https://www.nlm.nih.gov/research/umls/knowledge_
Regarding relations in IR, [VMdR17] study the sources/metathesaurus/
We define two methods for the extraction of re- follows:
lations from documents and queries: a rule-based
method and a learning method. Rule-based: a re- X |Rp ∩ Rq |
score(q, d) = BM 25(p, q) (1)
lation is assigned to a pair of concepts if it relates |Rq |
p∈d
them within UMLS. We assume that a UMLS relation
between two concepts always occurs, even when it is where d is the document, q is the query, p is a passage
not explicitly mentioned in the sentence containing the belonging to document d, Rq is the set of relations
two concepts. extracted from query q and Rp is the set of relations
Learning: we train a distantly supervised [MBSJ09] extracted from passage p.
sentence-level Bidirectional Long Short-Term Memory
(BiLSTM) neural network to detect if a relation ex- 4 Experiments and Results
ists between two concepts based on the context of the
sentence. The network architecture is composed of an We employed the OHSUMED test collection which
input (word embedding) layer of concatenated word contains 348,566 references from the on-line medical
features and positional features. Words are first con- information database MEDLINE, consisting of titles
verted into pre-trained word embeddings trained on and/or abstracts from 270 medical journals over a five-
26 million abstracts and citations in PubMed — re- year period (1987-1991). The available fields are: title,
leased by [PGM+ 13]. Then these word features are abstract, MeSH indexing terms, author, source, and
concatenated with two sets of positional features — publication type. There are 106 queries in the collec-
to explicitly account for the pairs of words to which tion. Each query is composed of two sentences: title
we expect to assign relations [ZLL+ 14]. We apply + description. Title is the brief summary of the med-
a max-pooling layer right after the bidirectional re- ical case at hand, description is the information need
current layer and before the output layer — in order required to answer a specific question for the case.
to combine segment-level features that, although not Experimental Setup: We performed two experi-
very strong in representing the entire sentence, repre- ments: i) one using the rule-based method to extract
sent local patterns well [ZW15]. In this way, we try relations out of documents and queries; ii) the other
to overcome the tendency of recurrent connections to using the learning method to extract relations out of
forget long-term information too quickly, leading the documents and queries. We compared the results ob-
supervision at the end of the sentence to be hardly tained applying BM25 to the three representations (i.e.
propagated to early steps in model training (due to BoW, BoC and BoR) and we evaluated the results us-
gradient vanishing [BSF94]). ing the nDCG measure.
Results: i) The rule-based method was able to ex-
3.2 Information Retrieval tract relations from a subset of 44 queries. Therefore,
to investigate the effectiveness of relations, we restrict
Documents are indexed by considering all terms as the experiments to this subset only — since the re-
in the Bag-of-Words (BoW) representation. We ex- maining queries lead to no results when considering
tend the BoW representation to both concepts (BoC) relations. Of these 44 queries, only 39 have relations
and relations (BoR) by considering for the indexing matching with some documents. Regarding the rela-
all the extracted concepts and relations respectively. tions, we obtained the best results with the passage-
The ranking is obtained using Okapi BM25 ranking level approach. We set the passage length to 2, in or-
formula [RW99]. der to be compliant with queries’ length. Documents’
Since relations are extracted at sentence level, we score was computed using the formula shown above
also index passages — i.e. groups of consecutive sen- (1). The nDCG results on these 39 queries are vari-
tences — by considering all the relations occurring able — ranging from 0 (18 cases) to 1 (5 cases), as can
within each group of sentences (passage-level BoR). be seen in Figure 1.
Relevant passages should contain a higher number of Such a variance gives us some hints about the infor-
relations related to the information need when com- mative power of relations. When properly extracted,
pared to non relevant passages — being more similar relations can be highly effective, indeed, we compared
in their semantic contents to the query. Therefore, the average nDCG values of concepts and relations on
documents that contain more relevant passages can be only those topics where relations give a result differ-
considered more relevant for the query. ent than 0 and we found a statistically significant av-
We define a weighting scheme such that a docu- erage improvement of 20%. A t-test was performed to
ment score is computed as the weighted sum of its pas- validate the improvement. Regarding the comparison
sages scores, where scores are computed using BM25 between relations and terms, the behavior of relations
as above. The passage-level weighting scheme is as is similar to the one of terms (baseline approach), and
Figure 1: nDCG values for topics containing relations. OHSU{56,60,29,46,58} returned NA values for the BoR
representation. Queries are sorted in descending order first by BoR (relations) nDCG values, then by BoC
(concepts) nDCG values.
there is no statistically significant difference between Acknowledgements
the two.
Supported by the CDC-STARS project of the Univer-
ii) The learning method was able to extract rela- sity of Padua.
tions from a subset of 25 queries. Of these 25 queries,
only 12 have relations matching with some documents.
The results on these 12 queries are comparable to those
presented for the rule-base method, with nDCG values
ranging from 0 (in 7 cases) to 1 (in 1 case). The reason
for this is two-fold: (a) the shortness of queries that
limits the relations that can be extracted; and, (b) the
highly different syntactic structure of queries if com-
pared to the sentences within the medical abstracts
leading to a mismatch between the query-relations and
abstract-relations.
5 Discussion
In this work, we proposed and evaluated the effective-
ness of semantic relations as basic constituents for a
CDS system. We defined two methods for extracting
relations from queries and documents: a rule-based
method and a learning method. We found that re-
lations — when pertinent to the initial information
need — are highly valuable, outperforming concepts.
The challenge lies in how to limit those cases where
relations provide no relevant results for the informa-
tion need. To this end, considering collections where
queries present a long and narrative structure (e.g.
TREC CDS tracks) might be a possible direction to
balance such issue. Furthermore, defining more IR-
oriented relation extraction algorithms that are capa-
ble of overcoming the high precision-low recall nature
of state-of-the-art methods is a direction we will inves-
tigate.
References [LCJY16] F. Liu, J. Chen, A. Jagannatha, and
H. Yu. Learning for biomedical infor-
[APTK10] S. Ananiadou, S. Pyysalo, J. Tsujii, and
mation extraction: methodological re-
D. B Kell. Event extraction for sys-
view of recent advances. arXiv preprint
tems biology by text mining the litera-
arXiv:1606.07993, 2016.
ture. Trends in Biotechnology, 28(7):381–
390, 2010. [LMO13a] N. Limsopatham, C. Macdonald, and
[BDS+ 04] D. T. Burke, M. C. DeVito, J. C. Schnei- I. Ounis. Inferring conceptual relation-
der, S. Julien, and A. L. Judelson. Read- ships to improve medical records search.
ing habits of physical medicine and reha- In Proceedings of the 10th Conference on
bilitation resident physicians. American Open Research Areas in Information Re-
Journal of Physical Medicine & Rehabili- trieval, pages 1–8, 2013.
tation, 83(7):551–559, 2004. [LMO13b] N. Limsopatham, C. Macdonald, and
[Ber07] E. S. Berner. Clinical decision support sys- I. Ounis. A task-specific query and doc-
tems, volume 233. Springer, 2007. ument representation for medical records
search. In European Conference on
[BSF94] Y. Bengio, P. Simard, and P. Frasconi. Information Retrieval, pages 747–751.
Learning long-term dependencies with Springer, 2013.
gradient descent is difficult. IEEE Trans-
actions on Neural Networks, 5(2):157– [MBSJ09] M. Mintz, S. Bills, R. Snow, and D. Juraf-
166, 1994. sky. Distant supervision for relation ex-
traction without labeled data. In Proceed-
[EMG11] O. Egozi, S. Markovitch, and ings of the Joint Conference of the 47th
E. Gabrilovich. Concept-based infor- Annual Meeting of the ACL and the 4th
mation retrieval using explicit semantic International Joint Conference on Natu-
analysis. ACM Trans. Inf. Syst., ral Language Processing of the AFNLP:
29(2):8:1–8:34, April 2011. Volume 2-Volume 2, pages 1003–1011. As-
sociation for Computational Linguistics,
[GVDW06] F. A. Grootjen and T. P. Van Der Weide.
2009.
Conceptual query expansion. Data &
Knowledge Engineering, 56(2):174–193, [PGM+ 13] S. Pyysalo, F. Ginter, H. Moen,
2006. T. Salakoski, and S. Ananiadou. Distribu-
[HBLH94] W. Hersh, C. Buckley, T.J. Leone, and tional semantics resources for biomedical
D. Hickam. Ohsumed: an interactive re- text processing. In Proceedings of the 5th
trieval evaluation and new large test col- International Symposium on Languages
lection for research. In SIGIR’94, pages in Biology and Medicine, Tokyo, Japan,
192–201. Springer, 1994. pages 39–43, 2013.
[KD17] A. Kadry and L. Dietz. Open relation [RW99] S. E. Robertson and S. Walker.
extraction for support passage retrieval: Okapi/keenbow at trec-8. In TREC,
Merit and open issues. In Proceedings of volume 8, pages 151–162. Citeseer, 1999.
the 40th International ACM SIGIR Con- [SRPD16] M. Schuhmacher, B. Roth, S. P. Ponzetto,
ference on Research and Development in and L. Dietz. Finding relevant relations in
Information Retrieval, pages 1149–1152. relevant documents. In European Confer-
ACM, 2017. ence on Information Retrieval, pages 654–
[KZN+ 12] B. Koopman, G. Zuccon, A. Nguyen, 660. Springer, 2016.
D. Vickers, L. Butt, and P. D. Bruza. [USSD11] Özlem Uzuner, Brett R South, Shuying
Exploiting snomed ct concepts and rela- Shen, and Scott L DuVall. 2010 i2b2/va
tionships for clinical information retrieval: challenge on concepts, assertions, and re-
Australian e-health research centre and lations in clinical text. Journal of the
queensland university of technology at the American Medical Informatics Associa-
trec 2012 medical track. In The Twenty- tion, 18(5):552–556, 2011.
First Text REtrieval Conference Proceed-
ings (TREC 2012)[NIST Special Publica- [VMdR17] N. Voskarides, E. Meij, and M. de Ri-
tion: SP 500-298], pages 1–8, 2012. jke. Generating descriptions of entity
relationships. In European Conference
on Information Retrieval, pages 317–330.
Springer, 2017.
[WF14] C. Wang and J. Fan. Medical relation ex-
traction with manifold models. In Pro-
ceedings of the 52nd Annual Meeting of
the Association for Computational Lin-
guistics (Volume 1: Long Papers), vol-
ume 1, pages 828–838, 2014.
[ZHZ+ 15] J. Zheng, D. Howsmon, B. Zhang,
J. Hahn, D. McGuinness, J. Hendler, and
H. Ji. Entity linking for biomedical liter-
ature. BMC Medical Informatics and De-
cision Making, 15(1):S4, 2015.
[ZLL+ 14] D. Zeng, K. Liu, S. Lai, G. Zhou, and
J. Zhao. Relation classification via convo-
lutional deep neural network. In Proceed-
ings of COLING 2014, the 25th Interna-
tional Conference on Computational Lin-
guistics: Technical Papers, pages 2335–
2344, 2014.
[ZW15] D. Zhang and D. Wang. Relation classifi-
cation via recurrent neural network. arXiv
preprint arXiv:1508.01006, 2015.