Extracting clinical entities and their relations from clinical texts is a preliminary task for constructing medical knowledge graph. Existing end-to-end models for extracting entity and relation have limited performance in clinical text because they rarely take both latent syntactic information and the effect of context information into account. Thus, this paper proposed a context-aware end-toend neural model for extracting relations between entities from clinical texts with two level attention. We show that entity-level attention effectively acquires more syntactic information by learning a weighted sum of child word nodes rooted at the target entities. Meanwhile, sub sentence-level attention in an effort to capture the interactions between the target entity pairs and context entity pairs by assigning weights of each context representation within one sentence. Experiments on real-world clinical texts from Shanghai Ruijin Hospital demonstrate that our model significantly gains better performance in the application of clinical texts compared with existing joint models.
Clinical texts such as ultrasound reports and CT reports provide a wealth of clinical
factual knowledge, mainly embodied in various clinical entities, such as organizational
entities, location entities, index entities and attribute entities. Extracting these entities
and their relation from unstructured reports are standard tasks of clinical information
extraction and the foundation of constructing medical knowledge graphs. Named Entity
Recognition (NER) and Relation Extraction (RE) as information extraction techniques
are critical to natural language understanding and knowledge acquisition. Tradition
work usually treat them as two steps to perform information extraction in a pipeline
model, where the final performance will be hurt by the preceding errors generated in
NER. To resolve this problem, joint models for NER and RE have been proposed and
found effective for alleviating the problem of error propagation and utilizing the
interactions between the two sub-tasks [
Recently, deep learning has attracted much attention and become an alternative work
because it employs automatic feature learning without handcrafted and complicated
feature engineering. Several neural models have dominated the joint models of RE due
to the better results. Miwa M. and Bansal M. [
Given a sentence “左侧[left]甲状腺[thyroid]内[inside]可见[indicate]多个[multiple]
混合性[mixed]回声[echos]，之一[one of which]大小[size]为[is]7*4mm” 1, whose
detail of annotation is shown in Figure 1, “大小” is an index and it is related to another
index “回声”. In another case, “甲状腺[thyroid]左叶[left lobe]大小[size]及[and]形态
[appearance]正常[normal]”1, “大小” is also an index, but it is directly related to two
anatomical locations “左” and “右叶”, not related to the other indexes. Tree-structured
relation extraction models [
of two examples in English are as followed: (1) There are multiple mixed echoes in the left thyroid, one of which is 7*4 mm big. (2) The size and appearance of the thyroid are normal. sentence in general contains at least one anatomic location, one index and one attribute. We need to find the related indexes for the anatomic locations, the related attributes for the indexes and detect whether the links between the indexes exist or not, so the interactions between multiple relations (at least two) within one sentence should be captured but mostly be neglected in the tradition models.
In this paper, we propose a novel context-aware end-to-end neural RE model with
entity-level and sub sentence-level attention mechanism to boost performance in
clinical information extraction. We introduce entity-level attention to capture richer
syntactical information by adopting a weighted sum of different child word nodes under the
entities instead of the direct sum in traditional tree-structured models. Inspired by the
previous work [
Experimental results show that the proposed model significantly achieves better performance than state-of-the-art methods. We also analysis the contribution of different attention methods at entity-level and sub sentence-level based on the tests of composition change. We find entity-level attention’s performance of F1 no worse than no attention mechanism and it generally obtains 6-9% improvement of F1 when syntactic rules are more complex. Sub sentence-level attention boosts F1 by at least 4.2% compared with no attention mechanism and achieves an 3.1% improvement compared with joint models with simple attention.
As for named entity recognition, traditional models are based on Hidden Markov
Models (HMM) [
ral models have also been proposed, such as convolutional neural network
(CNN)based models [
framework via integer linear programming, Kate R. J. and Mooney R. J. [
strated that attention mechanism is able to capture the important semantic information.
Sorokin D. et al [
In this section, we introduce the proposed context-aware end-to-end model of entity and relation extraction with entity-level and sub sentence-level attention. The framework comprises of one preliminary task and two primary tasks: (1)input representation: segment the sentences into words, retrieve Part Of Speech(POS) tags, parse the dependency of the sentences and convert them into vectors as inputs of the whole network; (2)clinical entity extraction: Bi-GRU with CRF layer to identify the entity types, as shown in Figure 3; (3)clinical relation extraction: Bi-Tree-GRU with attention mechanism at entity-level and sub sentence-level to extract relationships between the entity pairs recognized in step 2 and details are described in Figure 4. The input vectors are shared and affected by both clinical entity and relation extraction so that the interactions between the two steps could be exploited and the error delivery from the previous step could be reduced. 3.1
Clinical Entity Recognition represented by one-hot encoding and need to be transformed into a dense vector by dictionary and is a hyper-parameter to be chosen. learned embedding matrix ∈ ℝV× , where V is the total size of vocabularies in the
= = [ , ] (1) (2)
Where and denote the embeddings of the i-th word and the POS tag embeddings of , respectively. Learned POS embedding matrix ∈ ℝ 1× 1 reduces the dimensionality of original representations of POS tags, where is the numbers of POS tags in our biomedical datasets and
is the dimension of dense POS vectors. The concatenation of the word embedding and POS tag embedding is taken as the input for the task of clinical entity recognition and its dimension is = + .
Bi-GRU Layer.
Given the input vectors model as the following implementation: and the previous hidden state h− , we mainly use GRU (1) (4) (5) (6)
is (7) z r = σW
= σW ⃗⃗⃗⃗⃗⃗⃗ and ⃐h⃗⃗⃗⃗⃗⃗⃗, referring to the forward and backward GRU, are concatenated to represent h
tasks because of the bidirectional information included. a word as h = [ h ⃗⃗⃗⃗⃗⃗⃗ ; h
⃐⃗⃗⃗⃗⃗⃗⃗], which are effective for numerous sequential tagging CRF Layer. to y and , is the score of tag y of word : Without CRF layer, the model will predict the tags based on the independence assumptions and ignore the strong dependencies (e.g, I-LOC cannot follow B-IND) between the adjacent outputs. Thus, we take the dependence between the two labels into consideration through stacking CRF layer on Bi-GRU. Given by an input sentence, the model could jointly decode the best chain of labels effectively by adopting Viterbi algorithm after learning the distribution of annotated datasets.
First, we compute the score of prediction sequences y = {y , y , ⋯ , y } of the input sentence X as followed, where denotes the transition score from tag y+ , +1 SX, y
= ∑= , +1 + ∑= ,
Afterwards, we maximize the conditional probability of the prediction sequences via softmax layer. In practice, we usually the log-likelihood to choose the parameters during CRF training. Where Y(X) denotes all the possible prediction sequences including those go against the BIO tagging scheme: |X =
ep SX, ∑ ̃ ∈
ep SX, ̃ = l ( |X ) = S X, y − log ∑ ̃ ∈ exp SX, ỹ
̂ = argmax̃ ∈
SX, ỹ beddings from dependency parsing tree are fed into Bi-Tree-GRU with entity-level and sub sentence-level attention and finally softmax predicts the relations between entity pairs.
= [ℎ , ， ] (11)
Where ℎ denotes the hidden state which has been explained in Eq.(6), denotes the label embeddings converted from the outputs of clinical entity recognition, the representation of the entity tags. After dependency-parsing our clinical documents, we convert the dependency types between the words and their parents in the parsing tree into the dense vectors
representing the syntactic information. For example, as shown in Figure 2, the syntactical dependency relationship between the word “左侧 [left]” and its parent “甲状腺[thyroid]” is attributive relation(“定中关系” in Chinese). We directly concatenate the three vectors and take them as the input vectors of relation extraction task.
its children embeddings.
Supposed the t-th word node has N children, matrix consists the representation vectors of the children nodes,
Wh and W are shared parameters to be learned, is the attention weight and the representation vector ℎ is generated by a weighted sum of (12) (13) (14) (15) (16) (17) (18) (19) (20) = [ , , … ] = tanh Wh = W
ℎ = Then, we calculate the hidden state vector of t-th word node in the GRU unit: z r = σ W
= σ W
We adopt the concatenation strategy to represent a sub sentence S , which is similar with the entity recognition, but we call the two directions as bottom-up and top-down in the tree-structured network.
Where ↑ h
denotes the hidden state vector in bottom-up direction for the lowest common ancestor of the focus entity pair, and ↓ h′ , ↓ h ′′ denote the hidden state vector in top-down direction for the first and second entities. If the entity is comprised of more than one word, we take the word node which is less near to the root. Sub Sentence-Level Attention With Context.
One sub-sentence incorporates the words of the target pairs and other words on the path to their lowest common ancestor. For example, as shown in Figure 2, the entity “左侧 [left]” pairs up with “回声[echos]”, and “甲状腺[thyroid]” is the word on the path to their lowest common ancestor “可见[indicate]”, so these four words count as one subsentence. The sub-sentences may have overlaps. Take the above-mentioned example again, the word “回声[echos]” is also contained in another sub-sentence.
Supposed that S = { , , … , } is comprised of k context sub-sentences, which are related to target entity pairs, we use a weighted sum of the context sub-sentences as followed:
, = = W
= ∑=
∗ = [ ; ] , 3.3
The Adjusted Optimization Function.
To increase the valid entity or relation prediction and reduce the impact of large amounts of O(outsider) tags in NER task and U(unknown) tags in RE task, we introduce the hyper-parameter to adjust the imbalance of data. The smaller is, the less impact of ‘O’ tags in the model. We take adjusted as the example: = ∑ l ( |X ) ∙ − (
|X ) ∙ = { , ,
+ l = ′′ ≠ ′′
Unlike the ways of simply getting a weighted sum of sub-sentences, we use to capture the relationship between the target sub-sentence and it context sub-sentences using the weight matrix W to be learned.
denotes the resulting rep’s context sub-sentences, a weighted sum of each single context rep. Finally, we obtain S* as a context vector, the concatenation of and , and feed it into the softmax layer.
We maximize the conditional probability of the relation sequences via softmax, compute log-likelihood and predict the outputs of relations via the highest conditional prob
≤ ≤ resentation of resentation ability.
( | ∗; , ) = ∗ +
= log (
| ∗; , ) ̂ = argmax ∈ ∗ ( | ∗; , ) (21) (22) (23) (24) (25) (26) (27) (28)
= + = − − (29)
Experimental Setting ance them. 4 4.1 Dataset.
Metrics. els.
The experiment datasets in this paper are collected from Ruijin Hospital. We evaluated the model on the ultrasonic reports, X-ray/CT reports, Puncture reports, pathology reports of thyroid and mammary gland. ually checkout the outputs and finally add some medicine-specific words and rules to complete annotations.
Hyper-Parameters.
While training, we use Adagrad optimization to tune the models with three-fold validation by adopting grid search for optional hyper-parameters and randomized search for scoped ones.
Initially, we try {0.01/0.05/0.1/0.2} for learning rate, {50/100/150/200} for batch size, {25/50/100/150/200} for the dimension of variant embeddings and 0.0-1.0 for drop probability and importance β and γ. During experiments, we find a set of effective configuration, which is shown in Table 2.
Comparison with the baseline model.
Table 3 compares our model with the baseline model of Miwa M. and Bansal M. [
To measure the contribution and effect of entity-level attention, we conduct the tests of
composition change to compare three types of dependency layers with different
selection of weighted nodes on four kinds of reports from the thyroid dataset. Shortest Path
Tree(SP-Tree) [
To assess the sub sentence-level attention mechanism, we also conduct the tests of
composition change to compare no attention and two types attention mechanism on four
kinds of reports from the thyroid dataset. “No attention” directly outputs the prediction
through the softmax layer, “Simple Attention” [
tion(p<0.1).2 No sub sentence-level attention degrades at least 4.2% in F1 on every kind of report and achieves a 8.6% reduction of F1 on the whole Thyroid dataset. “Context Attention” performs better than “Simple Attention”, though the difference in prediction is not big. 2 The statistical significance is computed by the Wilcoxon rank-sum test on F1-scores of every report from the thyroid dataset. In this paper, we propose a context-aware end-to-end neural model with two level attention to exploit the interaction between these two sub tasks and considering latent syntactic information within entities themselves and multiple relations in a single sentence. Our experiments give the following findings: (1) In comparison to the baseline model, taking entity-level as well as sub-sentence level attention into account is beneficial to clinical texts. (2) Compared with no attention models, entity-level attention based on parsing paths between the entity pairs, has been demonstrated to be able to acquire underlying syntactic information within entities. Entity-level attention using SP-Tree is a good and effective choice when saving time is more prior based on the tests of composition change. (3) By incorporating attention at sub sentence-level, we are allowed to capture the interaction between a sub-sentence, including the target entity pair, and its context sub-sentences, including other entity pairs, in the same sentence. Context attention achieves significantly greater performance than no attention and shows a slight improvement compared with simple sentence-level attention. Acknowledgments.
6