Clinical named entity recognition aims to identify and classify clinical terms in electronic medical records, including diseases, symptoms, treatments, exams, and body parts. Challenges occur due to ambiguity in the boundary of Chinese words and the number limitation of annotated training data. In this paper, we propose a Bi-LSTM CRF model along with self-taught learning, active learning, and ensemble learning to recognize clinical named entities. The results achieved on CCKS-2017 Task 2 dataset with a F1-Measure of 89.88% ranks among the top systems.
Electronic medical record systems have been widely used in China. Many tasks in clinical text mining rely on accurate clinical named entity recognition (NER), the identi cation of text spans mentioning a concept of a speci c class, including disease, symptom, exam, treatment, and body part. Challenges occur due to ambiguity in the boundary of Chinese words and number limitation of annotated training data.
Traditionally, most of the e ective NER approaches are based on machine
learning techniques, such as Support Vector Machines (SVM) [
In this paper, we regard the clinical NER as a sequence labeling problem,
and use the Bi-LSTM CRF model which is similar to the one presented by
Huang et al. [
The clinical named entity recognition task is de ned as a sequence labeling problem in this paper. Given a text sequence X =< x1, ..., xn >, the goal is to label X with tag sequence Y =< y1, ..., yn >. We experiment with three di erent tagging formats for the recognition, including BIO (Begin, Inside, Outside), BIOS (Begin, Inside, Outside, Single), and BIEOS (Begin, Inside, End, Outside, Single). Examples of the three tagging formats can be found in Table 1.
This model is similar to the one presented by Huang et. al. [
The raw natural language input sentence is processed into sequence of
characters X = [x]1T . The character sequence is fed into an embedding layer, which
produces dense vector representation of characters. The character vectors are
then fed into a bidirectional LSTM layer. The LSTM [
Then the network use sentence level tag information via a CRF layer fed by a fully connected hidden layer. The CRF layer is represented by lines which connect consecutive output layers, and has a state transition matrix as parameters. With such a layer, we can e ciently use past and future tags to predict the current tag, which is similar to the use of past and future input features via a bidirectional LSTM network. We consider the matrix of scores f ([x]1T ) are output by the network. The element [f ]i,t of the matrix is the score output by the network with parameters , for the sentence [x]1T and for the i-th tag, at the t-th character. We introduce a transition score [A]i,j to model the transition from i-th state to j-th for a pair of consecutive time steps. Note that this transition matrix is position independent. We now denote the new parameters for our network as ~ = [ f[A]i,j 8i, jg. The score of a sentence [x]1T along with a path of tags [i]1T is then given by the sum of transition scores and network scores: S([x]1T , [i]1T , ~) =
T
X([A][i]t 1,[i]t + [f ][i]t,t)
t=1
The dynamic programming [
Generally, the more training data we have, the better performance we can get. Considering that we have some unlabeled sentences, a self-taught learning algorithm is introduced to enlarge the training set. First, we train a Bi-LSTM CRF model using the original training set. Second, we apply the trained model to annotate the unlabeled sentences. Third, we choose some high-quality annotation results and add them to the original training set. Speci cally, we de ne an annotating con dence (AC) to evaluate the quality of the annotation results: (1) (2) AC([x]1T , [y]1T , ~) =
eS([x]1T ,[y]1T ,~) P eS([x]1T ,[j]1T ,~) j where [y]1T is the tag sequence predicted by the trained model and [j]1T is the set of all possible output sequences. We choose those annotation results whose AC is greater than a threshold. In addition, considering that disease and treatment entities are much fewer than symptom and exam entities, and the body part entity recognition is not well performed, we only choose the annotation results which must include disease, treatment, or body part entities. 3.3
We also use active learning algorithm to improve the recognition performance. First, we use the original training data along with the self-taught highquality annotation results to train a Bi-LSTM CRF model. Second, we apply the trained model to annotate the rest unlabeled sentences. Third, we manually re-label a few low-quality annotation results whose AC is less than a threshold, and then add them to the training set. Similar to the self-taught algorithm, we only choose the annotation results which must include disease, treatment, or body part entities to re-label manually. 3.4
The task has ve types of clinical named entities to be recognized. If we train models for each type of entities respectively, a class imbalance problem (too many O-tags in sequences) will exist, and the models cannot utilize the information of other types of entities. Thus, we train models to annotate all ve types of entities at the same time, but nally choose ve models in which each model has the best performance for the corresponding type on validation data, so that we can obtain the best recognition performance for all ve types of entities. For example, for disease entities, we choose the model which has the best performance in recognizing diseases, and only use its disease recognition results as part of the nal recognition results. 4 4.1
We use the CCKS-2017 Task 2 dataset to perform our experiments. The dataset contains 10,420 unannotated instances and 1,596 annotated instances with ve types of clinical named entities, including diseases, symptoms, exams, treatments, and body parts. The annotated instances are already partitioned into 1,198 training instances and 398 test instances. Each instance has one or several sentences. We further partition the training sentences, take 70% of them as training data, and the rest 30% as validation data. We score our methods by using the CCKS-2017 Task 2 o cial metrics, which computes F1-Measure. 4.2
As for the Bi-LSTM CRF model, we initialize character embeddings via
word2vec[
To explore the impact of di erent settings, we train Bi-LSTM CRF models using di erent tagging formats on training data both in clauses and sentences, and test them on validation data. The results are shown in Table 2. First, characterlevel models outperform word-level models. It is because word-level approaches may have segmentation error. What's more, the word set is much bigger than the character set. This means the corpus is not big enough to learn word embeddings e ectively. Second, the recognition of diseases, treatments, and body parts is not well performed, which indicates the necessity of self-taught learning and active learning. Third, di erent types of entities are recognized the best by di erent settings, respectively. This indicates the necessity of ensemble learning.
To dissect the e ectiveness of self-taught learning, active learning, and ensemble learning, we train Bi-LSTM CRF models in character level only, and test them on test data. Table 3 shows that the Bi-LSTM CRF model along with self-taught learning, active learning, and ensemble learning achieves the best performance in clinical NER, with a F1-Measure of 89.88%.
In this paper, we propose a Bi-LSTM CRF model along with self-taught learning, active learning, and ensemble learning to recognize clinical named entities. We exploit character embedding to deal with the ambiguity in the boundary of Chinese words, and employ self-taught learning and active learning to increase training data. After comparing di erent tagging schemes, we use ensemble learning to obtain the best recognition performance for all ve types of entities. The results achieved on CCKS-2017 Task 2 dataset ranks among the top systems. Acknowledgements. This work is supported by the 863 Program funded by China Ministry of Science and Technology (Program No.2015AA020107).