In this paper, We presents a Chinese medical term recognition system submitted to the competition held by China Conference on Knowledge Graph and Semantic Computing. I compare the performance of Linear Chain Conditional Random Field (CRF) with that of Bi-Directional Long Short Term Memory (LSTM) with Convolutional Neural Network (CNN) and CRF layers performance and find that CRF with augmented features performs best with F1 0.927 on the offline competition dataset using cross-validation. Hence, this system was built by using a conditional random field model with linguistic features such as character identity, N-gram, and external dictionary features.
named entities do not follow any nomenclature, which makes rule-based methods hard to be perfect. Besides, rule-based systems require domain experts, and they are not flexible to other NE types and domains.
Machine learning methods are more robust and they can identify potential
biomedical entities which are not previously included in standard dictionaries. More
and more machine learning methods are explored to solve the Bio-NER problem, such
as Hidden Markov Model (HMM), Support Vector Machine (SVM), Maximum
Entropy Markov Model (MEMM), and Conditional Random Fields (CRF)
Further, many deep learning methods are employed to tag sequence data. For
example, Convolutional network based models (Collobert et al., 2011) have been
proposed to tackle sequence tagging problem. Such model consists of a convolutional
network and a CRF layer on the output. In speech language understanding community,
recurrent neural network
In this paper, I make a comparison between classical statistical machine learning method- Conditional Random Fields and deep learning method – Bi-LSTM with CNN and CRF layers in terms of their performance on the competition dataset. 3
We make comparison between the performance of Conditional Random Field with designed linguistic features and that of Bi-directional LSTM with CNN and CRF layer on the competition dataset. CRF experiment is carried out using Python sklearncrfsuite 0.3 package, LSTM is conducted using Tensorflow r1.2. 3.1
Labeling In order to conduct supervised learning, we label the Chinese character sequence, There are 5 kinds of entities, which are encoded into 0 to 4. Since we label data on character level and entities usually consist of multiple characters, we label the beginning character of the entity B- with corresponding coded category, the rest of the entity character I- with corresponding coded category. If a character is not part of the entity, we label this character O. For example:”右肩左季肋部” is an entity belongs to Body Parts category. This entity is labeled as followed: 右 B-4, 肩I-4, 左I-4, 季I-4, 肋I-4, 部I-4 3.2
Conditional Random Field Conditional Random Fields (CRFs) are undirected statistical graphical models, a special case of which is a linear chain that corresponds to a conditionally trained finite-state machine. Such models are well suited to sequence analysis, and CRFs in particular have been shown to be useful in part-of-speech tagging, shallow parsing, and named entity recognition for newswire data.
Let o=o1, o2 ,...on be an sequence of observed words of length n. Let S be a set of states in a finite state machine, each corresponding to a label l L , Let s s1, s2 ,...sn be the sequence of states in S that correspond to the labels assigned
P(s | o) to words in the input sequence o. Linear chain CRFs define the conditional probability of a state sequence given an input sequence to be: 1 n exp(
m j1 j f j (si1, si , o, i))
Z0 i1 where Z0 is a normalization factor of all state sequences, f j (si1, si , o, i) is one of m functions that describes a feature, and j is a learned weight for each such feature function. This paper considers the case of CRFs that use a first order Markov independence assumption with binary feature functions.
Intuitively, the learned feature weight j for each feature f j should be positive for features that are correlated with the target label, negative for features that are anticorrelated with the label, and near zero for relatively uninformative features. These weights are set to maximize the conditional log likelihood of labeled sequences in a training set D {o, l(1) ,..., o, l(n)} :
n m LL(D) log(P(l(i) | o(i) ))
i1 j1 2 2
When the training state sequences are fully labeled and unambiguous, objective function is convex, thus the model is guaranteed to find the optimal weight settings in terms of LL(D). Once these settings are found, the labeling for a new, unlabeled sequence can be done using a modified Viterbi algorithm. CRFs are presented in more complete detail by Lafferty et al. (2001). 2 j 3.2
Bi-directional LSTM with CNN and CRF layer
Convolution is widely used in sentence modeling to extract features. Generally, let l and d be the length of sentence and word vector, respectively. Let C dl be the sentence matrix. A convolution operation involves a convolutional kernel H dw which is applied to a window of w words to produce a new feature. For instance, a feature ci is generated from a window of words C , i : i w by ci ( (C , i : i w H ) b)
Here b is a bias term and is a non-linear function, normally tanh or ReLu. is the Hadamard product between two matrices. The convolutional kernel is applied to each possible window of words in the sentence to produce a feature map. c [c1, c2 ,..., clw1] with c lw1 .
Next, I apply pairwise max pooling operation over the feature map to capture the most important feature. The pooling operation can be considered as feature selection in natural language processing.
Specifically, the output of convolution, the feature map c = [c1, c2 ,..., clw1]is the input of the pooling operation. The adjacent two features in the feature map be calculated as follows: p max(ci1, ci )
i
The output of the max pooling operation is p [ p1, p2 ,..., pl ] , p l pi captures the neighborhood information around character i within a window of specified step size. If I apply 100 different kernels, 2 different step sizes to extract features, then p will become 200l . Then I concatenate convolutional features to their corresponding original character features (word2vec features) to get feature sentence matrix.
Long Short Term Memory is a special kind of Recurrent Neural Network. It can maintain a memory based on history information using purpose-built memory cells, which enables the model to predict the current output conditioned on long distance features. LSTM memory cell is implemented as the following: it (Wxi xi Whi ht1 Wcict1 b i ) ft (Wxf xt Whf ht1 Wcf ct1 bf ) ct ft ct 1 it tanh(Wxc xt W hc ht 1 bc ) ot (Wxo xt Whoht1 Wcoct bo ) ht ot tanh(ct )
Where sigma is logistic sigmoid function and i, f, o and c are the input gate, forget gate, output gate, and cell vectors all of which are the same size as the hidden vector h. The weight matrix subscripts have the meaning as the name suggests. For example, Whi is the hidden-input gate matrix, Wxo is the input-output gate matrix etc. The weight matrices from the cell to gate vectors (e.g. Wci) are diagonal, so element m in each gate vector only receives input from element m of the cell vector. LSTM cell’s structure is illustrated in Figure 1.
Fig. 1 - LSTM CELL
Here we use bi-directional LSTM as proposed in
4.1
Features Template For CRF For convenience, features are generally organized into some groups called feature templates. For example, a bigram feature template C1 stands for the next character occurring in the corpus after each character.
We use dictionary such as ICD10, ICD9, and other Medicine, Pathology dictionary (72000 terms in total)to summarize common bigram, trigram, 4gram prefix and suffix. For example, if a bigram from sentence appears in common prefix or suffix, we make this feature 1 otherwise 0. 4.2
Hyperparameter Tuning In Conditional Random Field, we use Elastic Nets as regularizing term and set optimization algorithm as LBFGS and maximum iteration as 500. After random search to tune the regularization coefficients C1, C2, I get the best C1,C2 as 0.089 and 0.004.
As for deep learning models, we set the parameters as below: We use cross-validation to evaluate F1 performance across models, here list only one validation result for your reference. Models Precision Conditional Random Field(only character 92.85% features) Conditional Random Field(all template 93.10% features) Bi-directional LSTM+CRF layer 84.19% Bi-directional LSTM with CNN, CRF layers 90.31%
Using the hyperparameters listed above, we could see with basic character features such as unigram, bigram, trigram, CRF is able to perform better than bidirectional LSTM with CRF layers. With other augmented features, CRF’s performance could be improved further. Convolutional Neural Network layers could help bi-directional LSTM extract features better and improve its performance, but still not as good as CRF. 5
Conditional Random Field with augmented features performs better in my experiment compared to Bi-directional LSTM with CNN and CRF layers in terms of F1 performance. Hence I used CRF with augmented features model for the competition. Future works will concentrate on hyperparameter tuning for deep learning models to get a better sense of how good the model is.
We want to give special thanks to Dr. Liang Tao’s guidance during this competition.