Introduction

Extracting and linking medical information from textual documents has attracted extensive interest from both academia and industry. Automatic matching of text phrases to medical concepts and corresponding classification codes is a highly important task for many clinical applications in the fields of health management and patient safety.

The International Classification of Diseases (ICD) is the diagnostic system that is used to monitor and classify causes of health problems and death and provide information for clinical purposes. Each medical concept is mapped onto a unique identifier which consists of a single alphabet prefix and several digits. Single alphabet prefix represents a class of common diseases (e.g. "J" covers diseases of the respiratory system, "V" covers external causes of morbidity) and digits represent specific type of disease (e.g. "J20.2" covers acute bronchitis Table 1. Examples of causes of death from the international classification of diseases. J189 Pneumonia, unspecified organism I48 Atrial fibrillation and flutter M726 Necrotizing fasciitis G20

Parkinson's disease F102 Alcohol dependence A419 Sepsis, unspecified organism D696 Thrombocytopenia, unspecified E119 Type 2 diabetes mellitus without complications V892 Person injured in unspecified motor-vehicle accident, traffic due to streptococcus", "V25" covers "Motorcycle rider injured in collision with railway train or railway vehicle"). Table 1 contains examples of ICD-10 codes.

Machine learning methods have been widely successful in various NLP applications including named entity recognition and relation extraction [1][2][3], machine translation [4][5][6], opinion mining [7][8][9], detection of demographic information from health-related user posts [10,11]. Recurrent Neural Networks (RNN), in particular, Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) are considered to be among the most powerful methods for sequence modeling [12][13][14]4]. Motivated by the recent success of deep recurrent networks, herein we have explored an application of RNN-based encoder-decoder models to the task of automated ICD coding.

We describe participation of our team in the task 1 for English death certificates. The goal of this task is to assign one or more relevant ICD-10 codes to sentences in the death certificates. We employ an annotated corpus named the CepiDC Causes of Death Corpus, which contains free-text descriptions of causes of death reported by physicians. More specifically, we employ the part of the corpus with English texts. The CepiDC corpus of French texts was initially provided for the task of ICD-10 coding in CLEF eHealth 2016 (task 2) [15,16]. The organizers recently extended this corpus with additional data for CLEF eHealth 2017 [17,18]. Our neural network relies on two sources of information: word representations learned from unannotated corpora and a manually curated ICD-10 dictionary provided by the organizers of the task.

The rest of the paper is structured as follows. Section 2 contains our system description, Section 3 provides evaluation results. In Section 4, we discuss some related work from CLEF eHealth 2016. Finally, Section 5 provides concluding remarks.

Our Approach

The basic idea of our approach is to map the input sequence to a fixed-sized vector, more precisely, some semantic representation of this input, and then unroll this representation in the target sequence using a neural network model. This intuition is formally captured in a encoder-decoder architecture. In the following subsections, we provide a brief description of recurrent neural networks (RNNs) and the encoder-decoder model.

Recurrent Neural Networks

RNNs are naturally used for sequence learning, where both input and output are word and label sequences, respectively. RNN has recurrent hidden states, which aim to simulate memory, i.e., the activation of a hidden state at every time step depends on the previous hidden state [12]. The recurrent unit computes a weighted sum of the input signal. There is the difficulty of training RNNs to capture long-term dependencies due to the effect of vanishing gradients [19], so the most widely used modification of a RNN unit is the Long Short-Term Memory (LSTM) [20]. LSTM provides the "constant error carousel" and does not preclude free gradient flow. The basic LSTM architecture contains three gates: input gate, forget gate, and output gate, together with a recurrent cell. LSTM cells are usually organized in a chain, with outputs of previous LSTMs connected to the inputs of subsequent LSTMs.

An important modification of the basic RNN architecture is bidirectional RNNs, where the past and the future context is available in every time step [13]. Bidirectional LSTMs, developed by Graves and Schmidhuber [14,21], contain two chains of LSTM cells flowing in both forward and backward direction, and the final representation is either a linear combination or simply concatenation of their states.

Encoder-Decoder Model

We introduce the sequence-to-sequence architecture, more precisely, an encoderdecoder model proposed earlier [4,6] for the ICD-10 coding task. As shown in Figure 1, the model consists of two components based on RNNs: an encoder and a decoder. The encoder processes the input sequence, while the decoder generates the output sequence. We adopted the architecture as described in [4]. As encoder RNN we used bidirectional LSTM, as decoder RNN we used left-to-right LSTM. The input layer of our model is vector representations of individual words. Word embedding models represent each word using a single real-valued vector. Such representation groups together words that are semantically and syntactically similar [22]. The word embeddings are trained using an unlabelled corpus of user reviews.

In order to incorporate prior knowledge, we additionally concatenated cosine similarities vector to the encoded state. CLEF participants were provided with a manually created dictionary. This dictionary named AmericanDictionary contains quadruplets (diagnosis text, codes Icd1, IcdC, Icd2). We only consider pairs (diagnosis text, Icd1) for our system since most entries in the dictionary are associated with these codes.

Cosine similarities vector was calculated as follows. First, for each ICD-10 code present in the dictionary a document was constructed by simply concatenating diagnosis texts belonging to that code. For the resulting document set, TF-IDF transformation was computed; thus, every ICD-10 code was provided with a vector representation. For a given input sequence, the TF-IDF vector representation was calculated. Using the vector representation of the input sequence and each ICD-10 code, vector of cosine similarities was constructed such as to have in the i-th position the cosine similarity measure between input sequence representation and i-th ICD-10 code representation.

We have made the implementation of our model available at the github repository1 .

Experiments

In this section, we discuss the performance of our LSTM-based encoder-decoder model for ICD coding.

Evaluation Dataset

The CLEF e-Health 2017 Task 1 participants were provided with data from 13,330 death certificates for training. Each certificate contains information about the demographic attributes of each person (gender, age), other metadata (e.g., a location of death) and one or more codes of the primary cause of death. Diagnostic statements with multiple codes were repeated for each code assigned by physicians. The test set contained 14,833 certificates.

The experiments were also carried out on the following sets:

1. The full version of the CepiDC test set named the "ALL" set.

2. The part of the full test set named the "EXTERNAL" set.

The "ALL" test set consists of texts associated with all ICD codes. The "EX-TERNAL" test set is limited to textual fragments with ICD codes linked with a particular type of deaths, called "external causes" or violent deaths. The "EX-TERNAL" set was selected due to two reasons: (i) there is a special interest for the public health policies that can target ICD codes specifically, e.g. suicide prevention; (ii) the semantic analysis of the context associated with these deaths is more complex in terms of comorbidity, affected people and language models used to describe the event. External causes are characterized by codes V01 to Y98. Please refer to the task overview paper [18] for more details.

Experimental Setting

Word embeddings We used the word embeddings trained on 2,5 millions of health-related reviews from [1]. Statistics of there reviews is presented in Table 2. The embeddings were trained with the Continuous Bag of Words model with the following parameters: vector size of 200, the length of local context of 10, negative sampling of 5, vocabulary cutoff of 10. Model tuning To find optimal neural network configuration and word embeddings, the 5-fold cross-validation procedure was applied to the training set. We compared architectures with different numbers of neurons in hidden layersof encoder and decoder LSTM. The best cross-validation F-score is obtained for the architecture with 600 neurons in the hidden layer of encoder LSTM and 1000 neurons in the hidden layer of the decoder LSTM. We tested bidirectional LSTM as decoder but did not achieve an improvement over the left-to-right LSTM. We also established that 10 epochs are enough for stable performance on the validation sets.

We have implemented networks with the Keras library [23]. LSTM is trained on top of the embedding layer. We use the 600-dimensional hidden layer for the encoder RNN chain. Finally, the last hidden state of LSTM chain output concatenated with cosine similarities vector is fed into a decoding LSTM layer with 1000-dimensional hidden layer and softmax activation. In order to prevent neural networks from overfitting, we used dropout of 0.5 [24]. We used categorical cross entropy as the objective function and the Adam optimizer [25] with the batch size of 20.

In addition, we have evaluated word embeddings trained on biomedical literature indexed in PubMed from [26] as well as on health-related reviews from [1]. Embeddings on health-related reviews showed better results during crossvalidation. We also tried to exploit meta-information along with cosine similarities vectors but we did not observe any significant improvement.

Results

Our neural models were evaluated on texts in English using common evaluation metrics such as precision, recall and balanced F-measure. We trained our model for 10 epochs (Run1) and 15 epochs (Run2). The reported results are presented in Tables 3 and 4. As shown in Tables 3 and 4 our performance results are significantly better than the average and median score of all submitted runs. The system obtained F-scores of 85.01% and 44.33% on the full test set and the "EXTERNAL" set, respectively. The difference of results on these sets is explained by a small number of codes in the latter case. The "ALL" set includes 18,928 codes (900 unique codes), while the "EXTERNAL" set includes only 126 codes (28 unique codes). We note that RNNs and word embeddings can be successfully applied to medical concept coding tasks without any task-specific feature engineering effort.

Related Work

Different approaches have been developed for ICD coding task, mainly falling into two categories: (i) knowledge-based methods [27][28][29]; and (ii) machine learning approaches [30,31].

In the CLEF eHealth 2016, five teams participated in the shared task 2 about the ICD-10 coding of death certificates in French [15]. Most methods utilized dictionary-based semantic similarity and, to some extent, string matching. Mulligen et al. [27] obtained the best results by combining a Solr tagger with ICD-10 terminologies. The terminologies were derived from the task training set and a manually curated ICD-10 dictionary. They achieved F-measure of 84.8%. Cabot et al. [28] applied an approximate string matching method and obtained F-measure of 68.0%. Mottin et al. [29] used a pattern matching approach and obtained F-measure of 55.4%. Dermouche et al. [30] applied two machine learning methods: (i) a supervised extension of Latent Dirichlet Allocation (LDA), i.e., Labeled-LDA and (ii) Support Vector Machine (SVM) based on bag-of-word features. For Labeled-LDA, they used ICD-10 codes from the training set as documents classes. The Labeled-LDA and SVM classifier archived F-measures of 73.53% and 75.19%, respectively. This study did not focus on designing effective features to obtain better classification performance. Zweigenbaum and Lavergne [31] proposed a classifier with TF-IDF transformer for tokens and used cosine similarity for ranking of classification codes. They studied the problem of learning to accurately rank a set of candidate codes obtained as a result of classification. The authors explored the effectiveness of several groups of features including meta-information and n-grams of normalized tokens. They focused only on statements which are associated with a singular code. The proposed approach obtained F-measure of 65.2% due to low recall of 56.8%. In recent work [32], Zweigenbaum and Lavergne utilized a hybrid method combining simple dictionary projection and mono-label supervised classification. They used Linear SVM trained on the full training corpus and the 2012 dictionary provided for CLEF participants. This hybrid method obtained an F-measure of 85.86%. Overall, the participants of task 2 did not use word embeddings or deep neural networks, which are proved useful in many natural language processing tasks.

Besides experiments on CLEF eHealth data sets, the medical concept coding task has also been studied by several researchers. Ontologies of medical concepts such as the Unified Medical Language System (UMLS) [33], SNOMED CT [34], and ICD-9 or ICD-10 are widely used for this task. In order to map texts to medical concepts in the UMLS, the National Library of Medicine (NLM) developed MetaMap [35]. This system is based on a linguistic approach using variants of terms and rules. Recent studies applied machine learning methods such as learning-to-rank methods [36] and convolutional neural networks [37]. Leaman et al. introduced a DNorm system based on pairwise learning-to-rank technique with a predefined set of features [36]. Features were based on a dictionary of diseases derived from the UMLS Metathesaurus. Recently, Limsopatham and Collier [37] experimented with convolutional and recurrent neural networks with pre-trained word embeddings for mapping social media texts to medical concepts. The authors observed that training can be effectively achieved at 40-70 epochs for corpora of tweets and user reviews. Experiments showed that both neural networks outperformed the DNorm system and a multi-class logistic regression. Word embeddings trained on a Google News corpus improved significantly over embeddings on medical articles downloaded from BioMed Central. In [1], using word embeddings trained on social media produces better scores than using embeddings trained on PubMed articles for disease named entity recognition. We also mark word embeddings trained on electronic health records [38][39][40] for future work.

Conclusion

In this paper, we have developed RNN-based encoder-decoder models for ICD-10 coding on Task 1 of the 2017 CLEF eHealth evaluation lab. Our results show that the neural network performs significantly better than the official median and average computed using the participants' runs, reaching F-measure of 85.01% on the full test set. In further studies, we plan to implement other encoderdecoder architectures and convolutional neural networks. We also plan to carry out a qualitative analysis on the extracted codes. Additionally, we would like to explore alternative distributed word representations trained on medical notes from electronic health records.