This paper describes the participation of the WBI team in the CLEF eHealth 2018 shared task 1 ("Multilingual Information Extraction -ICD-10 coding"). Our contribution focus on the setup and evaluation of a baseline language-independent neural architecture for ICD-10 classification as well as a simple, heuristic multi-language word embedding space. The approach builds on two recurrent neural networks models to extract and classify causes of death from French, Italian and Hungarian death certificates. First, we employ a LSTM-based sequenceto-sequence model to obtain a death cause from each death certificate line. We then utilize a bidirectional LSTM model with attention mechanism to assign the respective ICD-10 codes to the received death cause description. Both models take multi-language word embeddings as inputs. During evaluation our best model achieves an F-score of 0.34 for French, 0.45 for Hungarian and 0.77 for Italian. The results are encouraging for future work as well as the extension and improvement of the proposed baseline system.
Automatic extraction, classification and analysis of biological and medical concepts from unstructured texts, such as scientific publications or electronic health documents, is a highly important task to support many applications in research, daily clinical routine and policy-making. Computer-assisted approaches can improve decision making and support clinical processes, for example, by giving a more sophisticated overview about a research area, providing detailed information about the aetiopathology of a patient or disease patterns. In the past years major advances have been made in the area of natural-language processing (NLP). However, improvements in the field of biomedical text mining lag behind other domains mainly due to privacy issues and concerns regarding the processed data (e.g. electronic health records).
The CLEF eHealth lab1 attends to circumvent this situation through organization of various shared tasks to exploit electronically available medical content [34]. In particular, Task 12 of the lab is concerned with the extraction and classification of death causes from death certificates originating from different languages [27]. Participants were asked to classify the death causes mentioned in the certificates according to the International Classification of Disease version 10 (ICD-10) 3 . The task was concerned with French and English death certificates in previous years. In contrast, this year the organizers provided annotated death reports as well as ICD-10 dictionaries for French, Italian and Hungarian. The development of language-independent, multilingual approaches was encouraged.
Inspired by the recent success of recurrent neural network models (RNN) [6,20,10] in general and the convincing performance of the work from Miftahutdinov and Tutubalina [21] in the last edition of the lab, we opt for the development of a deep learning model for this year's competition. Our work introduces a prototypical, language independent approach for ICD-10 classification using multi-language word embeddings and long short-term memory models (LSTMs). We divide the proposed pipeline into two tasks. First, we perform named entity recognition (NER), i.e. extract the death cause description from a certificate line, with an an encoder-decoder model. Given the death cause, named entity normalization (NEN), i.e. assigning an ICD-10 code to extracted death cause, is performed by a separate LSTM. Our approach builds upon a heuristic multilanguage embedding space and therefore only needs one single model for all three data sets. With this work we want to experiment and evaluate which performance can be achieved with such a simple shared embedding space.
This section highlights previous work related to our approach. We give a brief introduction to the methodical foundations of our work, RNNs and word embeddings. The section concludes with a summary of ICD-10 classification approaches used in previous eHealth Lab competitions.
RNNs are a widely used technique for sequence learning problems such as machine translation [1,6], image captioning [2], NER [20,40], dependency parsing [10] and part-of-speech tagging [39]. RNNs model dynamic temporal behaviour in sequential data through recurrent units, i.e. the hidden, internal state of a unit in one time step depends on the state of the unit in the previous time step.
These feedback connections enable the network to memorize information from recent time steps and add the ability to capture long-term dependencies.
However, training of RNNs can be difficult due to the vanishing gradient problem [16,3]. The most widespread modifications of RNNs to overcome this problem are LSTMs [17] and gated recurrent units (GRU) [6]. Both modifications use gated memories which control and regulate the information flow between two recurrent units. A common LSTM unit consists of a cell and three gates, an input gate, an output gate and a forget gate. In general, LSTMs are chained together by connecting the outputs of the previous unit to the inputs of the next one.
A further extension of the general RNN architecture are bidirectional networks, which make the past and future context available in every time step. A bidirectional LSTM model consists of a forward chain, which processes the input data from left to right, and and backward chain, consuming the data in the opposite direction. The final representation is typically the concatenation or a linear combination of both states.
Distributional semantic models (DSMs) have been researched for decades in NLP [37]. Based on a huge amount of unlabeled texts, DSMs aim to represent words using a real-valued vector (also called embedding) which captures syntactic and semantic similarities between the words. Starting with the publication of the work from Collobert et al. [7] in 2011, learning embeddings for linguistic units, such as words, sentences or paragraphs, is one of the hot topics in NLP and a plethora of approaches have been proposed [4,23,28,30].
The majority of todays embedding models are based on deep learning models trained to perform some kind of language modeling task [29,30,32]. The most popular embedding model is the Word2Vec model introduced by Mikolov et al. [22,23]. They propose two shallow neural network models, continuous bag-ofwords (CBOW) and SkipGram, that are trained to reconstruct the context given a center word and vice versa. In contrast, Pennington et al. [28] use the ratio between co-occurrence probabilities of two words with another one to learn a vector representation. In [30] multi-layer, bi-directional LSTM models are utilized to learn word embeddings that also capture different contexts of it.
Several recent models focus on the integration of subword and morphological information to provide suitable representations even for unseen, out-ofvocabulary words. For example, Pinter et al. [32] try to reconstruct a pre-trained word embedding by learning a bi-directional LSTM model on character level. Similarly, Bojanowski et al. [4] adapt the SkipGram by taking character n-grams into account. Their fastText model assigns a vector representation to each character n-gram and represents words by summing over all of these representations of a word.
In addition to embeddings that capture word similarities in one language, multi-and cross-lingual approaches have also been investigated. Proposed methods either learn a linear mapping between monolingual representations [12,41] or utilize word- [13,38], sentence- [31] or document-aligned [36] corpora to build a shared embedding space.
The ICD-10 coding task has already been carried out in the 2016 [26] and 2017 [25] edition of the eHealth lab. Participating teams used a plethora of different approaches to tackle the classification problem. The methods can essentially be divided into two categories: knowledge-based [5,18,24] and machine learning (ML) approaches [8,11,15,21]. The former relies on lexical sources, medical terminologies and other dictionaries to match (parts of) the certificate text with entries from the knowledge-bases according to a rule framework. For example, Di Nunzio et al. [9] calculate a score for each ICD-10 dictionary entry by summing the binary or tf-idf weights of each term of a certificate line segment and assign the ICD-10 code with the highest score. In contrast, Ho-Dac et al. [14] treat the problem as information retrieval task and utilize the Apache Solr search engine4 to classify the individual lines.
The ML-based approaches employ a variety of techniques, e.g. Conditional Random Fields (CRFs) [15], Labeled Latent Dirichlet Analysis (LDA) [8] and Support Vector Machines (SVMs) [11] with diverse hand-crafted features. Most similar to our approach is the work from Miftahutdinov and Tutubalina [21], which achieved the best results for English certificates in the last year's competition. They use a neural LSTM-based encoder-decoder model that processes the raw certificate text as input and encodes it into a vector representation. Additionally, a vector which captures the textual similarity between the certificate line and the death causes of the individual ICD-10 codes is used to integrate prior knowledge into the model. The concatenation of both vector representations is then used to output the characters and numbers of the ICD-10 code in the decoding step. In contrast to their work, our approach introduces a model for multi-language ICD-10 classification. Moreover, we divide the task into two distinct steps: death cause extraction and ICD-10 classification.
Our approach models the extraction and classification of death causes as two-step process. First, we employ a neural, multi-language sequence-to-sequence model to receive a death cause description for a given death certificate line. We then use a second classification model to assign the respective ICD-10 codes to the obtained death cause. The remainder of this section gives a detailed explanation of the architecture of the two models.
The first step in our pipeline is the extraction of the death cause from a given certificate line. We use the training certificate lines (with their corresponding ICD-10 codes) and the ICD-10 dictionaries as basis for our model. The dictionaries provide us with death causes for each ICD-10 code. The goal of the model is to reassemble the dictionary death cause text from the certificate line.
For this we adopt the encoder-decoder architecture proposed in [35]. Figure 1 illustrates the architecture of the model. As encoder we utilize a unidirectional LSTM model, which takes the single words of a certificate line as inputs and scans the line from left to right. Each token is represented using pre-trained fastText5 word embeddings [4]. We utilize fastText embedding models for French, Italian and Hungarian trained on Common Crawl and Wikipedia articles 6 . Independently from the original language a word we represent it by looking up the word in all three embedding models and concatenate the obtained vectors. Through this we get a simple multi-language representation of the word. This heuristic composition constitutes a naive solution to build a multi-language embedding space. However we opted to evaluate this approach as a simple baseline for future work. Encoders' final state represents the semantic representation of the certificate line and serves as initial input for decoding process. Fig. 1. Illustration of the encoder-decoder model for death cause extraction. The encoder processes a death certificate line token-wise from left to right. The final state of the encoder forms a semantic representation of the line and serves as initial input for the decoding process. The decoder will be trained to predict the death cause text from the provided ICD-10 dictionaries word by word (using special tags \s and \e for start resp. end of a sequence). All input tokens will be represented using the concatenation of the fastText embeddings of all three languages.
For the decoder we utilize another LSTM model. The initial input of the decoder is the final state of the encoder model. Moreover, each token of the dictionary death cause text (padded with special start and end tag) serves as (sequential) input. Again, we use fastText embeddings of all three languages to represent the input tokens. The decoder predicts one-hot-encoded words of the death cause. During test time we use the encoder to obtain a semantic representation of the certificate line and decode the death cause description word by word starting with the special start tag. The decoding process finishes when the decoder outputs the end tag.
The second step in our pipeline is to assign a ICD-10 code to the generated death cause description. For this we employ a bidirectional LSTM model which is able to capture the past and future context for each token of a death cause description. Just as in our encoder-decoder model we encode each token using the concatenation of the fastText embeddings of the word from all three languages. To enable our model to attend to different parts of the death cause we add an extra attention layer [33] to the model. Through the attention mechanism our model learns a fixed-sized embedding of the death cause description by computing an adaptive weighted average of the state sequence of the LSTM model. This allows the model to better integrate information over time. Figure 2 presents the architecture of our ICD-10 classification model. We train the model using the provided ICD-10 dictionaries from all three languages. Hungarian Embedding Italian Embedding French Embedding Fig. 2. Illustration of the ICD-10 classification model. The model utilizes a bidirectional LSTM layer, which processes the death cause from left to right and vice versa. The attention layer summarizes the whole description by computing an adaptive weighted average over the LSTM states. The resulting death cause embedding will be feed through a softmax layer to get the final classification. Equivalent to our encoder-decoder model all input tokens will be represented using the concatenation of the fastText embeddings of all three languages.
In this section we will present experiments and obtained results for the two developed models, both individually as well as combined in a pipeline setting.
The CLEF e-Health 2018 Task 1 participants where provided with annotated death certificates for the three selected languages: French, Italian and Hungarian. Each of the languages is supported by training certificate lines as well as a dictionary with death cause descriptions resp. diagnosis for the different ICD-10 codes. The provided training data sets were imbalanced concerning the different languages: the Italian corpora consists of 49,823, French corpora of 77,348 7 and Hungarian corpora 323,175 certificate lines. We split each data set into a training and a hold-out evaluation set. The complete training data set was then created by combining the certificate lines of all three languages into one data set. Beside the provided certificate data we used no additional knowledge resources or annotated texts.
Due to time constraints during development no cross-validation to optimize the (hyper-) parameters and the individual layers of our models was performed. We either keep the default values of the hyper-parameters or set them to reasonable values according to existing work. During model training we shuffle the training instances and use varying instances to perform a validation of the epoch.
Pre-trained fastText word embeddings were trained using the following parameter settings: CBOW with position-weights, embedding dimension size 300, with character n-grams of length 5, a window of size 5 and 10 negative samples. Unfortunately, they are trained on corpora not related with the biomedical domain and therefore do not represent the best possible textual basis for an embedding space for biomedical information extraction. Final embedding space used by our models is created by concatenating individual embedding vectors for all three languages. Thus the input of our model is embedding vector of size 900. All models were implemented with the Keras8 library.
To identify possible candidates for a death cause description, we focus on the use of an encoder-decoder model. The encoder model uses an embedding layer with input masking on zero values and a LSTM layer with 256 units. The encoders' output is used as the initial state of the decoder model.
Based on the input description from the dictionary and a special start token, the decoder generates a death cause word by word. This decoding process continues until a special end token is generated. The entire model is optimized using the Adam optimization algorithm [19] and a batch size of 700. Model training was performed either for 100 epochs or until an early stopping criteria is met (no change in validation loss for two epochs).
As the provided data set are imbalanced regarding the tasks' languages, we devised two different evaluation settings: (1) DCEM-Balanced, where each language was supported by 49.823 randomly drawn instances (size of the smallest corpus) and ( 2) DCEM-Full, where all available data is used. Table 4
The classification model is responsible for assigning a ICD-10 code to death cause description obtained during the first step. Our model uses an embedding layer with input masking on zero values, followed by a bidirectional LSTM layer with 256 dimension hidden layer. Thereafter an attention layer builds an adaptive weighted average over all LSTM states. The respective ICD-10 code will be determined by a dense layer with softmax activation function. We use the Adam optimizer to perform model training. The model was validated on 25% of the data. As for the extraction model, no cross-validation or hyper-parameter optimization was performed.
Once again, we devised two approaches. This was mainly caused by the lack of adequate training data in terms of coverage for individual ICD-10 codes. Therefore, we defined two training data settings: (1) minimal (ICD-10 Minimal), where only ICD-10 codes with two or more supporting training instances are used. This leaves us with 6,857 unique ICD-10 codes and discards 2,238 unique codes with support of one. This, of course, minimizes the number of ICD-10 codes in the label space. Therefore, (2) an extended (ICD-10 Extended) data set was defined. Here, the original ICD-10 code mappings, found in the supplied dictionaries, are extended with the training instances from individual certificate data from the three languages. This generates 9,591 unique ICD-10 codes. Finally, for the remaining ICD-10 codes that have only one supporting description, we duplicate those data points.
The goal of this approach is to extend our possible label space to all available ICD-10 codes. The results obtained from the two approaches on the validation set are shown in Table 4.3. Using the minimal data set the model achieves an accuracy of 0.937. In contrast, using the extended data set the model reaches an accuracy of 0.954 which represents an improvement of 1.8%.
The two models where combined to create the final pipeline. We tested both death cause extraction models (based on the balanced and unbalanced data set) in the final pipeline, as their performance differs greatly. On the contrary, both ICD-10 classification models perform similarly, so we just used the extended ICD-10 classification model, with word level tokens9 , in the final pipeline. To evaluate the pipeline we build a training and a hold-out validation set during development. The obtained results on the validation set are presented in Table 4.4. The scores are calculated using a prevalence-weighted macro-average across the output classes, i.e. we calculated precision, recall and F-score for each ICD-10 code and build the average by weighting the scores by the number occurrences of the code in the gold standard.
Although the individual models, as shown in Tables 4.2 and 4.3 are promising, the performance decreases considerably in a pipeline setting . The pipeline model based on the balanced data set reaches a F-score of 0.61, whereas the full model achieves a slightly higher value of 0.63. Both model configurations have a higher precision than recall (0.73/0.61 resp. 0.74/0.62).
This can be contributed to several factors. First of all, a pipeline architecture always suffers from error-propagation, i.e. errors in a previous step will influence the performance of the following layers and generally lower the performance of the overall system. Investigating the obtained results, we found that the imbalanced distribution of ICD-10 codes represents one the main problems. This
Recall F-score Final-Balanced 0.73 0.61 0.61 Final-Full 0.74 0.62 0.63 Table 3. Evaluation results of the final pipeline on the validation set of the training data. Reported figures represent the prevalence-weighted macro-average across the output classes. Final-Balanced = DCEM-Balanced + ICD-10 Extended. Final-Full = DCEM-Full + ICD-10 Extended severely impacts the decoder-encoder architecture used here as the token generation is biased towards the available data points. Therefore the models misclassify certificate lines associated with ICD-10 codes that only have a small number of supporting training instances very often.
Results obtained on the test data set, resulting from the two submitted official runs, are shown in Table 4. Similar to the evaluation results during development, the model based on the full data set performs slightly better than the model trained on the balanced data set. The full model reaches a F-score of 0.34 for French, 0.45 for Hungarian and 0.77 for Italian. All of our approaches perform below the mean and median averages of all participants. Surprisingly, there is a substantial difference in results obtained between the individual languages. This confirms our assumptions about the (un-) suitability of the proposed multi-lingual embedding space for this task. The results also suggest that the size of the training corpora is not influencing the final results. As seen, best results were obtained on the Italian data set were trained on the smallest corpora. Worst results were obtained on the middle, French, corpus while the biggest corpus, Hungarian, is in second place.
We identified several possible reasons for the obtained results. These also represent (possible) points for future work. One of the main disadvantages of our approach is the quality of the used word embeddings as well as the properties of the proposed language-independent embedding space. The usage of out-ofdomain word embeddings which aren't targeted to the biomedical domain are likely a suboptimal solution to this problem. We tried to alleviate this by finding suitable external corpora to train domain-dependent word embeddings for each of the supported languages, however we were unable to find any significant amount of in-domain documents (e.g. PubMed search for abstracts in either French, Hungarian or Italian found 7843, 786 and 1659 articles respectively). Furthermore, we used a simple, heuristic solution by just concatenating the embeddings of all three languages to build a shared vector space.
Besides the issues with the used word embeddings, the inability to obtain full ICD-10 dictionaries for the selected languages has also negatively influenced the results. As a final limitation to our approach, lack of multi-label classification support has also been identified (i.e. not recognizing more than one death cause in a single input text). 4. Test results of the final pipeline. Final-Balanced = DCEM-Balanced + ICD-10 Extended. Final-Full = DCEM-Full + ICD-10 Extended
In this paper we tackled the problem of information extraction of death causes in an multilingual environment. The proposed solution was focused on the setup and evaluation of an initial language-independent model which relies on a heuristic mutual word embedding space for all three languages. The proposed pipeline is divided in two steps: possible token describing the death cause are generated by using a sequence to sequence model first. Afterwards the generated token sequence is normalized to a ICD-10 code using a distinct LSTM-based classification model with attention mechanism. During evaluation our best model achieves an F-score of 0.34 for French, 0.45 for Hungarian and 0.77 for Italian. The obtained results are encouraging for further investigation however can't compete with the solutions of the other participants yet.
We detected several issues with the proposed pipeline. These issues serve as prospective future work to us. First of all the representation of the input words can be improved in several ways. The word embeddings we used are not optimized to the biomedical domain but are trained on general text. Existing work was proven that in-domain embeddings improve the quality of achieved results. Although this was our initial approach, the difficulties of finding adequate in-domain corpora for selected languages has proven to be to a hard to tackle. Moreover, the multi-language embedding space is currently heuristically defined as concatenation of the three word embeddings models for individual tokens. Creating an unified embedding space would create a truly language-independent token representation. The improvement of the input layer will be the main focus of our future work.
The ICD-10 classification step also suffers from lack of adequate training data. Unfortunately, we were unable to obtain extensive ICD-10 dictionaries for all languages and therefore can't guarantee the completeness of the ICD-10 label space. Another disadvantage of the current pipeline is the missing support for multi-label classification.