This paper describes the participation of the WBI team in the CLEF eHealth 2018 shared task 1 (\Multilingual Information Extraction - ICD-10 coding"). Our contribution focus on the setup and evaluation of a baseline language-independent neural architecture for ICD-10 classi cation as well as a simple, heuristic multi-language word embedding space. The approach builds on two recurrent neural networks models to extract and classify causes of death from French, Italian and Hungarian death certi cates. First, we employ a LSTM-based sequenceto-sequence model to obtain a death cause from each death certi cate line. We then utilize a bidirectional LSTM model with attention mechanism to assign the respective ICD-10 codes to the received death cause description. Both models take multi-language word embeddings as inputs. During evaluation our best model achieves an F-score of 0.34 for French, 0.45 for Hungarian and 0.77 for Italian. The results are encouraging for future work as well as the extension and improvement of the proposed baseline system.
Automatic extraction, classi cation and analysis of biological and medical concepts from unstructured texts, such as scienti c publications or electronic health documents, is a highly important task to support many applications in research, daily clinical routine and policy-making. Computer-assisted approaches can improve decision making and support clinical processes, for example, by giving a more sophisticated overview about a research area, providing detailed information about the aetiopathology of a patient or disease patterns. In the past years major advances have been made in the area of natural-language processing (NLP). However, improvements in the eld of biomedical text mining lag behind other domains mainly due to privacy issues and concerns regarding the processed data (e.g. electronic health records).
The CLEF eHealth lab1 attends to circumvent this situation through
organization of various shared tasks to exploit electronically available medical content
[
Inspired by the recent success of recurrent neural network models (RNN)
[
This section highlights previous work related to our approach. We give a brief introduction to the methodical foundations of our work, RNNs and word embeddings. The section concludes with a summary of ICD-10 classi cation approaches used in previous eHealth Lab competitions. 2.1
Recurrent neural networks (RNN)
RNNs are a widely used technique for sequence learning problems such as
machine translation [
extraction-icd10-coding
These feedback connections enable the network to memorize information from recent time steps and add the ability to capture long-term dependencies.
However, training of RNNs can be di cult due to the vanishing gradient
problem [
A further extension of the general RNN architecture are bidirectional networks, which make the past and future context available in every time step. A bidirectional LSTM model consists of a forward chain, which processes the input data from left to right, and and backward chain, consuming the data in the opposite direction. The nal representation is typically the concatenation or a linear combination of both states. 2.2
Word Embeddings
Distributional semantic models (DSMs) have been researched for decades in NLP
[
The majority of todays embedding models are based on deep learning models
trained to perform some kind of language modeling task [
Several recent models focus on the integration of subword and
morphological information to provide suitable representations even for unseen,
out-ofvocabulary words. For example, Pinter et al. [
In addition to embeddings that capture word similarities in one language,
multi- and cross-lingual approaches have also been investigated. Proposed
methods either learn a linear mapping between monolingual representations [
ICD-10 Classi cation
The ICD-10 coding task has already been carried out in the 2016 [
The ML-based approaches employ a variety of techniques, e.g. Conditional
Random Fields (CRFs) [
Our approach models the extraction and classi cation of death causes as two-step process. First, we employ a neural, multi-language sequence-to-sequence model to receive a death cause description for a given death certi cate line. We then use a second classi cation model to assign the respective ICD-10 codes to the obtained death cause. The remainder of this section gives a detailed explanation of the architecture of the two models. 3.1
Death Cause Extraction Model The rst step in our pipeline is the extraction of the death cause from a given certi cate line. We use the training certi cate lines (with their corresponding
ICD-10 codes) and the ICD-10 dictionaries as basis for our model. The dictionaries provide us with death causes for each ICD-10 code. The goal of the model is to reassemble the dictionary death cause text from the certi cate line.
For this we adopt the encoder-decoder architecture proposed in [
Encoder
Decoder Input colite infectieuse ou ischemique \s
For the decoder we utilize another LSTM model. The initial input of the decoder is the nal state of the encoder model. Moreover, each token of the 5 https://github.com/facebookresearch/fastText/ 6 https://github.com/facebookresearch/fastText/blob/master/docs/crawlvectors.md dictionary death cause text (padded with special start and end tag) serves as (sequential) input. Again, we use fastText embeddings of all three languages to represent the input tokens. The decoder predicts one-hot-encoded words of the death cause. During test time we use the encoder to obtain a semantic representation of the certi cate line and decode the death cause description word by word starting with the special start tag. The decoding process nishes when the decoder outputs the end tag. 3.2
ICD-10 Classi cation Model
The second step in our pipeline is to assign a ICD-10 code to the generated
death cause description. For this we employ a bidirectional LSTM model which
is able to capture the past and future context for each token of a death cause
description. Just as in our encoder-decoder model we encode each token using the
concatenation of the fastText embeddings of the word from all three languages.
To enable our model to attend to di erent parts of the death cause we add
an extra attention layer [
Input Embedding Bi - LSTM Attention Softmax Output colite ischémique
sigmoïde Death Cause Emb.
K559 French Embedding
Italian Embedding
Hungarian Embedding Fig. 2. Illustration of the ICD-10 classi cation model. The model utilizes a bidirectional LSTM layer, which processes the death cause from left to right and vice versa. The attention layer summarizes the whole description by computing an adaptive weighted average over the LSTM states. The resulting death cause embedding will be feed through a softmax layer to get the nal classi cation. Equivalent to our encoder-decoder model all input tokens will be represented using the concatenation of the fastText embeddings of all three languages.
In this section we will present experiments and obtained results for the two developed models, both individually as well as combined in a pipeline setting. 4.1
Training Data and Experiment Setup The CLEF e-Health 2018 Task 1 participants where provided with annotated death certi cates for the three selected languages: French, Italian and Hungarian. Each of the languages is supported by training certi cate lines as well as a dictionary with death cause descriptions resp. diagnosis for the di erent ICD-10 codes. The provided training data sets were imbalanced concerning the di erent languages: the Italian corpora consists of 49,823, French corpora of 77,3487 and Hungarian corpora 323,175 certi cate lines. We split each data set into a training and a hold-out evaluation set. The complete training data set was then created by combining the certi cate lines of all three languages into one data set. Beside the provided certi cate data we used no additional knowledge resources or annotated texts.
Due to time constraints during development no cross-validation to optimize the (hyper-) parameters and the individual layers of our models was performed. We either keep the default values of the hyper-parameters or set them to reasonable values according to existing work. During model training we shu e the training instances and use varying instances to perform a validation of the epoch.
Pre-trained fastText word embeddings were trained using the following parameter settings: CBOW with position-weights, embedding dimension size 300, with character n-grams of length 5, a window of size 5 and 10 negative samples. Unfortunately, they are trained on corpora not related with the biomedical domain and therefore do not represent the best possible textual basis for an embedding space for biomedical information extraction. Final embedding space used by our models is created by concatenating individual embedding vectors for all three languages. Thus the input of our model is embedding vector of size 900. All models were implemented with the Keras8 library. 4.2
Death cause extraction model To identify possible candidates for a death cause description, we focus on the use of an encoder-decoder model. The encoder model uses an embedding layer with input masking on zero values and a LSTM layer with 256 units. The encoders' output is used as the initial state of the decoder model.
Based on the input description from the dictionary and a special start token,
the decoder generates a death cause word by word. This decoding process
continues until a special end token is generated. The entire model is optimized using
the Adam optimization algorithm [
was performed either for 100 epochs or until an early stopping criteria is met (no change in validation loss for two epochs).
As the provided data set are imbalanced regarding the tasks' languages, we devised two di erent evaluation settings: (1) DCEM-Balanced, where each language was supported by 49.823 randomly drawn instances (size of the smallest corpus) and (2) DCEM-Full, where all available data is used. Table 4.2 shows the results obtained on the training and validation set. The gures indicate that the distribution of training instances per language have a huge in uence on the performance of the model. The model trained on the full training data achieves an accuracy of 0.678 on the validation set. In contrast using the balanced data set the model reaches an accuracy of 0.899 (+ 32.5%).
Setting Trained Epochs AccuraTcryain Loss AccVuraalicdyatioLnoss DCEM-Balanced 18 0.958 0.205 0.899 0.634 DCEM-Full 9 0.709 0.098 0.678 0.330 Table 1. Experiment results of our death cause extraction sequence-to-sequence model concerning balanced (equal number of training instances per language) and full data set setting. 4.3
ICD-10 Classi cation Model The classi cation model is responsible for assigning a ICD-10 code to death cause description obtained during the rst step. Our model uses an embedding layer with input masking on zero values, followed by a bidirectional LSTM layer with 256 dimension hidden layer. Thereafter an attention layer builds an adaptive weighted average over all LSTM states. The respective ICD-10 code will be determined by a dense layer with softmax activation function. We use the Adam optimizer to perform model training. The model was validated on 25% of the data. As for the extraction model, no cross-validation or hyper-parameter optimization was performed.
Once again, we devised two approaches. This was mainly caused by the lack of adequate training data in terms of coverage for individual ICD-10 codes. Therefore, we de ned two training data settings: (1) minimal (ICD-10 Minimal), where only ICD-10 codes with two or more supporting training instances are used. This leaves us with 6,857 unique ICD-10 codes and discards 2,238 unique codes with support of one. This, of course, minimizes the number of ICD-10 codes in the label space. Therefore, (2) an extended (ICD-10 Extended) data set was de ned. Here, the original ICD-10 code mappings, found in the supplied dictionaries, are extended with the training instances from individual certi cate data from the three languages. This generates 9,591 unique ICD-10 codes. Finally, for the remaining ICD-10 codes that have only one supporting description, we duplicate those data points.
The goal of this approach is to extend our possible label space to all available ICD-10 codes. The results obtained from the two approaches on the validation set are shown in Table 4.3. Using the minimal data set the model achieves an accuracy of 0.937. In contrast, using the extended data set the model reaches an accuracy of 0.954 which represents an improvement of 1.8%.
ICD-10 Minimal ICD-10 Extended*
The two models where combined to create the nal pipeline. We tested both death cause extraction models (based on the balanced and unbalanced data set) in the nal pipeline, as their performance di ers greatly. On the contrary, both ICD-10 classi cation models perform similarly, so we just used the extended ICD-10 classi cation model, with word level tokens9, in the nal pipeline. To evaluate the pipeline we build a training and a hold-out validation set during development. The obtained results on the validation set are presented in Table 4.4. The scores are calculated using a prevalence-weighted macro-average across the output classes, i.e. we calculated precision, recall and F-score for each ICD-10 code and build the average by weighting the scores by the number occurrences of the code in the gold standard.
Although the individual models, as shown in Tables 4.2 and 4.3 are promising, the performance decreases considerably in a pipeline setting . The pipeline model based on the balanced data set reaches a F-score of 0.61, whereas the full model achieves a slightly higher value of 0.63. Both model con gurations have a higher precision than recall (0.73/0.61 resp. 0.74/0.62).
This can be contributed to several factors. First of all, a pipeline architecture always su ers from error-propagation, i.e. errors in a previous step will in uence the performance of the following layers and generally lower the performance of the overall system. Investigating the obtained results, we found that the imbalanced distribution of ICD-10 codes represents one the main problems. This
their performance fared poorly compared to the word level tokens.
Final-Balanced Final-Full severely impacts the decoder-encoder architecture used here as the token generation is biased towards the available data points. Therefore the models misclassify certi cate lines associated with ICD-10 codes that only have a small number of supporting training instances very often.
Results obtained on the test data set, resulting from the two submitted o cial runs, are shown in Table 4. Similar to the evaluation results during development, the model based on the full data set performs slightly better than the model trained on the balanced data set. The full model reaches a F-score of 0.34 for French, 0.45 for Hungarian and 0.77 for Italian. All of our approaches perform below the mean and median averages of all participants.
Surprisingly, there is a substantial di erence in results obtained between the individual languages. This con rms our assumptions about the (un-) suitability of the proposed multi-lingual embedding space for this task. The results also suggest that the size of the training corpora is not in uencing the nal results. As seen, best results were obtained on the Italian data set were trained on the smallest corpora. Worst results were obtained on the middle, French, corpus while the biggest corpus, Hungarian, is in second place.
We identi ed several possible reasons for the obtained results. These also represent (possible) points for future work. One of the main disadvantages of our approach is the quality of the used word embeddings as well as the properties of the proposed language-independent embedding space. The usage of out-ofdomain word embeddings which aren't targeted to the biomedical domain are likely a suboptimal solution to this problem. We tried to alleviate this by nding suitable external corpora to train domain-dependent word embeddings for each of the supported languages, however we were unable to nd any signi cant amount of in-domain documents (e.g. PubMed search for abstracts in either French, Hungarian or Italian found 7843, 786 and 1659 articles respectively). Furthermore, we used a simple, heuristic solution by just concatenating the embeddings of all three languages to build a shared vector space.
Besides the issues with the used word embeddings, the inability to obtain full ICD-10 dictionaries for the selected languages has also negatively in uenced the results. As a nal limitation to our approach, lack of multi-label classi cation support has also been identi ed (i.e. not recognizing more than one death cause in a single input text).
French Hungarian Italian
Final-Balanced Final-Full Baseline Average Median Final-Balanced Final-Full Baseline Average Median Final-Balanced Final-Full Baseline Average Median In this paper we tackled the problem of information extraction of death causes in an multilingual environment. The proposed solution was focused on the setup and evaluation of an initial language-independent model which relies on a heuristic mutual word embedding space for all three languages. The proposed pipeline is divided in two steps: possible token describing the death cause are generated by using a sequence to sequence model rst. Afterwards the generated token sequence is normalized to a ICD-10 code using a distinct LSTM-based classi cation model with attention mechanism. During evaluation our best model achieves an F-score of 0.34 for French, 0.45 for Hungarian and 0.77 for Italian. The obtained results are encouraging for further investigation however can't compete with the solutions of the other participants yet.
We detected several issues with the proposed pipeline. These issues serve as prospective future work to us. First of all the representation of the input words can be improved in several ways. The word embeddings we used are not optimized to the biomedical domain but are trained on general text. Existing work was proven that in-domain embeddings improve the quality of achieved results. Although this was our initial approach, the di culties of nding adequate in-domain corpora for selected languages has proven to be to a hard to tackle. Moreover, the multi-language embedding space is currently heuristically de ned as concatenation of the three word embeddings models for individual tokens. Creating an uni ed embedding space would create a truly language-independent token representation. The improvement of the input layer will be the main focus of our future work.
The ICD-10 classi cation step also su ers from lack of adequate training data. Unfortunately, we were unable to obtain extensive ICD-10 dictionaries for all languages and therefore can't guarantee the completeness of the ICD-10 label space. Another disadvantage of the current pipeline is the missing support for multi-label classi cation.