MLT-DFKI at CLEF eHealth 2019: Multi-label Classi cation of ICD-10 Codes with BERT

Abstract

With the adoption of electronic health record (EHR) systems, hospitals and clinical institutes have access to large amounts of heterogeneous patient data. Such data consists of structured (insurance details, billing data, lab results etc.) and unstructured (doctor notes, admission and discharge details, medication steps etc.) documents, of which, latter is of great signi cance to apply natural language processing (NLP) techniques. In parallel, recent advancements in transfer learning for NLP has pushed the state-of-the-art to new limits on many language understanding tasks. Therefore, in this paper, we present team DFKIMLT's participation at CLEF eHealth 2019 Task 1 of automatically assigning ICD-10 codes to non-technical summaries (NTSs) of animal experiments where we use various architectures in multi-label classi cation setting and demonstrate the e ectiveness of transfer learning with pre-trained language representation model BERT (Bidirectional Encoder Representations from Transformers) and its recent variant BioBERT. We rst translate task documents from German to English using automatic translation system and then use BioBERT which achieves an F1-micro of 73.02% on submitted run as evaluated by the challenge.

Keywords

• Semantic Indexing
• Transfer Learning
• Multi-label Classi cation
• ICD-10 Codes

EHR systems o er rich source of data that can be utilized to improve health care systems by applying information extraction, representation learning and

?On behalf of the PRECISE4Q consortium Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CLEF 2019, 9-12 September 2019, Lugano, Switzerland. predictive modeling [ 32 ] techniques. Among many other applications, one such task is the automatic assignment of International Statistical Classi cation of

Diseases (ICD) codes [27] to clinical notes, otherwise called semantic indexing of

clinical documents [ 9 ]. The problem is to learn a mapping from natural language free-texts to medical concepts such that, given a new document, the system can assign one or more codes to it. Approximating the mapping in this setting can be seen as multi-label classi cation and is one way to solve the problem, besides hierarchical classi cation [ 33 ], learning to rank and unsupervised methods.

In this study, we describe our work on CLEF eHealth 2019 [19] Task 1 [26],

which is about multilingual information extraction from German non-technical summaries (NTSs) of animal experiments collected from AnimalTestInfo database to classify according to ICD-10 codes, German modi cation version 2016 1. The AnimalTestInfo database was developed in Germany to make the non-technical summaries (NTSs) of animal research studies available in a searchable and easily accessible web-based format. Each NTS was manually assigned an ICD-10 code with the goal of advancing the integrity and reporting of responsible animal research [ 5 ]. This task requires an automated approach to classify the NTSs, whereby the data exhibits challenging attributes of multilingualism, domain speci city and codes skewness with hierarchical structure.

We explore various models, starting with traditional bag-of-words support vector machines (SVM) to standard deep learning architectures of convolutional neural networks (CNN) and recurrent neural networks (RNN) with three types of attention mechanisms; namely, hierarchical attention Gated Recurrent Unit (GRU) [ 8 ], self-attention Long-Short Term Memory (LSTM) [ 14 ], and codes attentive LSTM. Finally, we show the e ectiveness of ne-tuning state-of-the-art pre-trained BERT models [ 10, 22 ], which requires minimal task speci c changes and works well for small datasets. However, the signi cant performance boost comes from translating the German NTSs to English and then applying the same models, yielding an absolute gain of 6.22% f-score on dev set, from best German model to English model. This can be attributed to the fact that each language has its own linguistic and cultural characteristics that may contain di erent signals to e ectively classify a speci c class [ 1 ]. Given translated texts, we also nd that domain speci c embeddings have more e ect when considering static word embeddings [ 25 ], giving an avg. gain of 2.77% over contextual embeddings [ 34 ], where the gain is 0.86%. 2

Related Work

Automatic assignment of ICD codes [ 9 ] to health related documents has been well studied, both in previous CLEF shared tasks and in general. Traditional approaches range from rule based and dictionary look ups [ 6 ] to machine learning models [ 12 ]. However, more recently the focus has been on applying deep learning. 1 https://www.dimdi.de/static/de/klassifikationen/icd/icd-10-gm/ kode-suche/htmlgm2016/

Many techniques have been proposed using CNNs, RNNs and hybrid systems. [ 11 ] uses shallow CNN and improves its predictions for rare labels by dictionary-based lexical matching. [ 4 ] addresses the challenges of long documents and high cardinality of label space in MIMIC-III [ 17 ] by modifying Hierarchical Attention Network [ 39 ] with labels attention. More recent focus has been on using sequence-to-sequence (seq2seq) [ 35 ] based encoder-decoder based architectures. [ 31 ] rst builds a multilingual death cause extraction model using

LSTMs encoder-decoder, with concatenated French, Hungarian and Italian fast

Text emebddings as inputs and causes extracted from ICD-10 dictionaries as outputs. The output representations are then passed to an attention based biLSTM classi er which predicts the codes. [ 15 ] uses character level CNN [41] encoders for French and Italian, which are genealogically related languages and similar on a character level, with a biRNN decoder. [ 16 ] enriches word embeddings with language-speci c Wikipedia and creates an ensemble model from a CNN classier and GRU encoder-decoder. Few other techniques have also been proposed to use sequence-to-sequence framework and obtained good results [ 2, 24 ].

While successful, these approaches make an auto-regressive assumption on

output codes, which may hold true only when there is one distinct path from parent to child code for a given document. However, in the ICD codes assignment, a document can have multiple disjoint paths in a directed acyclic graph (DAG), formed by concepts hierarchy [ 33 ]. Also, for a smaller dataset, the decoder may su er from low variance vocabulary and data sparsity issues. In [ 7 ], a novel

Hierarchical Multi-label Classi cation Network (HMCN) with feed-forward and

recurrent variations is proposed that jointly optimizes local and global loss functions for discovering local hierarchical class-relationships in addition to global information from the entire class hierarchy while penalizing hierarchical violations (a child node getting a higher score than parent). However, they only consider tree based hierarchies where a node strictly has one parent.

Contextualized word embeddings, such as ELMo [ 29 ] and BERT [ 10 ], derived from pre-trained bidirectional language models (biLMs) and trained on large texts have shown to substantially improve performance on many NLP tasks; question answering, entailment and sentiment classi cation, constituency parsing, named entity recognition, and text classi cation. Such transfer learning involves ne-tuning of these pre-trained models on a down-stream supervised task to get good results with minimal e ort. In this sense, they are simple, e cient and performant. Motivated by this, and recent work of [ 22 ], we use BERT models for this task and achieve better results than CNN and RNN based methods. We also show great improvements with translated English texts. 3

Data

The dataset contains 8,385 training documents (including dev set) and 407 test documents, all in German. Each document has six text elds: document title, uses (goals) of the experiment, possible harms caused to animals and comments about replacement, reduction and re nement (in the scope of 3R principles).

ICD-10 Code

II C00-C97

IX

VI C00-C75 No. of documents (train + dev) 1515 1479 930 799 732

The documents are assigned one or more codes from ICD-10-GM (German

Modi cation version 2016) which exhibits a hierarchy forming a DAG [33], where

the highest-level nodes are called chapters and their direct child nodes are called groups. The depth of most chapters is one but in some cases it goes to secondlevel (e.g. M00-M25, T20-T32) and, in one case, up to third-level (C00-C97). Documents are assigned mixed codes such that a parent and child node can coexists and a child node can have multiple parents. Moreover, 91 documents are missing one or more of six text elds and only 6,472 have labels (5,820 in train set and 652 in dev set), while 52 of them have only chapter level codes. Table

1 shows top-5 most frequent codes. These classes account for more than 90% of the dataset leading to a high imbalance. Due to a shallow hierarchy, we consider the problem as multi-label classi cation instead of hierarchical classi cation.

4

Methods

Since the documents are domain speci c and in German, we argue that it might be di cult for open-domain and multilingual pre-trained models to do e ective transfer learning. Furthermore, [ 1 ] suggests that each language has its own linguistic and cultural characteristics that may contain di erent signals to e ectively classify a speci c class. Based on this, and the fact that translations are always available as domain-free parallel corpora, we use them in our system and show improvements across all models. Since English has readily more accessible biomedical literature available as free texts, we use English translations for our documents. To perform a thorough case study, we tested several models and pre-trained embeddings. Below we describe each of them.

Baseline For baseline we use a TF-IDF weighted bag-of-words based linear SVM model.

CNN Convolutional Neural Network (CNN) learns local features of input repre

sentation through varying number and sizes of lters performing convolution operation. They have been very successful in many text classi cation tasks [ 18, 41 ].

While many advanced CNN architectures exist, we use a shallow model of [20].

Attention Models Attention is a mechanism that was initially proposed in sequence-to-sequence based Neural Machine Translation (NMT) [3] to allow decoder to attend to encoder states while making predictions. More generally, attention generates a probability distribution over features, allowing models to

put more weight on relevant features. In our study, we used three attention based models.

HAN Hierarchical Attention Network (HAN) deals with the problem of long

documents classi cation by modeling attention at each hierarchical level of document i.e. words and sentences [ 39 ]. This allows the model to rst attend word encoder outputs, in a sentence, followed by attending the sentence encoder outputs to classify a document. Like [ 39 ], we also use bidirectional Gated Recurrent

Units (GRUs) as word and sentence encoder.

SLSTM Self-Attention Long-Short Term Memory (SLSTM) network is a simple single layer network based on bidirectional LSTMs encoder. An input sequence is rst passed through the encoder and encoded representations are self-attended to produce outputs.

CLSTM All ICD codes have a textual description, e.g. code A80-A89 is about viral infections of the central nervous system that can help a model while classifying. Fig. 1 shows a document containing words related to those found in the descriptions of their labeled codes. Such words may or may not be present but the intuition is to use this additional meta-data to enrich the encoder representation by attention. To the best of our knowledge, this is the rst time that the codes' descriptions are directly used to align with input text via attention.

The closest work is from [4], where author uses codes attention but they directly

consider code as a unit of representation creating an embedding lookup. We also create an embedding layer for codes but using their texts where a code representation is obtained via average of word embeddings of each token. We call this network as Codes Attentive LSTM (CSLSTM) and describe it more formally.

Let X = fx1; x2; :::; xng 2 Rn d be an n-length input document sequence, where xi is a d-dimensional embedding vector for input word wi belonging to documents vocabulary VD. Let T = ft1; t2; :::; tmg 2 Rm l be m-codes by llength titles representation matrix, where each ti = fti1 ; ti2 ; :::; til g 2 Rl d and tij is d-dimensional embedding vector for code i's title word j, belonging to titles vocabulary VT . The embedding matrices are di erent for documents and codes titles, this is because the title words can be missing in documents vocab.

Similarly, we used di erent LSTM encoders for document and code words (shared

encoder under performed on dev set; not reported). The network then transforms input as Xout = CLSTM(X; T ), with following operations:

Xenc = [x1enc ; x2enc ; :::; xnenc ] xienc = LSTMW (xi) Tenc = [t1enc ; t2enc ; :::; tmenc ]

l tienc = 1l X LSTMC (tij )

j=1 Xout = [Xenc; Tenc] 2 R(n+m) h Xout = Xout + AT Xout

n 1 X Xout = n Xoutj

j=1

A = softmax(XoutXoTut) 2 R(n+m) (n+m) where, Xenc is a sequence of word encoder LSTMW outputs and Tenc is a sequence of averaged title words encoding by code encoder LSTMC . We concatenate document words sequence with titles sequence and perform self-attention

A, followed by residual connection and average over resulting sequence to get

nal representation.

BERT Pre-training large models on unsupervised corpus with language modeling objective and then ne-tuning the same model for a downstream supervised task eliminates the need of heavily engineered task-speci c architectures [ 10, 29, 30 ]. Bidirectional Encoder Representations from Transformers (BERT) is a recently proposed such model, following ELMo and OpenAI GPT. BERT is a multi-layer bidirectional Transformer (feed-forward multi-headed self-attention) [ 37 ] encoder that is trained with two objectives, masked language modeling (predicting a missing word in a sentence from the context) and next sentence prediction (predicting whether two sentences are consecutive sentences). BERT has improved the state-of-the-art in many language understanding tasks and recent works show that it sequentially model NLP pipeline, POS tagging, parsing, NER, sematic roles and coreference [ 36 ]. Similar works [ 13, 40 ] have been performed to understand and interpret BERT's learning capacity. We therefore use BERT in our task and show that it achieves best results compared to other models and is nearly agnostic to domain speci c pre-training (BioBERT; [ 22 ]). 5 5.1

Experiments

Pre-processing

We consider each document as one text eld i.e. all six elds are joined together to form one input text. As mentioned in section 3, only 6,472 documents are labeled, out of which 654 are in dev set form total of 840. Since there is no gold standard for these documents we cannot evaluate them, so we ignored them during training. We also abstained from adding an extra "no" class (i.e. proxy for predicting nothing) for such documents because we assume that all NTSs should be indexed (e.g. like MEDLINE auto-indexing of new PubMed articles) and therefore inherently has one or more true ICD-10 codes assigned to them.

However, the o cial evaluation script penalizes model predictions for such doc

uments by considering them all false positives. We will cover this in detail in results section.

To translate German documents to English we used automatic translation from Google Translate API v22. For both, German and English, we use language speci c sentence and word tokenizer o ered by NLTK [ 23 ] and spaCy3, respectively. Tokens with document frequencies outside 5 and 60% of training corpus were removed and only top-10000 tokens were kept to limit the vocabulary. This applies to all models other than BERT, which uses WordPiece tokenizer [ 38 ] and builds its own vocabulary. Lastly, we remove all the classes with frequency less than 15. All the experiments were performed without any cross-validation on dev set to nd best parameters. 5.2

Pre-trained Embeddings

We use following pre-trained models for German:

FTde: fastText DE Common Crawl (300d)4 BERTde: BERT-Base, Multilingual Cased (768d)5 and following for English:

FTen: fastText EN Common Crawl (300d)

PubMeden: PubMed word2vec (400d)6 BERTen: BERT-Base, Cased (768d)7

BioBERTen: BioBERT (768d)8 2 https://cloud.google.com/translate/docs/translating-text 3 https://spacy.io/usage/models 4 https://fasttext.cc/docs/en/crawl-vectors.html 5 https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_

H-768_A-12.zip 6 https://archive.org/details/pubmed2018_w2v_400D.tar 7 https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_

A-12.zip 8 https://github.com/naver/biobert-pretrained/releases/tag/v1.

0-pubmed-pmc

TF-IDF + Linear SVM For baseline, we use scikit-learn implementation of

LinearSVC with one-vs-all training [28].

For all the models, except BERT, we used a batch size of 64, max sequence

length of 256, learning rate of 0.001 with Adam [ 21 ] and 50 epochs with early stopping. We used binary cross-entropy for each class as our objective function and F1-micro score as performance metrics. All the experiments were performed on single 12 GB Nvidia TitanXp GPU. We implemented these models and our code is publicly available.9

CNN We con gured CNN with 64 channels and lter sizes of 3, 4 and 5.

HAN Following [39], we also used biGRU encoders but with hidden size of 300.

We set the maximum number of sentences in a documents and maximum number of words in a sentence as 40 and 10 respectively.

SLSTM A biLSTM encoder with hidden size of 300.

CLSTM Similar to SLSTM, but with additional T matrix of size total number of titles (230, collected from ICD-10-GM) max title sequence length of 10.

BERT We used PyTorch's implementation of BERT10 with default parameters.

To avoid memory issues, we used maximum sequence length of 256 with batch size 6.

Ensemble Based on dev set results, we also created an ensemble of top-2 models as weighted combination of their raw scores, where then the prediction for each example is given by:

y^ = 1f ( S1 + (1 )

S2) > 0:5g 2 f0; 1gjCj

S1, S2 are raw probability scores from rst and second best model respectively,

while is sigmoid function and jCj is number of classes. We select best value of on dev set such that the f1 score of ensemble is higher than individual models. Fig. 2 shows variation with performance metrics. 5.4

Results

9 https://github.com/suamin/multilabel-classification-bert-icd10 10 https://github.com/huggingface/pytorch-pretrained-BERT (English) improved the score by an avg. of 4.07%. This can be attributed to the fact that there is much more English texts than other languages, but it can also be argued that English may have stronger linguistic signals to classify the classes where German models make mistakes [ 1 ].

The baseline proved to be a strong one, with the highest precision of all and outperforming HAN and CNN models, for both German and English, with common crawl embeddings. HAN performs better when documents are relatively long e.g. [ 4 ] reports strong results with HAN based models on MIMIC dataset [ 17 ], where the average document size exceeds 1900 tokens. After pre-processing, the averaged document length in our case was approximately 340. For CNN, we believe advanced variants may perform better.

SLSTM and CLSTM, both being just one layer, performed comparably and

better than baseline. SLSTM is much simpler and relies purely on self-attention, which also compliments higher scores by BERT models, which are stacked multiheaded self-attention networks. For CLSTM, since many documents are missing the title words (in fact many title words never appeared in corpus), the model had weak alignment signals between documents and this additional meta-data.

However, it still performed really well, getting second best score with PubMed

embeddings.

BERT performed better than other models, both in German and English with an avg. score of 6% points higher. BioBERTen performed slightly (+0.86%) better than BERTen, this was also noticeable in Relation Extraction task in [ 22 ], where domain speci c and general BERT performed comparably. This partly shows BERT's ability to generalize and being robust to domain shifts (learning from only 5k training docs), however, this contradicts the recent ndings of [40], where authors re ect on such issues, and catastrophic forgetting in BERTlike models. On other hand, the e ect of using in-domain pre-trained models was more signi cant for static-embeddings; using pre-trained PubMeden vectors Baseline

CNN HAN SLSTM CLSTM BERT

TF-IDFde TF-IDFen

FTde

FTen PubMeden

FTde

FTen PubMeden

FTde

FTen PubMeden out-performed open-domain FTen by an avg. of 2.77%. Such analysis was not performed for German due to lack of medical domain German vectors. BERT models had highest recall but relatively poor precision. This is preferable in real-world medical applications, where the recall is of much more importance.

We also combined our top-2 models, BioBERTen and CLSTSM-PubMeden,

to get an ensemble which performed better than both and got highest score of 84.67% on dev set. The intuition was to improve on BERT's precision without losing too much of recall. At = 0:63 we got the highest score. This increased

BioBERTen precision by 7.24% at loss of 2.5% recall. Since our focus was mainly

on single model systems therefore we used best single model for submission. 5.5

Submission and test scores

The test set contains 407 documents, which we rst translate to English and then run predictions with BioBERTen as our submitted model. We obtained a test f1micro of 73% with 86% recall and 64% precision as posted by o cial results. Our system ranked second but the there was signi cant di erence between test and dev set performances, especially, low precision. After the gold set was released, we probed it and realized that the o cial script provided by the challenge considers all predictions on test examples for which there is no gold label (93 of them) as false positives. We think that it is intrinsically impossible to compare examples Baseline

CNN HAN SLSTM CLSTM BERT

TF-IDFde TF-IDFen

FTde

FTen PubMeden

FTde

FTen PubMeden

FTde

FTen PubMeden with predictions where gold standard is not available. To emphasize, we give an example, if we take test document with id=20486 where the gold labels are fC00C97, C76-C80, IIg and our best model predicted fC00-C97, C76-C80, IIg i.e. a perfect match with maximum score. Given o cial evaluation, if this example did not had gold standard available then our model predictions would all had been considered as false positives, which severely degrades precision of a model which may have generalized well to predict on future examples. Table 3 shows this comparison on test set, where in "Original" column we use the same evaluation as provided by the task and in "Modi ed" we remove all documents from evaluation for which gold labels are not available. As can be seen, recall column is just the same as original with only precision column changes which changes f1-score as well. With the modi cation, all the models have similar performance as it was on dev set, as we also evaluated trained and evaluated on dev set by removing unlabeled examples. With modi cation, the submitted system achieves a test score of 80.82% now compared to that of 82.90% on dev set. Finally, ensemble model gets highest scores of 77.98% and 82.49% with original and modi ed evaluation respectively.

Discussion

Biomedical text mining is generally a challenging eld but recent progresses of transfer learning in NLP can signi cantly reduce the engineering required to come up with domain sensitive models. Unsupervised data is cheap, and can be obtained in abundance to learn general language patterns [40], however, such data may not be readily available when dealing with in-domain and low-resource languages (e.g. Estonian medical documents). Such de ciencies encourage research for better cross-lingual and cross-domain embedding alignment methods that can transfered e ectively.

Acknowledgements

We thank Stalin Varanasi for helpful discussions. This work was partially funded by European Union's Horizon 2020 research and innovation programme under grant agreement No. 777107.

2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480{1489, 2016. 40. Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al. Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373, 2019. 41. Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classi cation. In Advances in neural information processing systems, pages 649{657, 2015.

References

1. Reinald Kim Amplayo , Kyungjae Lee , Jinyeong Yeo , and Seung-won Hwang. Translations as additional contexts for sentence classi cation . arXiv preprint arXiv:1806.05516 , 2018 .

2. A Atutxa , A Casillas , N Ezeiza , I Goenaga , V Fresno , K Gojenola , R Martinez , M Oronoz , and O Perez-de Vinaspre . Ixamed at clef ehealth 2018 task 1: Icd10 coding with a sequence-to-sequence approach . CLEF , 2018 .

3. Dzmitry Bahdanau , Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate . arXiv preprint arXiv:1409.0473 , 2014 .

4. Tal Baumel , Jumana Nassour-Kassis, Raphael Cohen, Michael Elhadad , and Noemie Elhadad. Multi-label classi cation of patient notes: case study on icd code assignment . In Workshops at the Thirty-Second AAAI Conference on Arti cial Intelligence , 2018 .

5. Bettina Bert , Antje Dorendahl, Nora Leich, Julia Vietze, Matthias Steinfath, Justyna Chmielewska, Andreas Hensel, Barbara Grune, and Gilbert Schonfelder. Rethinking 3r strategies: Digging deeper into animaltestinfo promotes transparency in in vivo biomedical research . PLoS biology , 15 ( 12 ): e2003217 , 2017 .

6. Rabia Bounaama and M El Amine Abderrahim . Tlemcen university at celf ehealth 2018 team techno: Multilingual information extraction-icd10 coding . CLEF , 2018 .

7. Ricardo Cerri , Rodrigo C Barros, and Andre CPLF De Carvalho. Hierarchical multi-label classi cation using local neural networks . Journal of Computer and System Sciences , 80 ( 1 ): 39 { 56 , 2014 .

8. Kyunghyun Cho , Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio . Learning phrase representations using rnn encoder-decoder for statistical machine translation . arXiv preprint arXiv:1406.1078 , 2014 .

9. Koby Crammer , Mark Dredze, Kuzman Ganchev, Partha Pratim Talukdar, and Steven Carroll . Automatic code assignment to medical text . In Proceedings of the workshop on bionlp 2007 : Biological, translational, and clinical language processing , pages 129 { 136 . Association for Computational Linguistics, 2007 .

10. Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . Bert: Pretraining of deep bidirectional transformers for language understanding . arXiv preprint arXiv:1810.04805 , 2018 .

11. Remi Flicoteaux . Ecstra-aphp@ clef ehealth2018-task 1: Icd10 code extraction from death certi cates . CLEF , 2018 .

12. Julien Gobeill and Patrick Ruch . Instance-based learning for icd10 categorization . CLEF , 2018 .

13. Yoav Goldberg . Assessing bert's syntactic abilities . arXiv preprint arXiv:1901.05287 , 2016 .

14. Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory . Neural computation , 9 ( 8 ): 1735 { 1780 , 1997 .

15. Julia Ive , Natalia Viani, David Chandran, Andre Bittar , and Sumithra Velupillai . Kcl-health-nlp@ clef ehealth 2018 task 1: Icd-10 coding of french and italian death certi cates with character-level convolutional neural networks . In 19th Working Notes of CLEF Conference and Labs of the Evaluation Forum, CLEF 2018 , Avignon, France, 10 September 2018 through 14 September 2018 , volume 2125 . CEUR-WS , 2018 .

16. Serena Jeblee , Akshay Budhkar, Sasa Milic, Je Pinto, Chloe Pou-Prom, Krishnapriya Vishnubhotla, Graeme Hirst, and Frank Rudzicz . Toronto cl at clef 2018 ehealth task 1: Multi-lingual icd-10 coding using an ensemble of recurrent and convolutional neural networks .

17. Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei , Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database . Scienti c data , 3 : 160035 , 2016 .

18. Rie Johnson and Tong Zhang. Deep pyramid convolutional neural networks for text categorization . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 562 { 570 , 2017 .

19. Liadh Kelly , Hanna Suominen, Lorraine Goeuriot, Mariana Neves, Evangelos Kanoulas, Dan Li , Leif Azzopardi , Rene Spijker, Guido Zuccon, Harrisen Scells, and Joa~o Palotti. Overview of the CLEF eHealth evaluation lab 2019 . In Fabio Crestani, Martin Braschler, Jacques Savoy, Andreas Rauber , et al., editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019 ). Lecture Notes in Computer Science, Berlin Heidelberg, Germany, 2019 . Springer.

20. Yoon Kim . Convolutional neural networks for sentence classi cation . arXiv preprint arXiv:1408.5882 , 2014 .

21. Diederik P Kingma and Jimmy Ba . Adam: A method for stochastic optimization . arXiv preprint arXiv:1412.6980 , 2014 .

22. Jinhyuk Lee , Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang . Biobert: pre-trained biomedical language representation model for biomedical text mining . arXiv preprint arXiv:1901.08746 , 2019 .

23. Edward Loper and Steven Bird . Nltk: the natural language toolkit . arXiv preprint cs/0205028 , 2002 .

24. Zulfat Miftahutdinov and Elena Tutubalina . Kfu at clef ehealth 2017 task 1: Icd10 coding of english death certi cates with recurrent neural networks . In CLEF (Working Notes) , 2017 .

25. Tomas Mikolov , Kai Chen, Greg Corrado, and Je rey Dean. E cient estimation of word representations in vector space . arXiv preprint arXiv:1301.3781 , 2013 .

26. Mariana Neves , Daniel Butzke, Antje Dorendahl, Nora Leich, Benedikt Hummel, Gilbert Schonfelder, and Barbara Grune. Overview of the CLEF eHealth 2019 Multilingual Information Extraction . In Fabio Crestani, Martin Braschler, Jacques Savoy, Andreas Rauber , et al., editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019 ). Lecture Notes in Computer Science, Berlin Heidelberg, Germany, 2019 . Springer.

27. World Health Organization. International statistical classi cation of diseases and related health problems , volume 1 . World Health Organization, 2004 .

28. Fabian Pedregosa , Gael Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer , Ron Weiss, Vincent Dubourg , et al. Scikit-learn: Machine learning in python . Journal of machine learning research , 12 (Oct): 2825 { 2830 , 2011 .

29. Matthew E Peters , Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee , and Luke Zettlemoyer . Deep contextualized word representations . arXiv preprint arXiv:1802.05365 , 2018 .

30. Alec Radford , Karthik Narasimhan, Time Salimans, and Ilya Sutskever . Improving language understanding with unsupervised learning . Technical report, Technical report, OpenAI , 2018 .

31. Jurica Seva , Mario Sanger, and Ulf Leser. Wbi at clef ehealth 2018 task 1: Language-independent icd-10 coding using multi-lingual embeddings and recurrent neural networks . CLEF , 2018 .

32. Benjamin Shickel , Patrick James Tighe, Azra Bihorac, and Parisa Rashidi . Deep ehr: a survey of recent advances in deep learning techniques for electronic health record (ehr) analysis . IEEE journal of biomedical and health informatics , 22 ( 5 ): 1589 { 1604 , 2018 .

33. Carlos N Silla and Alex A Freitas . A survey of hierarchical classi cation across di erent application domains . Data Mining and Knowledge Discovery , 22 ( 1-2 ): 31 { 72 , 2011 .

34. Noah A Smith. Contextual word representations: A contextual introduction . arXiv preprint arXiv:1902.06006 , 2019 .

35. Ilya Sutskever , Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks . In Advances in neural information processing systems , pages 3104 { 3112 , 2014 .

36. Ian Tenney , Dipanjan Das , and Ellie Pavlick . Bert rediscovers the classical nlp pipeline . arXiv preprint arXiv:1905.05950 , 2018 .

37. Ashish Vaswani , Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser , and Illia Polosukhin . Attention is all you need . In Advances in neural information processing systems , pages 5998 { 6008 , 2017 .

38. Yonghui Wu , Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao , Klaus Macherey , et al. Google's neural machine translation system: Bridging the gap between human and machine translation . arXiv preprint arXiv:1609.08144 , 2016 .

39. Zichao Yang , Diyi Yang , Chris Dyer, Xiaodong He, Alex Smola , and Eduard Hovy . Hierarchical attention networks for document classi cation . In Proceedings of the