Nosology is an important branch of Medical Science that concerns the classi cation and coding of diseases, conditions, procedures, and other medical information. This is a vital task for all stakeholders of the health sector, from hospitals and health regulators, to insurance companies and governments. The ICD10 system is the current revision of a Nosology system managed by the World Health Organization, being widely used internationally. Since medical coding is based on manual analysis of clinical textual data, it is ripe for automation, with Natural Language Processing (NLP) techniques used to address this challenge. This paper describes our contribution to the CLEF eHealth 2020 Task 1 Challenge, regarding Information Extraction of ICD10 codes on unstructured Spanish clinical text. We present two approaches for ICD10 code extraction based on Conditional Random Fields (CRFs) and the BERT Deep Learning Language Model. The BERT -based methodology achieved a mean average precision of 0:517 and 0:445 for ICD10-CM and ICD10-PCS codes, respectively, and a F1 score of 0:505 for the Explainable AI subtask. The results obtained show the exibility and robustness of pre-trained Deep Learning models for NLP, only requiring ne-tuning for a particular task, leading to reduced requirements both for labelled data and computational e ort.
BERT
Medical coding, also known as Nosology, is an important area for the health sector, with dedicated specialists manually annotating a large number of relevant clinical documents, such as in- and outpatient clinical reports. This operation is essential for several stakeholders, including health information management systems, insurance companies, governments, researchers, among others [5].
The most commonly used medical coding system is the International Classi cation of Diseases (ICD), developed and maintained by the World Health Organization (WHO), which aims to provide the nations of the world with a Nosology standard for disorders, diseases, and other conditions, structured in a hierarchical fashion. Currently it is in its tenth revision (ICD10) [35], with a new major release (ICD11) planed for early 2022 [5].
The ICD10 code system has been further augmented by WHO member states,
including the USA, where the Centers for Medicare and Medicaid Services have
the ICD10-CM (Clinical Modi cation) [
The annotation process is based on the manual analysis of clinical reports, leading to signi cant costs and time spent by specialists. Therefore, the introduction of (semi-)automatic processes of annotation is an important challenge. This is where Natural Language Processing (NLP) comes into play. Recently, the paradigm of NLP has shifted to the application of large, deep models, pretrained on extensive corpora, and ne-tuned on a particular task at hand [37]. This allows highly accurate models for a plethora of applications with low netuning e ort, when compared to tailor-made systems that require high amounts of annotated data, with high computational costs and training time.
In this paper we describe our contributions to CLEF eHealth 2020 Task 1, which have their basis on the application of Conditional Random Fields (CRF), and the BERT language model [12], pre-trained on Spanish corpora [10], and ne-tuned on NER of ICD10 codes. 2
The CLEF eHealth 2020 challenges researchers with real-world datasets,
fostering the application of state of the art NLP methodologies on the medical and
clinical domains [14]. In particular, the proposed Task 1 in this series concerns
Information Extraction (IE), focusing on the extraction of ICD10 codes from
clinical textual data in Spanish [
The dataset is composed of Spanish text data from clinical reports, partitioned into train (500 reports), dev (250), and test (250) subsets. In addition, 2751 reports are provided with the test set as a background set, to discourage manual corrections and promote scalable solutions. In total, 2172 ICD10-CM and 696 ICD10-PCS unique codes are referenced in the train and dev set.
Task 1 is subdivided into 3 subtasks: 1 | ICD10-CM codes assignment predict ICD10-CM codes present in a given clinical report, ranked by con dence; 2 | ICD10-PCS codes assignment predict ICD10-PCS codes present in a given clinical report, ranked by con dence; 3 | Explainable AI predict ICD10-CM and ICD10-PCS codes present in a given report and provide a textual reference (character span) that justi es said codes.
Subtasks 1 and 2 can be regarded as a multi-label classi cation problem on a report level, whereas subtask 3 is related to multiclass classi cation on a wordby-word level, i.e., a Named Entity Recognition (NER) problem, where each word present in a given text has 1 label associated with it (ICD10 codes in this case). 3
Medical coding has been a task mainly reserved to specialized personnel, although there are some recent e orts to automate this process.
Early attempts for automatic medical coding were mainly rule-based systems for ICD9 [34] code assignment [11,13]. These systems automatically nd relations between ICD9 codes, their descriptions, and medical text, creating a list of associations that allows medical text labelling. Another approach is described in [16], where the authors created an ICD10 coding system by applying Support Vector Machine classi ers in a cascaded architecture to automatically assign cancer related medical codes to death certi cates. Although all these systems achieve good results, their reach is reduced, with each only encompassing a small subset of the full ICD code list. Nevertheless, these systems have high interpretability, which makes them valuable and interesting to use in speci c, smaller scopes.
New methodologies developed with Deep Learning (DL) models and
architectures have had a great impact in recent NLP research. Most of the tasks
and challenges of this area have bene ted from deep word embedding
strategies, from global and context-free embeddings such as word2vec [
More and more of these DL methods are being applied in the eld of clinical
NLP, including in the extraction of ICD codes from text data [36]. For this task,
the annual CLEF eHealth IE challenges have had signi cant contributions with
the application of state of the art models on multilingual clinical text corpora
[
The contributions of Amin et al. [6] and Sanger et al. [
Amin et al. [6] applied an English version of BioBERT (BERT model trained on biomedical text data) [17] on machine-translated versions of the German summaries for multi-label classi cation. The best results were achieved by performing an ensemble of the predictions using the BioBERT model and a Code Attentive LSTM network [6] with pre-trained PubMed word2vec embeddings,1 reaching an F 1 score of 0:78 on the test set of the challenge.
Sanger et al. [
Both of these contributions show the versatility of deep language models for ICD10 code extraction, and in particular of the BERT architecture, achieving the top results for this challenge. 4
Two runs were submitted for subtasks 1 and 2, comprising two distinct methodologies to tackle this challenge: Conditional Random Fields (CRF) and the BERT Deep Learning Model. For subtask 3, we submitted a single run using the BERT based model.
Both methodologies tackle all subtasks at once, by considering the challenge as a NER problem, similar to what is described in Subtask 3. In this case, each token present in a given clinical report is classi ed using the available ICD10 codes (including an O tag for tokens that do not have a code associated with them). The predictions for Subtask 1 and 2 are derived from this NER schema by identifying all predicted ICD10-CM and ICD10-PCS codes (respectively) in a given clinical report, and associating them to the clinical report identi er, in a descending order of con dence. Thus, the only changing factor across methodologies is how token classi cation is performed: using CRF, or using BERT.
Each of the tested methodologies was implemented in Python and is described below in more detail.
4.1
CRFs are a commonly used technique for NER, since they take into account
context around neighbouring words to create a statistical model that can infer
their type [
For text pre-processing, namely tokenization, lemmatisation, Part-of-Speech
tagging, among other morphological token characteristics extraction, the
StanfordNLP toolbox stanza2 [
The methodologies applied for ICD10 code extraction are similar to those
described by Tawara et al. [
This approach is used to produce two distinct models: CRF CM CRF model trained exclusively on ICD10-CM codes; CRF PCS CRF model trained exclusively on ICD10-PCS codes.
Each of these models used to create predictions for subtasks 1 and 2, respectively, identifying all codes present in a given report and ranking them by con dence. 4.2
BERT A schema of the steps followed for the BERT methodology is shown in Figure 1.
The BERT model was employed as a means of performing NER on the clinical reports of the dataset. Since the reports are written in Spanish, it is adequate to use models pre-trained on Spanish corpora; thus we use the BETObase;uncased pre-trained model [10], available3 on the transformers4 Python package [33]. This model was pre-trained on text that was rst pre-processed to lowercase. BETObase;cased,5 where words can remain capitalised, was also considered, but since they provided similar results, the uncased version was chosen for simplicity.
Since BETObase;uncased (henceforth BETO ) is a pre-trained model, most of the computational e ort has already been performed, with the model only requiring ne-tuning for the NER task. This is performed by addition of a linear classi cation layer for the token embedding outputs of the model, and training all model parameters with this new task [12].
However, since the type of text present in clinical reports can vastly di er from commonly used corpora for model pre-training, a rst Language Modelling
(LM) step was considered, where the BETO model is rst tuned on the train set of the challenge's dataset, by performing Masked Language Modelling and Next Sentence Prediction (both unsupervised tasks) [12]. This is expected to further improve results by modelling the particular architecture, choice of words, and medical jargon used in clinical reports.
Furthermore, since ICD10-CM and ICD10-PCS codes di er in application, it is reasonable to separate the classi cation of each into two distinct models. Therefore, each pre-trained BETO model is ne-tuned on two distinct NER tasks for each type of ICD code, generating two classi ers: BETO CM and BETO PCS. Each of these is used to predict exclusively ICD10-CM or ICD10PCS codes, respectively, with their predictions combined to identify all relevant codes in a given text.
Consequently, four di erent BERT -based models are developed and tested for NER: BETO CM Pre-trained on Spanish corpora, ne-tuned on ICD10-CM NER; LM-BETO CM Pre-trained on Spanish corpora, further LM tuning on the challenge's dataset, ne-tuned on ICD10-CM NER; BETO PCS Pre-trained on Spanish corpora, ne-tuned on ICD10-PCS NER; LM-BETO PCS Pre-trained on Spanish corpora, further LM tuning on the challenge's dataset, ne-tuned on ICD10-PCS NER.
Model Training The dataset was rst pre-processed and converted to a NER dataset, with segmented sentences. Tokenization is performed by the BERT WordPiece tokenizer [12].
The BETO model was trained for LM for 50 epochs, using a linearly decreasing learning rate, starting at 5 10 5. Training was done with a batch size of 16 and a block size of 256 (maximum number of tokens per input sequence). This block size was found to be su cient, since most sentences have a much smaller number of tokens present. According to the results of this training step, a suitable number of epochs was chosen as a basis for the LM-BETO models, to avoid over tting.
All models are trained for NER for 15 epochs, with a batch size of 8, and block size of 256. The learning rate is determined by a cosine scheduler with warmup (2 epochs) and hard restarts (2 cycles), with a maximum of 5 10 5. 5
Two main sets of results are here reported, based on the evaluation on the dev (Subsection 5.1) and the test (Subsection 5.2) set.
Analysing the results on the dev set, two runs were submitted for evaluation: one using the CRF methodology, and another based on the LM-BETO CM and LM-BETO PCS models. The test set results were provided by the task evaluators6 after run submission. Language Modelling Results The evolution of LM training of BETO on the train set can be seen on Figure 2, with both training loss and dev set perplexity shown for each training epoch. Perplexity is a commonly used metric for LM, which measures how good a language model is at predicting an unknown sample, and is described in Equation 1, with H(m) being the cross-entropy loss of a given model m. Lower perplexity values indicate a better predicting language model.
P P (m) = 2H(m) (1)
As seen of Figure 2, the perplexity decreases with more training time, although it somewhat stabilizes after approximately 10 epochs. To avoid over tted LM-BETO models, their 10 epoch weights are used henceforth. This model is then used as a basis for ne-tuning LM-BETO CM and LM-BETO PCS on NER of ICD codes.
NER Results The results obtained for NER on the dev set for all considered trained models are shown in Table 1. The considered metrics are the microaveraged precision (P ), recall (R), and F1-score (F 1).
Note that this evaluation was performed considering the NER task, i.e., all presented metrics are on a token by token basis. Furthermore, although models were only trained on the codes present in the train set, metrics are shown considering all unique train and dev ICD10 codes. 6 https://github.com/TeMU-BSC/CodiEsp-Evaluation-Script 6:5
6 y t iex 5:5 l p r e P 5 4:5 4 2 1 1:5 0:5 s s o L g n i n i a r T 0 10 20
30 Epochs 40 50
P
R
F1 0.693 0.304 0.513 0.498 0.477 0.487 0.498 0.480 0.489 The submitted runs were evaluated by the task organizers, with reported results shown in Tables 2, 3, and 4, for subtasks 1, 2, and 3, respectively. Bold metrics indicate the o cial metric used for model evaluation. Bold values indicate the model that achieved the best value for a particular metric.
For the subtasks, three distinct evaluation modes were considered: All codes Considers all unique ICD10 codes present in the train, dev, and test sets; Train + Dev codes Considers only codes present in the train and dev sets, codes that are unique to the test set are ignored; Category Only consider as labels the rst 3 digits of ICD10-CM codes, and the rst 4 digits of ICD10-PCS codes.
For Tables 2 and 3, the metrics used are the Mean Average Precision (MAP)
and the Mean Average Precision at K (MAP@K) (K = 30 for ICD10-CM codes,
and K = 10 for ICD10-PCS codes) [
It is important to note that this BERT -based methodology was applied for Spanish clinical texts, but could have easily been applied to di erent languages, simply by using a model that is pre-trained on that speci c language, or using a multilingual model, and performing ne-tuning as described here.
The achieved results show the exibility of novel DL based NLP models for the execution of a number of tasks on several di erent elds of application, taking advantage of pre-trained models to ne-tune classi ers with little computational e ort and small amount of data, bringing Transfer Learning to NLP. 7.1
The aforementioned methodologies can be signi cantly improved in the future to create an even more robust system that can tackle a larger number of ICD10 codes.
A clear gap in the DL methodology is the lack of input provided by the ICD10 code descriptions, which often contain very relevant information regarding certain aspects of a given condition, disease or procedure. Moreover, there exists vast data online regarding these same codes and the underlying concepts they represent, which can be taken advantage to build a more robust system, employing techniques similar to those by Bai et al. [7].
BERT models can also be improved for NER with several di erent
techniques. For instance, Souza et al. [
Finally, the scope of the solution here presented was limited to the ICD10 codes present in the training set, which are a very small percentage of the total number of codes in this Nosology system (98288 ICD10-CM and 87170 ICD10PCS codes). A truly robust ICD IE system would have the possibility to predict any code, as well as have an inherent representation of their hierarchical structure, being able to predict the most accurate code for a given sequence when possible, or a more general but suitable code when not. 5. WHO | International Classi cation of Diseases (ICD) Information Sheet, http: //www.who.int/classi cations/icd/factsheet/en/, accessed on 2020-07-03 6. Amin, S., Neumann, G., Dun eld, K., Vechkaeva, A., Chapman, K.A., Wixted, M.K.: MLT-DFKI at CLEF eHealth 2019: Multi-label Classi cation of ICD-10 Codes with BERT. Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum p. 15 (2019) 7. Bai, T., Vucetic, S.: Improving Medical Code Prediction from Clinical Text via Incorporating Online Knowledge Sources. In: The World Wide Web Conference. pp. 72{82. WWW '19, Association for Computing Machinery (2019), DOI: 10.1145/ 3308558.3313485, https://doi.org/10.1145/3308558.3313485 8. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching Word Vectors with Subword Information. Transactions of the Association of Computational Linguistics 5, 135{146 (2017), DOI: 10.1162/tacl a 00051, https://www.mitpressjournals. org/doi/abs/10.1162/tacl a 00051 9. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language Models are Few-Shot Learners (2020), http://arxiv.org/abs/2005.14165 10. Can~ete, J., Chaperon, G., Fuentes, R., Perez, J.: Spanish pre-trained BERT model and evaluation data. In: To Appear in PML4DC at ICLR 2020 (2020) 11. Crammer, K., Dredze, M., Ganchev, K., Pratim Talukdar, P., Carroll, S.: Automatic Code Assignment to Medical Text. In: Biological, Translational, and Clinical Language Processing. pp. 129{136. Association for Computational Linguistics (2007), https://www.aclweb.org/anthology/W07-1017 12. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019), http://arxiv.org/ abs/1810.04805 13. Farkas, R., Szarvas, G.: Automatic construction of rule-based ICD-9-CM coding systems. BMC Bioinformatics 9(3), S10 (2008), DOI: 10.1186/1471-2105-9-S3S10, https://doi.org/10.1186/1471-2105-9-S3-S10 14. Goeuriot, L., Suominen, H., Kelly, L., Miranda-Escalada, A., Krallinger, M., Liu, Z., Pasi, G., Saez Gonzales, G., Viviani, M., Xu, C.: Overview of the CLEF eHealth evaluation lab 2020. In: Arampatzis, A., Kanoulas, E., Tsikrika, T., Vrochidis, S., Joho, H., Lioma, C., Eickho , C., Neveol, A., Cappellato, L., Ferro, N. (eds.) Experimental IR Meets Multilinguality, Multimodality, and Interaction: Proceedings of the Eleventh International Conference of the CLEF Association (CLEF 2020).
LNCS Volume Number: 12260 (2020) 15. Greene, E.: Extracting Structured Data From Recipes Using Conditional Random Fields, https://open.blogs.nytimes.com/2015/04/09/ extracting-structured-data-from-recipes-using-conditional-random- elds/, accessed on 2020-07-15 16. Koopman, B., Zuccon, G., Nguyen, A., Bergheim, A., Grayson, N.: Automatic ICD10 classi cation of cancers from free-text death certi cates. International Journal of Medical Informatics 84(11), 956{965 (2015), DOI: 10.1016/j.ijmedinf.2015.08.004, http://www.sciencedirect.com/science/article/pii/S1386505615300289 17. Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: BioBERT: A pre-trained biomedical language representation model for biomedical text mining p. btz682 (2019), DOI: 10.1093/bioinformatics/btz682, http://arxiv.org/abs/1901. 08746 33. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Brew, J.: HuggingFace's transformers: Stateof-the-art natural language processing abs/1910.03771 (2019) 34. World Health Organization: International Classi cation of Diseases : [9th] Ninth Revision, Basic Tabulation List with Alphabetic Index. World Health Organization (1978), https://apps.who.int/iris/handle/10665/39473, accepted: 2012-0616T14:05:20Z Journal Abbreviation: ICD-9 : basic tabulation list with alphabetic index 35. World Health Organization: International Statistical Classi cation of Diseases and
Related Health Problems. World Health Organization (2016), oCLC: 910334285 36. Wu, S., Roberts, K., Datta, S., Du, J., Ji, Z., Si, Y., Soni, S., Wang, Q., Wei, Q., Xiang, Y., Zhao, B., Xu, H.: Deep learning in clinical natural language processing: A methodical review. Journal of the American Medical Informatics Association 27(3), 457{470 (2020), DOI: 10.1093/jamia/ocz200, https://academic.oup.com/ jamia/article/27/3/457/5651084 37. Young, T., Hazarika, D., Poria, S., Cambria, E.: Recent Trends in Deep Learning Based Natural Language Processing (2018), http://arxiv.org/abs/1708.02709