English. The practice of assigning a uniquely identifiable and easily traceable code to pathology from medical diagnoses is an added value to the current modality of archiving health data collected to build the clinical history of each of us. Unfortunately, the enormous amount of possible pathologies and medical conditions has led to the realization of extremely wide international codifications that are difficult to consult even for a human being. This difficulty makes the practice of annotation of diagnoses with ICD-10 codes very cumbersome and rarely performed. In order to support this operation, a classification model was proposed, able to analyze medical diagnoses written in natural language and automatically assign one or more international reference codes. The model has been evaluated on a dataset released in the Spanish language for the eHealth challenge (CodiEsp) of the international conference CLEF 2020, but it could be extended to any language with latin characters. We proposed a model based on a two-step classification process based on BERT and BiLSTM. Although still far from an accuracy sufficient to do without a licensed physician opinion, the results obtained show the feasibility of the task and are a starting point for future studies in this direction.
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Italian. La pratica di assegnare un codice univocamente identificabile e facilmente riconducibile ad una patologia a partire da diagnosi mediche e un valore aggiunto alla attuale modalita` di archiviazione dei dati sanitari raccolti per costruire la storia clinica di ciascuno di noi. Purtroppo pero`, lenorme numero di possibili patologie e condizioni mediche ha portato alla realizzazione di codifiche internazionali estremamente ampie e di difficile consultazione anche per un essere umano. Tale difficolt rende la pratica di annotazione delle diagnosi con i codici ICD-10 molto complessa e raramente svolta. Col fine di supportare tale operazione si e` proposto un modello di classificazione, in grado di analizzare le diagnosi mediche scritte in linguaggio naturale ed assegnarle automaticamente uno o piu` codici internazionali di riferimento. Il modello e` stato valutato su un dataset rilasciato in lingua Spagnola per la challenge (CodiEsp) di eHealth della conferenza internazionale CLEF 2020 ma e` di semplice estensione su qualsiasi lingua con caratteri latini. Abbiamo proposto un modello basato su due passi di classificazione e basati sullutilizzo di BERT e delle BiLSTM. I risultati ottenuti, seppur ancora lontani da una accuratezza sufficiente per far a meno di un parere di un medico esperto, mostrano la fattibilita` del task e si pongono come punto di partenza per futuri studi in tale direzione. In many of the existing systems for storing patient clinical data, the medical report of the doctor is stored in the form of textual data. Only in a few cases, one or more identification codes are assigned to each of the diagnosed conditions. The process of assigning a unique code to pathologies, symptoms, clinical situations, and drugs is commonly referred to as Clinical Coding. Among the most widely used coding systems, we can find the tenth version of the international medical glossary published by WHO (World Health Organization), commonly known as ICD-10. It contains codes for diseases, signs, and symptoms, abnormal results, complaints, social situations, and external causes of injury or illness. The United States uses its national variant of the ICD-10 called the ICD10 Clinical Modification (ICD-10-CM). A procedure classification called the ICD-10 Procedure Coding System (ICD-10-PCS) has also been developed for the acquisition of hospitalization procedures. There are over 70,000 ICD-10-PCS procedure codes and over 69,000 ICD-10-CM diagnosis codes, compared to about 3,800 procedure codes and about 14,000 diagnosis codes in the previous ICD-9-CM. The use of an international classification to annotate medical diagnoses makes the health system interoperable between different countries. Among the many possibilities of using ICD codes, a doctor of any nationality could thus be able to read, analyze, and use the medical history of a patient even if of a different nationality. In addition, diagnostic patterns used by clinicians could be identified to improve automatic disease prediction strategies and provide automatic specialist support for decision making. These observations strongly support the need for automated systems to support clinicians to perform this task quickly and without human intervention. The contribution of our work is a novel model for ICD-10 coses annotation We proposed a model based on Bi-LSTM and BERT to assign one or more ICD10 codes to the medical diagnosis. Specifically, we have designed our approach as a two-step process. In the first one, we use a BERT-based classifier to select from the dataset only a subset of sentences that could be candidates for the annotation step. In particular, those phrases are them that could e annotated with one or more codes. The phrases left were generally speaking expressions. In the second step, we used a Bi-LSTM model to analyze the candidate sentences and assign one or more codes to them. The results are encouraging and a good starting point for further investigations. The rest of the paper is structured as follows. We start analyzing related works, and we go through the description of the model and the dates. Finally, we analyze the results obtained, and we expose our consideration of the task in the conclusion section. 2
The scientific community has long addressed the
task of analyzing medical diagnoses in order to
assign a unique code for each pathology. In
particular, since 1973, the first corpus and state of the
art analysis of the clinical coding task have been
released. In 1999, Chapman stated that a system
based on algorithms would be better than a
human being to perform this task. If we think about
the high number of codes in the ICD-10 glossary
(about 70000), it immediately comes to mind that
even a human being cannot be very accurate in
the assignment, as long as he must know perfectly
each of the countless codes. In 2006 Kukafka
et al. affirm that the algorithmic path to be
followed for the resolution of the task was through
the NLP. It is, in fact the most suitable for the
resolution of the task, currently. To get automatic
systems able to afford the task efficiently, it is
necessary to wait until 2017. Miftahutdinov
The CodiEsp evaluation track proposed by CLEF
2020
The CodiEsp corpora contains manually annotated clinical reports, written in Spanish, with corresponding clinical codes. The training set contains 500 clinical cases, while the development and the test set provide 250 clinical cases each. The CodiEsp corpus format is plain text with UTF8 encoding, where each clinical case is stored in a single file whose name is the clinical unique case identifier. The final collection of the 1https://github.com/dccuchile/spanish-word-embeddings 2http://crscardellino.github.io/SBWCE/ 3https://github.com/marcopoli/CODIESP-10 LSTM BiLSTM BiLSTM + CNN BiLSTM + CNN + SelfAtt.
CNN CNN + SelfAtt.
BERT (BETO) 0.10629 1,000 clinical cases of the corpus contains 16,504 sentences, with 16.5 sentences per clinical case on average. It contains 396,988 words, with 396.2 words per clinical report on average. The final architecture of the model, previously proposed was obtained after conducting several experiments on 20% of the training dataset released for the Codiesp-D subtask. For each task we used as classification labels only them with at least one example set in the training set. In particular we used 1788 codes for the Codiesp-D and 546 codes for the Codiesp-P task. For lacking of space, we are going to report int his contribution only the most relevant. As first experiment we trained different classification models with the purpose of directly classify the single medical reports with one ICD-10 code. In particular we developed the following models: LSTM, BiLSTM, CNN, CNN + Self Attention, BERT, BiLSTM + CNN, BiLSTM + CNN + Self Attention,(Pre-filtering) BERT - (Classification) BiLSTM + CNN + Self Attention.
Analyzing the results in Tab. 1, it is possible to notice that considering the F1 score as a metric, models based on deep learning approaches with LSTM and CNN strategy are able to obtain very similar results. It is evident that the differences between these methodologies are minimal and that generally, a combination of them improves the performance. Starting from an F1 measure of 0.09789 obtained using a biLSTM layer, a score of 0.10410 is reached when combining the biLSTM, CNN, and self-attention layers. The BERT-based classification model requires particular attention. It succeeds, in fact, to obtain a score of F1 comparable to that obtained by the model that combines the single techniques (BiLSTM+CNN+Self.Att). Thus, we considered good candidates for the final classification model, both BERT and the BiLSTM+CNN+Self.Att. models.The second step of experimentation was to understand if using a classifier that performs the task of pre-filtering diagnoses not containing an ICD-10 code could help the classification performance. Tab. 2. shows how we decided to test the combinations of the two models previously chosen as candidates with a pre-filtering approach followed by a classification approach. Observing the results in terms of F1 measure, it is possible to observe that the model using BERT in the first phase of selection and BiLST + CNN + Self.Att. in the phase of choice of the final codes, is the one that leads to better results. We were thus able to increase the F1 score by about 0.03 points compared to the previous step, reaching the value of 0.13632. Finally, we decided to use a threshold on the result of the last layer of the here proposed model (dense layer with a softmax function) in order to extract more than one label for each medical report. We performed different experiment in order to decide this threshold and finally we used a value of 0.10 able, from our evaluation, to maximize the F1 score of the model, reaching a score of 0.16011. The model we implemented, has been used for participating at both CodiEsp subtasks, i.e., CodiEsp-D and CodiEsp-P. 5
The ability to automatically annotate medical reports with international codes is an open and relevant research challenge for future technologies of global medical data sharing. In this work, we have proposed a model based on the state of the art technologies to support the doctors during this timeconsuming task. For example we can imagine the use of our model in a system that can suggest to the doctor a set of possible codes to choose from to note down the medical diagnosis. Such a suggestion could make this annotation process less complex for any human annotator. It is important to keep in mind that the ICD-10 code contains more than 70,000 codes and their number is constantly increasing. During the competition it was shown that the large number of possible labels was also the most difficult problem to deal with even for automatic systems. In this regard, in fact, many systems based on machine learning have found many more difficulties than hybrid systems able to select through linguistic rules a subset of possible assignable codes. This observation will be used by us in the future to improve the model proposed here. Currently, the results obtained are encouraging and confirm the possibility of tackling it with current technologies. Nevertheless, our system cannot obtain a score of reliability, such as acting independently, and the opinion of a human coder is still essential. The system proposed here has not won the challenge at CLEF 2020 but is still a good starting point for further studies that want to use those technologies to resolve the clinical coding task.
This work is funded by project ”DECiSION” codice raggruppamento: BQS5153, under the Apulian INNONETWORK programme, Italy. Aitziber Atutxa, Arantza Casillas, Nerea Ezeiza, V´ıctor Fresno, Iakes Goenaga, Koldo Gojenola, Raquel Mart´ınez, Maite Oronoz Anchordoqui, and Olatz Perez-de Vin˜aspre. 2018. Ixamed at clef ehealth 2018 task 1: Icd10 coding with a sequence-tosequence approach. In CLEF (Working Notes), page 1.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. Jos Caete, Gabriel Chaperon, Rodrigo Fuentes, and Jorge Prez. 2020. Spanish pre-trained bert model and evaluation data. In to appear in PML4DC at ICLR 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1).