International Classi cation of Diseases (ICD) codes is used to represent the diseases and health statuses which is useful for better communication and to facilitate research. Automatic assignment of these codes to the clinical records is essential nowadays whereas multilingual extraction task of eHealth@CLEF-2020 concentrates on predicting ICD10-CM and ICD10-PCS codes in Spanish language. Automatic prediction of these codes have been approached by using transformers such as BERT, RoBERTa, Electra and XLNet. Among the models, Electra predicts ICD10-CM codes better and RoBERTa predicts ICD10-PCS codes better than the other transformer models.
Electra RoBERTa Transformers XLNet
The International Classi cation of Diseases (ICD) is a healthcare classi cation system maintained by the World Health Organization. Diseases and health statuses are classi ed according to certain rules and uniquely identi ed by character codes. The e ciency of ICD coding has begun to receive more attention because of how crucial it is for making clinical and nancial decisions. Hospitals with better coding quality see many bene ts, including more accurate classi cation and retrieval of medical records, and better communication with other hospitals to jointly promote healthcare quality and facilitate research.
Human classi cation of diagnoses is a labor intensive process that consumes signi cant resources. Most medical practitioners use specially trained medical coders to categorize diagnoses for billing and research purposes. The coding process requires a comprehensive consideration of each patient's health condition. However, very few medical practitioners are capable of taking over the process since they lack training in professional coding.
By the Multilingual Extraction Task [7] from eHealth Lab [
The CodiEsp corpus of 1,000 clinical case studies, selected manually by a practicing physician was used for training the models. The CodiEsp corpus is distributed in plain text in UTF8 encoding, where each clinical case is stored as a single le whose name is the clinical case identi er. Each clinical case or its identi er is associated with one or multiple medical codes depending on the sub task's golden truth le.
The corpus consists of two variants of the clinical les. The rst one being the original set lled with Spanish text and the other one being the English machine-translated version of the former. For our system we had used only the English version of the clinical case les to train the models.
The 1000 clinical case les were distributed as to constitute 500 train les, 250 development les and 250 test les. Additionally 2000 clinical les were released along with test set release to promote that these systems would potentially be able to scale to larger data collections. 4
The task was approached as a multi-label text classi cation task with the use of transformers. The data is preprocessed and then data is made suitable for training by creating a data frame for all the instances of the data. Model training is done by ne tuning the parameters of four transformers that include BERT, RoBERTa, XLNet and Electra. After training, a le with the probability scores of all possible codes is generated which is post-processed to lter the suitable codes for the instances. 4.1
Initially, the data is preprocessed and then data is made suitable for training. Preprocessing steps include removal of punctuation and stop words followed by tokenization and lemmatization. Then, the text les are prepared by mapping the textual data codes to a data frame that has ones and zeros. For example : If there are 5 unique codes in a scenario whereas text data has code1, code3 as its code, Then the text le is trained with a data frame that has rows with ones only at the columns with that codes with remaining zeros.
Data frame for that text - 1 0 1 0 0 Here, there are 2194 unique labels in which each text le will have labels in the range 1 to 56 for each text le.Thus, the data frames are created for each text le and then given to the model for training.
text labels 0 describe case 37yearold man previous active.| [0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, ...] 1 present head neck auscultation cardiac .... [0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, ...] 4.2
The transformer models namely BERT, RoBERTa, Electra and XLNet were used whose implementation is explained below.
BERT [
In our implementation, the Bio-Clinical-BERT model was made use with ***.
RoBERTa [6] makes use of a robustly optimized method that improves on BERT by modifying key hyperparameters in BERT.
For our implementation, Roberta was ne tuned over \BioMed-RoBERTa-base" [4], a language model based on the RoBERTa-base architecture is trained over 100 epochs with train batch size of 20 and learning rate of 4e-5.
In general, electra [
For our work, electra was ned tuned over \Electra-base" model of 12 Transformer blocks, 768 hidden dimensions and 110 M parameters, trained over 100 epochs with train batch size of 20 and learning rate of 4e-5 to predict the probability of codes.
XLNet[8] is a generalized AR pretraining method that uses a permutation language modeling objective to combine the advantages of AR (autoregression) and AE (autoencoding) methods that integrate Transformer-XL and the careful design of the two-stream attention mechanism.
For predicting the codes, XLNet was ne tuned over \xlnet-base-cased" model of parameters, 12 Transformer blocks, 12 self-attention heads and 768 hidden dimensions which is trained over 100 epochs with train batch size of 20 and learning rate of 4e-5. 4.3
After training, a prediction le with a probability score of all 2194 codes is generated which has to be post processed. Post processing includes ltering the suitable codes. Here, we ltered with a threshold value of '0.002'. which seems to be e ective. The probability of the codes greater than 0.002 are ltered and considered as codes. 5
File Model MAP Precision Recall F1 Run-1 BERT 0.001 0.009 0.014 0.011 Run-2 Electra 0.007 0.025 0.049 0.033 Run-3 RoBERTa 0.004 0.014 0.019 0.016 Run-4 XLNet 0 0 0.001 0.001 Semantic indexing of clinical codes is very desirable since term mapping of clinical codes are crucial now. We approached the semantic indexing by utilizing the transformers to predict the diagonstic (ICD10-CM ) and procedural (ICD10PCS) codes for the data. BERT, RoBERTa, Electra and XLNet are the transformers that have been ne tuned to predict the codes. All the four transformer models have been used in ICD10-CM codes which Electra performs better and RoBERTa performs better in predicting ICD10-PCS codes among RoBERTa, Electra and XLNet.
We would like to thank DST-SERB and HPC laboratory for providing resources needed for this work. 4. Gururangan, S., Marasovic, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., Smith, N.A.: Don't stop pretraining: Adapt language models to domains and tasks.
In: Proceedings of ACL (2020) 5. Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics (Sep 2019). https://doi.org/10.1093/bioinformatics/btz682, http://dx.doi.org/10.1093/bioinformatics/btz682 6. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach (2019) 7. Miranda-Escalada, A., Gonzalez-Agirre, A., Armengol-Estape, J., Krallinger, M.: Overview of automatic clinical coding: annotations, guidelines, and solutions for non-english clinical cases at codiesp track of CLEF eHealth 2020. In: Working Notes of Conference and Labs of the Evaluation (CLEF) Forum. CEUR Workshop Proceedings (2020) 8. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding (2019)